Command line CLI · 06 · 04

awk essentials

awk splits each line into fields $1..$NF. NR counts records; NF counts fields. Pattern/action pairs run selectively. BEGIN initializes state; END prints summaries. Use associative arrays for per-key counts and sums in one pass.

CLI Middle ◷ 25 min

Level

FoundationsJuniorMiddleSenior

cut extracts columns. sort | uniq -c counts them. sed rewrites them. But none of these can add a column of numbers, compute a ratio, or print a summary line after processing all input. awk can. It’s a complete mini-language built around the idea that text is a stream of records (lines by default), each split into fields ($1, $2, … $NF). For every record, awk evaluates a list of pattern/action pairs: if the pattern matches, the action runs. A BEGIN block sets up state before any input; an END block summarizes after. That model covers 80% of ad-hoc log analysis, report generation, and data transformation in a single command.

By the end of this lesson you will be able to print selected fields, filter records by pattern, accumulate sums, and produce a formatted summary.

Goal

After this lesson you can access record fields with $1–$NF, use NR and NF built-in variables, write pattern { action } pairs, use BEGIN to initialize and END to print summaries, set a custom field separator with -F, and accumulate numeric totals across records.

awk splits each line into fields $1, $2, … $NF. Field splitting uses whitespace by default (any run of spaces or tabs). $0 is the entire line; $NF is the last field regardless of how many fields there are; $(NF-1) is the second-to-last.

echo "alice  admin  1024" | awk '{print $1, $3}'
# alice 1024

The comma in print $1, $3 outputs the fields separated by the output field separator (a space by default). Without the comma — print $1 $3 — the fields are concatenated with no separator.

-F sets a custom input field separator:

awk -F: '{print $1, $7}' /etc/passwd

Splits on : instead of whitespace. Now $1 is the username and $7 is the shell — the same columns cut -d: -f1,7 extracts, but with far more power available in the action.

NR is the current record number (line number); NF is the number of fields on the current record.

awk '{print NR, NF, $0}' /etc/passwd | head -3
# 1 7 root:x:0:0:root:/root:/bin/bash
# 2 7 daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
# 3 7 bin:x:2:2:bin:/bin:/usr/sbin/nologin

NR is useful for: skipping headers (NR > 1), printing every Nth line (NR % 10 == 0), numbering output lines.

NF is useful for: detecting malformed lines (NF != 7), accessing the last field ($NF), or filtering lines that have a minimum number of columns.

# Skip the CSV header (line 1) and print lines with exactly 5 fields
awk -F, 'NR > 1 && NF == 5 {print $2, $4}' data.csv

Pattern/action pairs: pattern { action }. The pattern is any expression; the action runs only when the pattern is true. Either part is optional:

No pattern: action runs on every record.
No action: default action is {print $0} — print the whole line.

# Print lines where field 3 (response size) is > 10000
awk '$3 > 10000 {print $1, $3}' access.log

# Print lines containing 'ERROR' (pattern is a regex)
awk '/ERROR/ {print NR, $0}' app.log

# Print lines NOT matching a pattern
awk '!/DEBUG/' app.log

You can have multiple pattern/action pairs in one awk program — awk evaluates every pair for every record, running each matching action in order. This is different from if/else: multiple patterns can match the same record.

BEGIN and END blocks run once — before and after all input.

BEGIN { ... } — initialize variables, print a header, set FS (the field separator).
END { ... } — print totals, averages, summaries.

awk 'BEGIN { total = 0 }
     { total += $3 }
     END { print "Total bytes:", total }' access.log

total is initialized to 0 in BEGIN. For every line, total += $3 accumulates the third field. After all lines, END prints the total. This is the awk accumulator pattern — it requires no intermediate sort or external tools.

Setting FS in BEGIN is equivalent to -F:

awk 'BEGIN { FS = ":" } { print $1, $7 }' /etc/passwd

Combining patterns, accumulators, and END for real summaries.

# Count lines by HTTP status code in an nginx access log
awk '{status[$9]++}
     END { for (code in status) print status[code], code }' access.log \
  | sort -rn

status[$9]++ uses $9 (the status code field) as an associative array key and increments its count. After all input, END iterates the array with for (key in array) and prints each code with its count. Piping to sort -rn produces the ranked table.

Average response size by status code:

awk '{count[$9]++; total[$9]+=$10}
     END { for (c in count) printf "%s  avg=%.0f\n", c, total[c]/count[c] }' access.log

Two parallel arrays, one pass through the file. This is a pattern you would normally reach for pandas or SQL to express; awk handles it in a one-liner.

awk reads each record, splits it into fields, evaluates every pattern/action pair, and prints results. BEGIN runs once before input starts; END runs once after all input is consumed.

Worked example

Compute total and average request duration from a log where field 5 is duration in milliseconds.

awk 'NR > 1 && $5 ~ /^[0-9]+$/ {
       sum += $5
       count++
     }
     END {
       if (count > 0)
         printf "Requests: %d  Total: %dms  Avg: %.1fms\n", count, sum, sum/count
       else
         print "No valid records"
     }' requests.log

NR > 1 skips the header line. $5 ~ /^[0-9]+$/ checks that field 5 is a pure integer before adding it (guards against malformed lines where the field is - or empty). printf formats the output precisely. The END block guards against division by zero.

This pattern — guard-filter in the body, summary in END — is the production-quality awk idiom for numeric aggregation.

▸Why this works

On macOS, the system awk is the original one-true-awk (BSD awk), not GNU awk (gawk). For the patterns in this lesson — $1–$NF, NR, NF, BEGIN, END, associative arrays, printf — both behave identically. Differences only appear with advanced gawk features (OFMT, FIELDWIDTHS, gensub()). For scripts that need gawk extensions, install it explicitly (brew install gawk) and invoke it as gawk. All examples here are compatible with both.

▸Common mistake

The most common awk mistakes: (1) unquoted awk programs in the shell — always wrap the awk program in single quotes to prevent the shell from interpreting $1, {, and } before awk sees them; (2) using == to compare numbers stored as strings — awk auto-coerces strings to numbers in numeric context, so $3 > 10 works correctly even though $3 is a string; but $3 == "10" and $3 == 10 may differ when trailing spaces exist; (3) forgetting END when accumulating — variables in the body accumulate correctly but are never printed without an END block.

Check yourself

Quiz

You run: awk 'END {print NR}' file.txt. What does this print?

Recap

awk processes text as records split into fields $1–$NF. $0 is the whole record; NR is the current line number; NF is the field count. -F DELIM sets the input field separator. Pattern/action pairs run selectively — /regex/, numeric comparisons, or boolean expressions as patterns. BEGIN initializes variables and headers before input; END prints summaries after all input. Associative arrays (arr[key]++) enable per-key counting and summing in a single pass. awk is the right tool when you need arithmetic, conditional logic, or structured output that cut, sort, and sed cannot express alone.

Practice

Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.

recallapplystretch0 of 4 done

Something unclear?

Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.