Text Processing

The Unix toolkit assumes text arrives as lines, each line as fields. A dozen small programs, each doing one transformation, compose into pipelines that replace a surprising amount of code.

The model

Almost every tool here reads stdin (or files named as arguments), writes stdout, and treats the newline as the record separator. That uniformity is what makes | work. Two habits follow from it:

• Filter early. Put grep before sort so you sort less.
• Set LC_ALL=C when you want byte semantics and speed. Locale-aware collation makes sort and grep noticeably slower and changes what [a-z] matches.

LC_ALL=C sort huge.txt | LC_ALL=C uniq -c

grep — find lines

grep prints lines matching a pattern. Exit status is 0 if anything matched, 1 if not, 2 on error — which makes it a natural if condition.

grep "ERROR" app.log                 # basic regex (BRE)
grep -E "ERROR|WARN" app.log         # extended regex (ERE): |, +, ?, () unescaped
grep -F "1.2.3.4" access.log         # fixed string: no regex, much faster
grep -i "error" app.log              # case-insensitive
grep -v "healthcheck" access.log     # invert: lines that do NOT match
grep -w "id" schema.sql              # whole word only ("id", not "uuid")
grep -c "ERROR" app.log              # count matching lines
grep -n "TODO" main.py               # prefix with line numbers
grep -l "TODO" *.py                  # just list filenames that match (-L for those that don't)
grep -q "ERROR" app.log              # quiet: no output, status only

-o prints only the matched part, one per line — the key to extracting data:

grep -oE '[0-9]{1,3}(\.[0-9]{1,3}){3}' access.log     # every IP address
grep -oE 'https?://[^"]+' page.html                   # every URL

Recursive search:

grep -rn "getUser" src/                     # -r recurses, -n shows line numbers
grep -rn --include='*.ts' "TODO" .          # only .ts files
grep -rn --exclude-dir={node_modules,.git} "apiKey" .
grep -rIn "secret" .                        # -I skips binary files

-r skips symlinks found during the walk; -R follows them.

Context around matches:

grep -A3 "Traceback" app.log     # 3 lines After
grep -B2 "ERROR" app.log         # 2 lines Before
grep -C2 "ERROR" app.log         # 2 lines of Context both sides

Caution

Always quote the pattern: grep "$pat" file — unquoted, the shell may glob it or split it. And use grep -e "$pat" or grep -- "$pat" when the pattern can start with -, otherwise grep reads it as an option.

sed — stream edit

sed applies editing commands to each line. In practice you use three of them.

Substitute

sed 's/old/new/' file            # first occurrence per line
sed 's/old/new/g' file           # every occurrence
sed 's/old/new/2' file           # only the 2nd occurrence on each line
sed 's/old/new/gi' file          # global + case-insensitive
sed -E 's/(a+)b/[\1]/g' file     # -E for extended regex; \1 is a capture group
sed 's|/usr/local|/opt|g' file   # any character can be the delimiter — handy for paths
sed 's/foo/&bar/' file           # & in the replacement means "the whole match"

Editing files in place:

sed -i 's/old/new/g' file          # GNU sed
sed -i.bak 's/old/new/g' file      # GNU: also keep file.bak
sed -i '' 's/old/new/g' file       # BSD/macOS requires an explicit (possibly empty) suffix

Caution

sed -i is not portable: GNU takes an optional attached suffix, BSD/macOS requires a separate argument. A script that must run on both should write to a temp file and mv it, or use perl -i -pe.

Delete and print

sed '/^$/d' file             # delete blank lines
sed '/^#/d' config           # delete comment lines
sed '1d' file                # delete the first line (a header)
sed '$d' file                # delete the last line
sed '2,5d' file              # delete a range

sed -n '10,20p' file         # -n suppresses auto-print; p prints -> lines 10-20
sed -n '/BEGIN/,/END/p' file # everything between two markers, inclusive
sed -n 's/^version: //p' f   # print only the substituted part (a value extractor)
sed '5q' file                # quit after line 5 — like head -5, but stops reading

Addresses (/regex/, line numbers, $, ranges) can prefix any command, and ! negates them: sed -n '/ERROR/!p' prints non-matching lines.

awk — fields and arithmetic

awk splits each line into fields and runs pattern { action } rules against it. It is a real programming language; these idioms cover most day-to-day use.

awk '{print $1}' access.log            # first whitespace-separated field
awk '{print $NF}' access.log           # last field (NF = number of fields)
awk '{print $(NF-1)}' access.log       # second to last
awk '{print NR": "$0}' file            # NR = record (line) number, $0 = whole line
awk -F: '{print $1, $7}' /etc/passwd   # -F sets the field separator
awk -F'\t' '{print $2}' data.tsv
awk 'BEGIN{FS=":"; OFS=" -> "} {print $1, $6}' /etc/passwd

Patterns filter which lines the action runs on:

awk '/ERROR/ {print $0}' app.log          # regex on the whole line
awk '$3 > 100' metrics.txt                # numeric comparison; default action is print
awk '$1 == "GET"' access.log
awk 'NF == 0 {blank++} END {print blank}' file
awk 'NR > 1' data.csv                     # skip a header row
awk 'length($0) > 80' src.c               # long lines

BEGIN runs before the first line, END after the last — which is where totals go:

awk '{sum += $3} END {print sum}' sales.txt
awk '{sum += $1; n++} END {printf "avg %.2f\n", sum/n}' nums.txt
awk '{sum += $5} END {printf "%.1f MB\n", sum/1048576}' sizes.txt

Associative arrays make grouping trivial — this is awk’s superpower:

# requests per HTTP status code
awk '{count[$9]++} END {for (c in count) print c, count[c]}' access.log

# bytes transferred per client IP, sorted
awk '{bytes[$1] += $10} END {for (ip in bytes) print bytes[ip], ip}' access.log | sort -rn | head

# unique values of column 2, preserving first-seen order
awk '!seen[$2]++ {print $2}' data.tsv

Passing shell values in safely (never interpolate into the program text):

threshold=100
awk -v limit="$threshold" '$3 > limit' metrics.txt

Use single quotes around the awk program so $1 stays awk’s field and not a shell parameter.

cut — fixed fields

cut is simpler and faster than awk when the delimiter is a single character.

cut -d: -f1 /etc/passwd            # field 1, colon-delimited
cut -d, -f1,3 data.csv             # fields 1 and 3
cut -d, -f2- data.csv              # field 2 to the end
cut -c1-10 file                    # characters 1-10 of each line
cut -d: -f1 --complement /etc/passwd   # everything except field 1 (GNU)

Limitations that push you back to awk: cut cannot treat runs of spaces as one delimiter, cannot reorder fields (-f3,1 still prints 1 then 3), and knows nothing about quoted CSV fields. For real CSV, use a CSV-aware tool or a scripting language.

sort

sort file                   # lexicographic, by locale
sort -n file                # numeric
sort -h file                # human-readable numbers: 2K, 3M, 1G (GNU)
sort -V file                # version strings: 1.9 before 1.10 (GNU)
sort -r file                # reverse
sort -u file                # sort and drop duplicates
sort -k2 file               # by field 2 to end of line
sort -k2,2 file             # by field 2 only
sort -t: -k3,3n /etc/passwd # colon-delimited, field 3, numeric
sort -k2,2 -k1,1r file      # multiple keys, second one reversed
sort -s -k1,1 file          # stable: keep original order within equal keys

The -k syntax is -k START[,END][OPTS]. Forgetting the ,END is the usual bug — -k2 means “from field 2 to the end of the line”, which rarely sorts how you expect.

uniq

uniq only collapses adjacent duplicates, so it is nearly always preceded by sort.

sort file | uniq            # deduplicate (or just: sort -u file)
sort file | uniq -c         # prefix each line with its count
sort file | uniq -d         # only lines that appear more than once
sort file | uniq -u         # only lines that appear exactly once
sort file | uniq -i         # case-insensitive

The single most-used pipeline in operations work:

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20
# => counts per client IP, most frequent first

tr — translate and delete characters

tr works on characters, not lines, and reads only stdin — it takes no filenames.

tr 'a-z' 'A-Z' < file          # uppercase
tr -d '\r' < dos.txt > unix.txt   # strip carriage returns
tr -d '[:space:]' <<< " a b "     # delete all whitespace
tr -s ' ' <<< "a    b"            # squeeze repeats: "a b"
tr ' ' '\n' <<< "one two three"   # split words onto lines
tr -cd '[:print:]\n' < file       # -c complements: keep only printable chars

Character classes: [:alpha:], [:digit:], [:alnum:], [:space:], [:punct:], [:upper:], [:lower:].

wc

wc -l file      # lines      wc -w words     wc -c bytes     wc -m characters
wc -l < file    # just the number, no filename
wc -l *.log     # per-file counts plus a total

wc -l counts newline characters, so a final line without a trailing newline is not counted.

head and tail

head -n 20 file        # first 20 lines (-20 also works)
head -c 100 file       # first 100 bytes
tail -n 20 file        # last 20 lines
tail -n +2 file        # from line 2 onward — i.e. skip the header
tail -f app.log        # follow: print new lines as they are appended
tail -F app.log        # follow by name; survives log rotation (GNU)
tail -f app.log | grep --line-buffered ERROR    # live filtering

--line-buffered matters: when grep’s output is a pipe it buffers in 4 KiB blocks, so a live tail appears frozen without it. The equivalent for sed is -u, and for awk fflush().

find — walk the filesystem

find evaluates expressions against every path it walks. Tests are combined implicitly with AND.

find . -name '*.log'                 # by name (glob — quote it, or the shell expands it first)
find . -iname '*.LOG'                # case-insensitive
find . -type f                       # files only  (d directories, l symlinks)
find . -maxdepth 1 -type d           # do not recurse
find /var/log -mtime -7              # modified in the last 7 days (+7 = older than 7)
find . -mmin -10                     # modified in the last 10 minutes
find . -size +100M                   # larger than 100 MB (+100k, +1G)
find . -empty                        # empty files and directories
find . -user deploy -perm -u+x       # by owner and permission bits
find . -path '*/node_modules/*' -prune -o -name '*.js' -print   # skip a subtree
find . -name '*.tmp' -delete         # delete matches (safer than -exec rm)

Running a command per result:

find . -name '*.log' -exec gzip {} \;      # one gzip process per file; {} is the path
find . -name '*.log' -exec gzip {} +       # batch many paths into few processes — much faster
find . -type f -exec grep -l TODO {} +
find . -name '*.bak' -ok rm {} \;          # -ok prompts before each command

\; runs the command once per file; + appends as many paths as fit per invocation. Prefer +.

For piping to other tools, use -print0 so newlines in filenames cannot break the stream:

find . -type f -name '*.log' -print0 | xargs -0 rm --

xargs — build command lines from stdin

xargs reads whitespace-separated words from stdin and appends them as arguments.

find . -name '*.pyc' -print0 | xargs -0 rm -f
grep -rl 'oldname' . | xargs sed -i 's/oldname/newname/g'
cat urls.txt | xargs -n1 curl -sS -o /dev/null -w '%{http_code} %{url_effective}\n'

Flag	Effect
-0	Input is NUL-separated (pairs with find -print0, grep -z)
-n N	At most N arguments per command
-I {}	Replace {} with the input item; implies one item per command
-P N	Run N commands in parallel
-r	Do not run at all if input is empty (GNU; BSD already behaves this way)
-t	Print each command before running it

# placeholder form: put the argument somewhere other than the end
cat hosts.txt | xargs -I{} ssh {} 'uptime'

# parallelism
find . -name '*.png' -print0 | xargs -0 -P 8 -n 1 optipng

# dry run first — always worth it before a destructive xargs
find . -name '*.tmp' -print0 | xargs -0 -t echo rm

Danger

Bare xargs splits on whitespace and interprets quotes, so a file called my report.txt becomes two arguments. Always pair find -print0 with xargs -0. With -I{}, note that GNU xargs splits input on newlines only, which is safe for normal filenames but still breaks on newlines in names.

Building real pipelines

Read a pipeline left to right as a series of narrowing steps.

Top 20 client IPs in an access log

awk '$9 == 404 {print $1}' access.log | sort | uniq -c | sort -rn | head -20

Count files by extension

find . -type f -name '*.*' | sed 's/.*\.//' | sort | uniq -c | sort -rn

Total size per subdirectory, largest first

du -sh -- */ | sort -h -r | head

Every unique TODO with its location

grep -rn --exclude-dir={.git,node_modules} -E 'TODO|FIXME' . |
  sed -E 's/:[0-9]+:/: /' |
  sort -u

Slowest requests from a log whose 11th field is a duration

awk '{print $11, $7}' access.log | sort -rn | head -10

Extract and count HTTP status codes

awk '{print $9}' access.log | sort | uniq -c | sort -rn

Find large files not touched in a year

find /data -type f -size +100M -mtime +365 -printf '%s\t%p\n' |
  sort -rn |
  awk '{printf "%.1f MB\t%s\n", $1/1048576, $2}'

(-printf is GNU find; on BSD use -exec stat instead.)

A word frequency count

tr -cs '[:alpha:]' '\n' < book.txt |
  tr '[:upper:]' '[:lower:]' |
  sort |
  uniq -c |
  sort -rn |
  head -20

Tip

Build pipelines incrementally: run the first stage, look at the output, then add | next. Piping through head while iterating keeps the feedback fast on large files.

Choosing the right tool

Task	Reach for
Does this line contain X?	grep
Extract a substring per line	grep -o or sed -n 's/…/…/p'
Replace text	sed s///
Work with columns, or do arithmetic	awk
Split on one fixed character	cut
Order or deduplicate	sort, sort -u, uniq -c
Character-level edits	tr
Select files by attribute	find
Turn a list into commands	xargs
Structured JSON	jq (not part of coreutils; install separately)

Two anti-patterns worth naming: do not parse ls (use globs or find), and do not chain three greps where one awk would do — but do not write a 40-line awk program either. When the logic gets real, switch to Python.

Key points

• Everything is lines of fields; tools compose through stdout/stdin.
• grep selects lines, sed edits them, awk understands fields and does arithmetic.
• sort | uniq -c | sort -rn is the counting idiom — memorize it.
• find -print0 | xargs -0 is the only safe way to feed arbitrary filenames into another command.
• find -exec … + batches; \; forks per file.
• LC_ALL=C speeds up sorting and grepping when you do not need locale collation.