I/O, Redirection and Pipes

Every process starts with three open files, and redirection is nothing more than swapping what those files point at before the program runs. Once you see it that way, 2>&1, &>, and <(...) stop being magic incantations.

File descriptors

A file descriptor (fd) is a small integer the kernel uses to identify an open file for a process. Three are opened for you:

fd	Name	Default	Purpose
0	stdin	keyboard / terminal	input
1	stdout	terminal	normal results
2	stderr	terminal	errors, warnings, progress

The split between 1 and 2 exists so you can redirect results without losing errors. Both go to your terminal by default, which is why they look identical until you redirect one.

ls /etc /nope
# /etc contents go to fd 1
# "ls: cannot access '/nope'" goes to fd 2

ls /etc /nope > out.txt      # out.txt gets the listing; the error still prints
ls /etc /nope 2> err.txt     # the listing prints; err.txt gets the error

Output redirection

cmd > file       # send stdout to file, TRUNCATING it first
cmd >> file      # append instead
cmd > /dev/null  # discard stdout

The redirection is set up before the command runs, and it happens even if the command does not exist — which is why > file empties a file instantly:

> logfile        # truncate a file to zero bytes, no command needed
: > logfile      # the explicit, clearer version

noclobber protects against accidental truncation:

set -o noclobber
echo hi > existing.txt     # bash: existing.txt: cannot overwrite existing file
echo hi >| existing.txt    # >| forces it anyway
set +o noclobber           # turn it back off

Input redirection

sort < names.txt          # feed a file to stdin
wc -l < access.log        # note: prints just the number, no filename

wc -l file prints 42 file; wc -l < file prints 42. Redirection is often the easiest way to get a bare value out of a tool.

Redirecting stderr

2> targets fd 2 specifically. The duplication operator >& makes one fd point at the same place as another.

cmd 2> errors.log          # errors to a file
cmd 2> /dev/null           # silence errors only
cmd > out.log 2>&1         # BOTH to out.log
cmd &> out.log             # bash shorthand for the same thing
cmd &>> out.log            # append form (bash 4.0+)
cmd 2>&1 | less            # merge, then pipe (or: cmd |& less, bash 4.0+)

Order matters, and it is the classic gotcha. 2>&1 means “make fd 2 a copy of whatever fd 1 currently is”, evaluated left to right:

cmd > out.log 2>&1    # 1 -> out.log, then 2 -> (copy of 1) = out.log     CORRECT
cmd 2>&1 > out.log    # 2 -> (copy of 1) = terminal, then 1 -> out.log    stderr still prints

The second form is occasionally deliberate — it swaps stderr onto the terminal while stdout goes to the file — but it is almost always a mistake.

Swapping stdout and stderr requires a temporary fd:

cmd 3>&1 1>&2 2>&3 3>&-

Useful device files

cmd > /dev/null 2>&1     # discard everything
cmd < /dev/null          # guarantee no stdin (stops interactive prompts hanging)
echo "note" > /dev/tty   # write straight to the terminal, ignoring redirection

/dev/null discards writes and returns EOF on reads. /dev/tty is the controlling terminal — useful for prompts that must be seen even when the script’s output is piped.

Redirecting blocks, loops, and whole scripts

Any compound command takes redirections:

while IFS= read -r line; do
  echo "> $line"
done < input.txt > output.txt      # loop reads input.txt, writes output.txt

{
  echo "header"
  generate_body
  echo "footer"
} > report.txt

for i in 1 2 3; do echo "$i"; done >> log.txt

exec with only redirections changes the current shell’s descriptors for everything that follows:

#!/usr/bin/env bash
exec >> /var/log/myscript.log 2>&1     # everything from here on is logged
echo "started at $(date)"              # goes to the log

You can also open your own descriptors — handy when a script needs a second input stream:

exec 3< hosts.txt          # open for reading on fd 3
read -r -u 3 first_host    # read one line from fd 3
while read -r -u 3 host; do
  ssh "$host" uptime       # ssh consumes fd 0, but not fd 3
done
exec 3<&-                  # close it

exec 4> results.txt        # open for writing
echo "line" >&4
exec 4>&-

Pipes

a | b connects a’s stdout to b’s stdin. Both run concurrently in separate processes, with the kernel buffering between them (typically 64 KiB); b starts immediately and blocks when the buffer is empty.

ps aux | grep nginx | awk '{print $2}' | xargs kill
cat access.log | wc -l          # works, but `wc -l < access.log` is one process fewer

Only stdout flows through a pipe. To pipe errors too, merge first:

make 2>&1 | tee build.log

Exit status of a pipeline

By default the pipeline’s status is that of the last command. This hides failures:

false | true ; echo "$?"        # => 0   — the failure vanished

Two fixes:

set -o pipefail                 # status = rightmost non-zero, or 0 if all succeeded
false | true ; echo "$?"        # => 1

# or inspect each stage
grep foo big.log | sort | head
echo "${PIPESTATUS[@]}"         # => 0 0 0   one status per stage

PIPESTATUS is overwritten by the next command, so copy it if needed: statuses=("${PIPESTATUS[@]}").

Pipelines run in subshells

Every stage of a pipeline is a separate process, so variable changes in a stage do not survive:

count=0
printf 'a\nb\nc\n' | while read -r _; do (( count++ )); done
echo "$count"          # => 0   — the loop ran in a subshell

Fixes, best first:

count=0
while read -r _; do (( count++ )); done < <(printf 'a\nb\nc\n')   # process substitution
echo "$count"          # => 3

shopt -s lastpipe      # bash 4.2+, runs the LAST pipeline stage in the current shell
                       # (only takes effect when job control is off, i.e. in scripts)

Here-documents

A here-document feeds literal text to a command’s stdin.

cat <<EOF
User: $USER
Home: $HOME
EOF

The delimiter (EOF is only a convention) must appear alone on a line, at column 0, to end the block. Inside an unquoted here-doc, $variables, $(commands), and backslashes are expanded — exactly like double quotes.

Quote the delimiter to turn expansion off, which is what you want for scripts, configs, and anything containing $:

cat <<'EOF' > deploy.sh
#!/usr/bin/env bash
echo "PATH is $PATH"      # written literally, not expanded now
EOF

<<- strips leading tab characters (not spaces) from each line, so the block can be indented with the surrounding code:

if true; then
  cat <<-EOF
    indented in the source
    flush-left in the output
  EOF
fi

Common uses:

mysql -u root <<'SQL'
SELECT COUNT(*) FROM users;
SQL

ssh server 'bash -s' <<'REMOTE'
set -euo pipefail
systemctl restart myapp
REMOTE

read -r -d '' template <<'EOF' || true   # capture a block into a variable
line one
line two
EOF

(read -d '' returns non-zero at EOF, hence the || true.)

Here-strings

<<< feeds a single string as stdin. It appends a trailing newline.

grep -o '[0-9]\+' <<< "order 66 confirmed"     # => 66
read -r a b c <<< "one two three"
jq '.name' <<< "$json_response"
tr 'a-z' 'A-Z' <<< "$message"

Much cheaper than echo "$x" | cmd, and it avoids the pipeline subshell.

tee

tee copies stdin to stdout and to files — a T-junction in the pipeline.

make 2>&1 | tee build.log                 # watch it and save it
make 2>&1 | tee -a build.log              # append instead of truncate
generate | tee raw.txt | grep ERROR       # save the full stream, filter what you see
generate | tee >(gzip > out.gz) | wc -l   # tee into another process

The sudo tee idiom exists because redirection is performed by your shell, not by sudo:

sudo echo "text" > /etc/protected      # FAILS: your shell opens the file, unprivileged
echo "text" | sudo tee /etc/protected >/dev/null    # works
echo "text" | sudo tee -a /etc/protected >/dev/null # append

Process substitution

<(cmd) runs a command and hands the reader a filename (/dev/fd/63 on Linux) that streams its output. It is how you give a program a “file” that does not exist on disk.

diff <(sort a.txt) <(sort b.txt)               # compare without temp files
diff <(ssh host1 'rpm -qa') <(ssh host2 'rpm -qa')
comm -13 <(sort old.txt) <(sort new.txt)       # lines added

The reason it matters beyond convenience: it keeps the loop in the current shell.

total=0
while IFS= read -r n; do (( total += n )); done < <(seq 1 5)
echo "$total"      # => 15

Note the space in < <( — <( is the substitution, < is the redirection.

>(cmd) is the output direction: the filename accepts writes and feeds them into the command’s stdin.

tar czf - /data | tee >(sha256sum > data.sha256) > data.tar.gz
./build 2> >(grep -v 'warning:' >&2)     # filter stderr only

For a durable version of the same idea, create a named pipe:

mkfifo /tmp/mypipe
producer > /tmp/mypipe &
consumer < /tmp/mypipe
rm /tmp/mypipe

How this interacts with `set -euo pipefail`

The best practices page covers the whole preamble; the redirection-specific parts:

set -e does not trigger on a failing pipeline stage unless pipefail is on. With set -euo pipefail, curl bad-url | jq . correctly aborts.
A command that fails only because the file it redirects to cannot be opened does abort under set -e — the redirection failure is the command’s failure.
pipefail plus head/grep -q early exit produces status 141 (SIGPIPE). Expect it.
cmd > "$file" with $file unset is caught by set -u; without it, bash creates a file literally named by the empty expansion or errors with “ambiguous redirect”.

set -euo pipefail

# ambiguous redirect: an unquoted variable that expands to multiple words
files="a.txt b.txt"
echo hi > $files      # bash: $files: ambiguous redirect
echo hi > "$files"    # creates a file literally named "a.txt b.txt"

Key points

Redirection rewires file descriptors before the command starts; > truncates, >> appends.
> file 2>&1 merges stderr into stdout; the reverse order does not, because 2>&1 copies fd 1’s current target.
Send diagnostics to stderr (>&2) so your script composes cleanly in pipelines.
A pipeline’s status is its last stage unless set -o pipefail; PIPESTATUS has them all.
Pipeline stages are subshells — use < <(cmd) when the loop must set variables.
Quote the here-doc delimiter (<<'EOF') whenever the body should stay literal.
tee, process substitution, and sudo tee cover almost every “I need this stream in two places” problem.