STARCK LIN

The Complete Guide to Bash I/O Redirection: From File Descriptors 0/1/2 to Advanced Pipeline Tricks

Coding, Log—Tags: bash, linux, devops, command—Posted by Starck on

Translated from the Chinese original

In the Unix and Linux universe, a cornerstone design philosophy states: “Everything is a file.”

Terminal displays, keyboard inputs, disk files, network sockets, and hardware devices are all perceived by the operating system kernel as readable and writable byte streams.

I/O Redirection is therefore one of the most fundamental yet potent mechanisms available to developers and DevOps engineers. Mastering redirection allows you to control the flow of data effortlessly: discarding noisy error outputs, segregating logs in real time, or linking multiple processes into composable workflows.

1. Core Principles: File Descriptors (FD)

Whenever a Linux process initiates, the kernel automatically provides three standard data streams and identifies them with non-negative integers called File Descriptors:

File Descriptor Name Default Source / Destination Description
0 stdin (Standard Input) Keyboard Where programs read incoming user data
1 stdout (Standard Output) Terminal Screen Normal program output and execution results
2 stderr (Standard Error) Terminal Screen Diagnostic warnings, errors, and panic traces 
flowchart LR
    subgraph Process ["Linux Process"]
        direction TB
        FD0["FD 0 (stdin)"]
        FD1["FD 1 (stdout)"]
        FD2["FD 2 (stderr)"]
    end

    InputSrc["Keyboard / File (< file) / Here-Doc (<<EOF)"] --> FD0
    FD1 -->|"Standard Output (> / >>)"| StdoutFile["Log File (output.log)"]
    FD1 -->|"Terminal Display (Default)"| Screen["Terminal Screen (TTY)"]
    FD2 -->|"Error Output (2>)"| StderrFile["Error Log (error.log)"]
    FD2 -->|"Discard / Silence (2> /dev/null)"| BlackHole["/dev/null (Bit Bucket)"]
    FD2 -->|"Merge Stream (2>&1)"| FD1

Why Separate stdout (1) and stderr (2)?

Consider piping a chain of commands:

cat data.txt | grep "user" | sort

If grep encountered an out-of-memory error and sent its diagnostic message to stdout, the string "grep: memory error" would be piped directly into sort, corrupting the data stream.

By strictly separating standard output (FD 1) from diagnostic error reporting (FD 2), pipes (|) by default pass only stdout downstream while letting stderr print directly to the user’s terminal, guaranteeing the composability of Unix utilities.

2. Basic Redirection Operators

1. Output Redirection: > vs. >>

  • > (Truncate & Overwrite): If the destination file exists, it is emptied before writing; if it does not exist, a new file is created.
  • >> (Append): Preserves existing contents and appends new data to the end of the file.
echo "Hello World" > greetings.txt   # Overwrites
echo "Another line" >> greetings.txt # Appends

[!TIP] Safety Tip: Setting set -o noclobber in Bash prevents > from accidentally clobbering existing files. If you genuinely want to force-overwrite, use >|.

2. Input Redirection: <, <<, and <<<

  • < file: Reads file content and feeds it directly into standard input. 
    wc -l < /etc/passwd
  • << EOF (Here-Document): Passes multi-line text into a command until the matching delimiter token is reached: 
    cat << EOF > config.yaml
server:
  host: 127.0.0.1
  port: 8080
EOF
    (Quoting the delimiter like << 'EOF' disables variable interpolation inside the block).
  • <<< "string" (Here-String): Feeds a single string straight into stdin, eliminating the overhead of spawning an echo ... | subshell: 
    tr 'a-z' 'A-Z' <<< "hello bash redirection"
# Output: HELLO BASH REDIRECTION

3. Controlling FD 1 and FD 2 with Precision

Prefixing redirection operators with the file descriptor integer routes that specific stream. Omitting the number defaults to 1 (stdout).

1. Splitting Standard Output and Errors into Separate Files

# Normal results go to output.txt, errors go to error.txt
./deploy-script.sh 1> output.txt 2> error.txt
# Equivalent to:
./deploy-script.sh > output.txt 2> error.txt

2. Silent Mode: Discarding Output to /dev/null

/dev/null is a special pseudo-device file that discards everything written to it.

# Search the entire root drive, discarding all "Permission denied" noise
find / -name "nginx.conf" 2> /dev/null

# Check if a Ruby gem is installed; suppress error output if missing
gem spec zurb-foundation 2> /dev/null

4. The Classic Trap: > file 2>&1 vs. 2>&1 > file

When you want to merge both standard output and standard error into a single log file, the order of evaluation matters.

The Correct Syntax:

command > all.log 2>&1

The Broken Syntax:

command 2>&1 > all.log   # WRONG: Error messages still leak onto the screen!

Why Does Order Matter?

flowchart TD
    subgraph Correct ["> file 2>&1 (Correct: Evaluated Left-to-Right)"]
        direction TB
        C1["Initial state: FD 1 -> TTY, FD 2 -> TTY"]
        C2["1. Evaluates > file: FD 1 redirected to all.log"]
        C3["2. Evaluates 2>&1: FD 2 points to where FD 1 currently points (all.log)"]
        C4["Final Result: Both stdout and stderr write to all.log ✅"]
        C1 --> C2 --> C3 --> C4
    end

    subgraph Wrong ["2>&1 > file (Flawed Order)"]
        direction TB
        W1["Initial state: FD 1 -> TTY, FD 2 -> TTY"]
        W2["1. Evaluates 2>&1: FD 2 points to where FD 1 currently points (still TTY!)"]
        W3["2. Evaluates > file: FD 1 redirected to all.log"]
        W4["Final Result: stdout writes to file, but stderr still prints to screen ❌"]
        W1 --> W2 --> W3 --> W4
    end

[!IMPORTANT] Modern Bash Shorthand (Bash 4.0+): Modern versions of Bash provide a cleaner syntax to combine both streams: &> and &>>:

command &> all.log     # Overwrite both
command &>> all.log    # Append both

5. Real-World Patterns and Advanced Techniques

1. The sudo Redirection Permission Trap

A classic blunder when attempting to write to privileged files:

sudo echo "nameserver 1.1.1.1" > /etc/resolv.conf
# ❌ Error: Permission denied

Why it fails: The shell initiates the redirection > with your normal user privileges before executing the sudo command!

The solution: Use tee to handle write operations with elevated privileges:

echo "nameserver 1.1.1.1" | sudo tee /etc/resolv.conf > /dev/null

2. Piping Standard Error: |&

By default, the pipe operator | forwards only stdout. To pipe both stdout and stderr into the next command (e.g., for filtering):

# Traditional syntax:
make 2>&1 | grep "error"

# Modern Bash shorthand:
make |& grep "error"

3. Process Substitution: <()

Need to compare the output of two commands without generating temporary disk files?

diff -u <(curl -s https://api.site.com/v1/users | jq .) <(curl -s https://api.site.com/v2/users | jq .)

<(...) connects the output of a command to a named pipe (FIFO), allowing utilities expecting file arguments to read command outputs directly.

4. Global Redirection with exec

When writing extensive multi-step provisioning scripts, you do not need to append >> log.txt 2>&1 to every single command. Redirect the entire script session once using exec:

#!/usr/bin/env bash

# Redirect all subsequent stdout and stderr to both terminal and disk
exec > >(tee -a "/var/log/deploy.log") 2>&1

echo "This message prints to the screen and is saved to the log file"
ls -la /some/directory

6. Bash Redirection Quick Reference

Operator Purpose Example
> Redirect stdout (FD 1), overwrite ls > files.txt
>> Redirect stdout (FD 1), append echo "log" >> app.log
2> Redirect stderr (FD 2), overwrite find / 2> err.log
2>> Redirect stderr (FD 2), append command 2>> err.log
2> /dev/null Silence error output grep "text" file 2> /dev/null
> f 2>&1 Merge stdout and stderr into file cmd > all.log 2>&1
&> / &>> Modern shorthand for merged redirection cmd &> all.log
< Redirect standard input from file wc -w < doc.txt
<< EOF Here-Document (multi-line inline text) cat << EOF > file
<<< Here-String (single string input) base64 -d <<< "U3RhcmNr"
` &` Combined pipe (stdout + stderr)
<(cmd) Process substitution (command as file) diff <(cmd1) <(cmd2)

Conclusion

I/O redirection is the lifeblood of the Linux shell. From a humble 2> /dev/null error suppression to sophisticated exec logging pipelines, a firm grasp of file descriptors gives you command and confidence over your systems and automation scripts.