How to Use the sed Command in Linux for Stream Editing and Regular Expression Replacements

In Unix systems engineering, DevOps pipeline automation, and configuration management, processing text streams and refactoring configuration files without opening interactive editors is a vital operational skill. The Stream Editor (sed) serves as the standard utility for non-interactive text parsing, filtering, and transformation across Linux and Unix environments. Reading line by line into an internal memory buffer known as the pattern space, sed applies regular expression substitutions, line deletions, text insertions, and multi-line transformations with unmatched execution speed and low resource consumption.

The sed Execution Model: Pattern Space and Hold Space

Understanding the internal memory cycle of sed is crucial for authoring predictable text processing pipelines. The table below delineates the memory architecture and processing phases of the stream editor:

Memory / Execution Phase	Architectural Role	Operational Mechanism
Pattern Space	Active working buffer.	Holds the current input line read from stdin or a file while sed commands evaluate and mutate it.
Hold Space	Auxiliary storage buffer.	Retains temporary data across line cycles; manipulated via exchange commands (h, H, g, G, x).
Read Cycle	Input ingestion.	Strips trailing newlines, loads the line into pattern space, and initialises the cycle counter.
Execution Cycle	Command evaluation.	Applies all matching commands sequentially in script order to the content in pattern space.
Output Cycle	Standard output flush.	Prints the pattern space to stdout (unless suppressed with -n) and clears the pattern buffer.

Mastering the Substitution Command (s///)

The substitution verb (s) is the most frequently deployed sed command. Its syntax follows s/regex/replacement/flags.

1. Basic Search and Replace

To substitute the first occurrence of the word “staging” with “production” on each line of a configuration file:

sed 's/staging/production/' app.conf

To replace all occurrences across each line globally, append the g flag:

sed 's/staging/production/g' app.conf

To perform case-insensitive matching, combine the global flag with the I (or i) flag:

sed 's/error/warning/gI' syslog.txt

2. Altering Delimiters for Path and URL Processing

When manipulating file paths or web URLs, standard forward slashes require tedious escaping (the “leaning toothpick syndrome”). sed allows you to designate any arbitrary character as the delimiter:

# Cluttered syntax:
sed 's/\/var\/www\/html/\/srv\/web\/public/g' config.php

# Elegant alternative using hash delimiters:
sed 's#/var/www/html#/srv/web/public#g' config.php

In-Place File Editing with sed -i

By default, sed outputs transformed text to standard output without altering the source file on disk. The -i flag instructs sed to write modifications directly back to the target file.

1. In-Place Editing with Automatic Backup

To modify a file in-place while generating a safety backup of the original:

sed -i.bak 's/DEBUG = True/DEBUG = False/' settings.py

This command alters settings.py and creates settings.py.bak containing the pre-modified state.

2. The GNU Linux vs. macOS BSD In-Place Discrepancy

A frequent cross-platform pitfall occurs when running scripts between Linux (GNU sed) and macOS (BSD sed):

• GNU Linux: sed -i 's/foo/bar/' file.txt works without specifying a backup extension.
• macOS BSD: sed -i '' 's/foo/bar/' file.txt strictly requires an explicit empty string argument ('') to omit backups.

Addressing Lines by Number, Range, and Regular Expression

Without an address prefix, sed executes commands across every line of input. Specifying addresses restricts execution to targeted segments.

1. Line Number Addresses

• Modify Strictly Line 42:

  sed '42 s/false/true/' config.json

• Target Line Ranges (e.g., Lines 10 through 25):

  sed '10,25 s/legacy_api/v2_api/g' service.py

• From Line 50 to the End of the File ($):

  sed '50,$ s/old/new/g' data.csv

2. Pattern-Based Addressing and Deletion (d)

Addresses can also be defined by regular expressions:

# Delete all blank lines from a file
sed '/^$/d' source.txt

# Delete all commented lines starting with '#'
sed '/^[[:space:]]*#/d' nginx.conf

# Delete lines between two matching delimiters
sed '/BEGIN_CERTIFICATE/,/END_CERTIFICATE/d' bundle.pem

Extended Regular Expressions with sed -E

Standard Basic Regular Expressions (BRE) in sed require escaping special metacharacters like +, ?, |, and capturing parentheses ( ). The -E (or -r) flag enables Extended Regular Expressions (ERE), drastically improving readability.

1. Backreferences and Capturing Groups

To extract IP addresses and swap them with hostnames using capture groups (\1, \2):

sed -E 's/([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})[[:space:]]+([a-zA-Z0-9.-]+)/\2 (\1)/' /etc/hosts

This expression captures the IP address in group 1 and the hostname in group 2, reformatting 192.168.1.50 server1.lan into server1.lan (192.168.1.50).

Inserting, Appending, and Printing Lines

Beyond substitutions, sed handles structural text injections and selective output generation.

• Insert Before Matching Line (i):

  sed '/server_name/i \    # Automated reverse proxy configuration' nginx.conf

• Append After Matching Line (a):

  sed '/Listen 80/a Listen 443' httpd.conf

• Selective Printing with -n and p: Suppress standard output and print strictly lines matching a specific pattern (ideal for log extraction):

  sed -n '/CRITICAL/p' /var/log/syslog

Troubleshooting Common sed Scripting Errors

When authoring multi-step stream pipelines, watch out for these operational traps:

• Unintended Greediness of Metacharacters: Standard regular expressions in sed are greedy. In the string <b>bold</b> text <b>word</b>, the pattern <.*> matches from the first < to the very last >. Restrict the match using negated character sets: <[^>]*>.
• Special Metacharacters in Shell Variables: When interpolating bash variables into sed expressions, use double quotes instead of single quotes, but beware of slashes inside the variable:

  TARGET_DIR="/opt/app/bin"
  # Will fail with syntax error:
  sed "s/PATH/$TARGET_DIR/" file.txt
  # Works correctly with alternative delimiter:
  sed "s|PATH|$TARGET_DIR|" file.txt

• Accidental Overwriting via Shell Redirection: Never run sed 's/foo/bar/' file.txt > file.txt. The shell truncates file.txt before sed reads it, resulting in a zero-byte empty file. Always use sed -i for in-place modifications.