How to Pipe Output and Redirect to Files in awk on Linux

When you are architecting a complex awk script within a Linux terminal that processes a master data stream (e.g., a unified server log), you may need to simultaneously route different categories of data to different output files. If you process the entire file once and then run separate scripts for each output, you are wasting catastrophic amounts of CPU cycles by reading the same multi-gigabyte file multiple times. To force the engine to read the file only once and dynamically pipe specific rows to specific external commands or output files in real time, you must deploy the Pipe Operator (|) and Output Redirection (>) protocols from within the awk execution block.

Executing the Pipe Routing Matrix

The GNU awk engine possesses a native ability to open external shell pipes and file handles directly from within a print statement. By appending a pipe (|) or a redirection arrow (>) after the print command, you instruct the engine to route the output payload to a specific destination instead of the default stdout terminal stream.

Deploying the Routing Vector

Imagine you have a massive unified log file named access.log. Column 1 is the server category (“WEB”, “API”, “DB”). You must split the log into three separate files (web.log, api.log, db.log) in a single pass.

To execute the precision routing sequence, analyze this structural command sequence:

awk '{
    if ($1 == "WEB") print $0 > "web.log";
    else if ($1 == "API") print $0 > "api.log";
    else if ($1 == "DB") print $0 > "db.log";
}' access.log

Analyzing the Routing Calculus

The exact millisecond you execute this script, the awk engine intercepts the payload.

• The engine reads the first row (e.g., WEB GET /index.html 200).
• It hits the first logic gate: Is $1 equal to “WEB”? True.
• It triggers print $0 > "web.log". Instead of routing the entire row to the terminal, the engine intercepts the output and opens a file descriptor to web.log. It writes the full row to that file.
• The engine reads the second row (e.g., API POST /users 201). The first gate is False. The second gate is True. The engine opens a separate file descriptor to api.log and routes the data there.
• Piping to External Commands: You can also route data to external shell commands in real time. For example, print $0 | "sort -k2" would pipe each line directly into the sort command. Or print $2 | "mail -s Alert [email protected]" would blast a specific column directly into an email pipeline.
• The engine iterates through the entire multi-gigabyte file in a single pass, mathematically routing millions of rows to their correct destination files at microscopic speeds.