The Concept of Named Pipes
In Linux, the standard “pipe” character (|) is used constantly by administrators to chain commands together. For example, ls -l | grep "txt" takes the output of the first command and instantly feeds it as the input to the second command. This happens entirely in the system’s RAM.
However, an anonymous standard pipe has limitations: it only works in a single straight line, and the commands must be run simultaneously in the same terminal session.
What if you want a background script to generate data, and a completely separate application running in a different terminal to read that data, without writing a massive, slow temporary text file to the hard drive?
The solution is a Named Pipe, also known as a FIFO (First In, First Out). A named pipe looks and acts exactly like a file on your hard drive, but it is actually a direct channel in memory. When Process A writes to the “file”, it blocks until Process B reads from the “file”. The data passes directly between the two programs without ever being written to the physical disk.
Step 1: Creating the Named Pipe
To create a named pipe, you use the mkfifo command (Make FIFO).
Open a terminal and run:
mkfifo /tmp/my_data_pipe
If you run ls -l /tmp/my_data_pipe, you will notice something unique in the output:
prw-rw-r-- 1 user group 0 Oct 26 10:00 /tmp/my_data_pipe
The very first character of the permissions block is a p. This stands for “pipe” and indicates to the Linux kernel that this is not a regular text file.
Step 2: Writing to the Pipe (Blocking)
Let’s see how the pipe behaves. In your current terminal (Terminal 1), try to write some data into the pipe using the echo command:
echo "Hello from Terminal 1" > /tmp/my_data_pipe
When you press Enter, you will notice that your terminal appears to hang. It does not return you to the prompt. This is the expected behavior. Process A (the echo command) is attempting to pour data into the pipe, but because it is a FIFO queue in memory, it will wait patiently until a reader (Process B) attaches to the other end to receive the data.
Step 3: Reading from the Pipe
Open a second, completely separate terminal window (Terminal 2).
In Terminal 2, read the contents of the pipe using the cat command:
cat /tmp/my_data_pipe
Instantly, two things happen:
- In Terminal 2, the text “Hello from Terminal 1” appears on the screen.
- In Terminal 1, the
echocommand finishes, and your command prompt returns.
The data was successfully transferred directly between the two independent processes.
Step 4: Practical Applications
While echoing text is a simple example, mkfifo is incredibly powerful for systems engineering.
- Database Backups: Instead of dumping a massive 100GB SQL database to disk and then compressing it, you can dump the database directly into a named pipe. A compression tool (like
gzip) running in the background can read from that pipe and write the final compressed file to disk. This bypasses the need for 100GB of free temporary disk space. - Log Aggregation: Multiple scripts can write their status updates into a single named pipe, while a master logging daemon reads from the pipe and formats the output.
Step 5: Deleting the Pipe
Because a named pipe exists in the file system hierarchy, you remove it exactly the same way you would a normal file, using the rm command.
rm /tmp/my_data_pipe
This removes the access point, ensuring no other processes can attempt to use it.