Standard Linux shell scripting loops—such as for file in *.log; do ... done or xargs pipelines—execute tasks sequentially on a single central processing unit (CPU) thread. On modern multi-core workstations and cloud servers equipped with 8, 16, or 64 CPU cores, sequential execution leaves 90% or more of available computational power idle while batch operations, image conversions, and database compressions take hours to complete.
GNU Parallel is a high-performance shell utility designed to execute command-line jobs concurrently across multi-core systems and remote server clusters. By replacing traditional loops with parallel workers, GNU Parallel maximizes CPU utilisation, manages job queues automatically, and ensures standard output streams remain uncorrupted.
Installing GNU Parallel Across Linux Distributions
GNU Parallel is packaged in official repositories across all major distributions:
- On Ubuntu, Debian, and Linux Mint:
sudo apt update && sudo apt install -y parallel
- On Fedora, RHEL, and Rocky Linux:
sudo dnf install -y parallel
- On Arch Linux and Manjaro:
sudo pacman -S parallel
Basic Syntax and Job Execution
The basic command structure of GNU Parallel accepts input items from standard input or arguments separated by ::::
parallel [options] command ::: arguments
The token {} represents the replacement placeholder for each input argument:
- To compress a list of individual log files concurrently using
gzip:
parallel gzip ::: /var/log/app/*.log
- To convert a directory of high-resolution PNG images to compressed WebP format using
cwebp:
parallel cwebp -q 80 {} -o {.}.webp ::: *.png
The {.} modifier strips the file extension, generating output filenames such as photo.webp from photo.png.
Controlling Concurrency and CPU Core Allocation
By default, GNU Parallel spawns one job worker per available CPU core. You can manually adjust concurrency using the -j (jobs) flag:
- To limit execution to 4 concurrent processes:
parallel -j 4 ffmpeg -i {} -c:v libx265 -crf 28 {.}.mp4 ::: *.mov
- To utilize 200% of available CPU cores (ideal for I/O-bound tasks waiting on disk or network sockets):
parallel -j 200% curl -O -s {} :::: download_urls.txt
- To leave one CPU core free for interactive system tasks on a busy server:
parallel -j -1 bzip2 ::: *.raw
Displaying Progress Indicators and ETA
When running long-running batch jobs involving hundreds or thousands of files, you can monitor execution velocity directly:
- Add the
--progressflag to display an interactive progress counter:
parallel --progress convert {} -resize 1200x {.}_thumb.jpg ::: *.jpg
The console displays live completion percentages, total elapsed time, average job runtime, and estimated completion time.
- To display a visual text progress bar, use
--bar:
parallel --bar md5sum {} ::: /large/dataset/*
Preventing Output Interleaving with Output Grouping
When multiple concurrent processes write to standard output or standard error simultaneously, conventional terminal pipelines produce garbled, overlapping text. GNU Parallel solves this by capturing each job’s output in isolated buffers, printing results sequentially only after each individual job completes.
If you require streaming output without waiting for job completion, pass the -u (ungroup) flag, or use -k (keep order) to guarantee that output arrives in the exact order arguments were specified regardless of which worker finishes first:
parallel -k ping -c 1 {} ::: 1.1.1.1 8.8.8.8 9.9.9.9
Processing Records from Pipe Streams and CSV Files
GNU Parallel easily integrates with standard shell tools like find and awk:
- Find and process files containing special characters or spaces using null-delimited inputs:
find /data -type f -name "*.tar" -print0 | parallel -0 tar -xf {}
- Parse multi-column CSV datasets:
parallel --colsep ',' process_user.sh {1} {2} {3} :::: users.csv
Here, {1}, {2}, and {3} map directly to columns in the CSV data stream, enabling powerful automated ETL workflows directly from the Linux command line.