How to Run Commands in Parallel Across Multiple CPU Cores in Linux Using GNU Parallel

In standard Bash and POSIX shell scripts, processing thousands of files using a traditional for or while loop executes operations sequentially on a single CPU thread. On multi-core servers or modern developer workstations equipped with 8, 16, or more processing cores, serial execution leaves the vast majority of your hardware compute power completely idle. While utilities like xargs -P provide basic multiprocessing, they struggle with complex argument quoting, command substitutions, progress monitoring, and grouped output. GNU Parallel solves these limitations, allowing you to parallelise command-line workflows across all available CPU cores seamlessly.

Installing GNU Parallel on Linux

GNU Parallel is packaged across all major Linux distributions and can be installed via your system package manager:

• Ubuntu, Debian, and Linux Mint:

  sudo apt update && sudo apt install -y parallel

• RHEL, AlmaLinux, Rocky Linux, and Fedora:

  sudo dnf install -y parallel

• Arch Linux:

  sudo pacman -S parallel

Verify successful installation by checking the utility version:

parallel --version

Core Syntax and Replacement Strings

The basic syntax of GNU Parallel follows an intuitive pattern:

parallel [options] [command] ::: [arguments]

GNU Parallel provides special substitution tokens to manipulate input filenames effortlessly:

• {}: The full input argument (e.g. photos/holiday.jpg).
• {.}: The input argument with the file extension removed (e.g. photos/holiday).
• {/}: The basename of the input argument with leading directory paths stripped (e.g. holiday.jpg).
• {/.}: The basename without path or extension (e.g. holiday).
• {//}: The directory component of the input path (e.g. photos).

Example 1: Parallelising Image Conversion and Compression

Suppose you have hundreds of high-resolution JPEG images that need to be converted to modern WebP format using ImageMagick or cwebp. Running a serial loop might take 10 minutes, whereas GNU Parallel accomplishes it in seconds:

parallel -j+0 cwebp -q 80 {} -o {.}.webp ::: *.jpg

Explanation:

• -j+0: Instructs Parallel to spawn exactly as many concurrent worker jobs as you have physical CPU cores (the default behaviour).
• {}: Inserts each .jpg filename.
• {.}.webp: Replaces the .jpg extension with .webp for the output target.
• ::: *.jpg: Supplies the input list via shell globbing.

Example 2: Compressing Log Files with Piped Input

You can pipe input directly from other utilities such as find, handling filenames that contain spaces safely:

find /var/log/app -type f -name "*.log" | parallel -j 4 gzip -9 {}

In this example, -j 4 restricts execution to 4 simultaneous processes, preventing background log compression from consuming 100% of CPU cycles on a live production server.

Example 3: Tracking Real-Time Progress and ETA

When running extensive multi-hour data processing tasks across millions of records, lack of feedback can make it difficult to determine whether a script has stalled. Adding the --progress or --bar flag renders an interactive progress bar showing elapsed time, percentage complete, and estimated time of arrival:

parallel --bar ffmpeg -i {} -c:v libx265 -crf 28 {.}.mp4 ::: *.mkv

Preventing Output Mixing and Race Conditions

When multiple processes write to standard output (stdout) or standard error (stderr) simultaneously, terminal output can interleave, corrupting logs. GNU Parallel automatically buffers output for each job in memory, printing the complete output of each process atomically only after it terminates. If your tasks must execute in strict input sequence, pass the -k (keep order) switch.

Frequently Asked Questions

What is the difference between GNU Parallel and xargs -P?

While xargs -P can spawn concurrent child processes, it merges outputs randomly as lines arrive, cannot calculate progress bars, lacks path substitution variables like {.}, and struggles with arguments containing spaces or quotation marks unless paired with -0. GNU Parallel handles complex quoting, output buffering, and distribution natively.

Can GNU Parallel distribute jobs across multiple remote servers via SSH?

Yes. By specifying the -S [server1,server2] parameter, GNU Parallel can transfer input files, execute commands remotely over SSH connections, and retrieve completed output back to your local terminal transparently.

How do I silence the GNU Parallel citation notice?

Upon first launch, GNU Parallel requests academic citations. You can permanently silence this notice by executing: parallel --citation and typing will cite.