Over months of software development, multi-terabyte data transfers, automated downloads, and regular system backups, hard drives and solid-state drives steadily accumulate duplicate files. Redundant media assets, mirrored code repositories, and duplicate archives waste precious storage space and slow down file system indexing. While tools such as fdupes have historically assisted with finding duplicate data, scanning large file systems with millions of files often becomes unacceptably slow.
To overcome performance bottlenecks in duplicate detection, jdupes was developed as an intensely optimised, multi-threaded C fork of fdupes. Written specifically to maximise I/O throughput and minimise processor overhead, jdupes can scan storage pools up to 10 to 50 times faster. Furthermore, jdupes supports cross-filesystem hardlinking, file deduplication, and modern copy-on-write (CoW) block cloning on filesystems like Btrfs and XFS. This tutorial explains how to install jdupes, scan for redundant files, and safely clean up duplicate data in the Linux terminal.
Installing jdupes on Linux
The jdupes package is readily available across all major Linux distribution repositories. Verify whether jdupes is installed on your workstation or server:
jdupes --version
If it is not present, install it via your distribution’s package manager:
On Ubuntu, Debian, and Linux Mint:
sudo apt update
sudo apt install jdupes
On Fedora, RHEL, and CentOS Stream:
sudo dnf install jdupes
On Arch Linux and Manjaro:
sudo pacman -S jdupes
How jdupes Accurately Identifies Duplicate Files
Rather than calculating cryptographic hashes across every file immediately (which strains disk read queues), jdupes employs a tiered matching pipeline:
- File Size Comparison: Files with differing byte sizes are discarded from comparison instantly.
- Partial Hash Checking: Candidate files of identical size have their first few disk blocks hashed.
- Full Cryptographic Hashing: Full checksums are generated only for files whose preliminary block hashes match.
- Byte-for-Byte Verification: Confirmed candidates undergo direct byte comparison to eliminate any theoretical hash collision risk before action is taken.
Scanning a Directory Recursively for Duplicates
To scan a directory and all of its sub-directories without deleting anything, pass the recursive flag (-r or -R):
jdupes -r /path/to/directory
jdupes will output matched groups of duplicate files separated by blank lines:
/path/to/directory/project/archive.tar.gz
/path/to/directory/backup/archive_copy.tar.gz
Displaying Storage Savings Summary
If you want to evaluate how much storage space you stand to reclaim before taking action, append the -m (summary) flag:
jdupes -r -m /path/to/directory
At the end of the scan, jdupes displays the total duplicate file count and exact recoverable gigabytes.
Interactive Deletion of Duplicate Files
To manually review each set of duplicates and select which copies to preserve or destroy, invoke the -d (delete) flag:
jdupes -r -d /path/to/directory
For each duplicate group, jdupes prompts you with numbered choices:
[1] /path/to/directory/report_v1.pdf
[2] /path/to/directory/old/report_v1.pdf
Set [1 - 2] to preserve (or 'all'):
Type 1 to keep the first file and delete the second, or type all to skip deletion for that set.
Reclaiming Space Without Deleting (Hardlinking & Cloning)
If multiple applications or directory trees require identical files to remain accessible at their respective paths, deleting files breaks links. Instead, jdupes allows you to replace redundant duplicate copies with filesystem hard links using the -L flag:
jdupes -r -L /path/to/directory
This links duplicate directory entries to the exact same physical inode on disk, instantly reclaiming wasted storage while keeping both paths completely functional for your scripts and applications.