How to Install and Use ddh in Linux for Fast Duplicate File Detection and Disk Recovery

Accumulating duplicate files across massive storage arrays, development directories, and backup disks wastes valuable NVMe and SSD capacity. While legacy Unix utilities like fdupes and rmlint remain functional, scanning large multi-terabyte filesystems often encounters performance bottlenecks. ddh (Dont-Duplicate-Hacks) is a modern CLI duplicate file finder written in Rust that leverages progressive hashing, multi-threaded filesystem traversal, and an interactive terminal user interface to identify identical files and clean up storage with minimal CPU overhead.

How Progressive Hashing Accelerates Duplicate Detection

Traditional tools often compute complete cryptographic hashes (such as SHA-256) for every file in a directory tree. In contrast, ddh uses a four-stage progressive evaluation pipeline:

1. Filesize Pre-Filtering: Instantly compares file sizes via metadata stat calls. Files with unique byte lengths are excluded immediately without reading file contents.
2. Initial Chunk Hashing: For files sharing identical sizes, ddh reads and hashes only the first 4 KB of data. If the initial headers differ, the files are disregarded.
3. Mid-Point Chunk Hashing: If headers match, intermediate bytes are sampled to weed out similar headers in structured file formats.
4. Full Content Verification: Only files that survive all preliminary stages undergo full block hashing, guaranteeing byte-for-byte fidelity while skipping 95% of unnecessary I/O reads.

Installing ddh on Linux Distributions

You can install ddh across standard Linux distributions using pre-compiled binaries, distribution packages, or Cargo:

Method 1: Install via Cargo (Recommended for All Distributions)

# Ensure Rust and Cargo are installed
sudo apt install cargo || sudo dnf install cargo || sudo pacman -S cargo

# Install ddh globally
cargo install ddh

Method 2: Install via Arch Linux AUR

yay -S ddh-bin

Method 3: Download Pre-Compiled Binary from GitHub Releases

curl -s https://api.github.com/repos/darakian/ddh/releases/latest \
  | grep "browser_download_url.*linux" \
  | cut -d : -f 2,3 \
  | tr -d \" \
  | wget -qi -

tar -xvf ddh*.tar.gz
sudo mv ddh /usr/local/bin/
sudo chmod +x /usr/local/bin/ddh

Running Scans and Analysing Storage Consumption

Basic scans against target directories provide immediate insights into wasted disk capacity:

Command	Operation Description	Best Used For
ddh -d /path/to/scan	Scans a specific directory recursively and outputs duplicate groups.	Standard home folder and document audits.
ddh -d /dir1 -d /dir2	Compares two distinct directories against each other.	Verifying external backup redundancy.
ddh -d ~/Projects -m size	Sorts duplicate clusters strictly by total consumed byte size.	Prioritising large video and dataset cleanups.
ddh -d ~/Downloads -v	Enables verbose output detailing scanning speeds and file metrics.	Performance benchmarking and troubleshooting.

Interactive Deduplication and File Deletion

To safely delete redundant duplicates while keeping one original copy, run ddh with interactive mode:

ddh -d ~/Storage --interactive

When interactive mode launches, ddh steps through each duplicate cluster and provides immediate prompt actions:

• [0..N]: Specify the numeric index of the file you wish to retain as the original master copy.
• all: Keep all instances in this cluster without deleting anything.
• delete: Remove selected secondary duplicate files immediately from the filesystem.
• hardlink: Replace secondary duplicates with hardlinks pointing to the original inode, preserving path references while freeing physical storage blocks.

Exporting Duplicate Lists to JSON for Automation

For headless servers and automated system maintenance cron scripts, export the scan output as structured JSON:

ddh -d /var/data -f json -o ~/duplicates_report.json

You can then inspect the output using jq to extract total wasted bytes or target specific file extensions:

jq '.[] | {size: .size, count: (.paths | length), files: .paths}' ~/duplicates_report.json

Safety Best Practices for System Folders

When running duplicate finders across Linux installations, observe these operational safeguards:

• Avoid System Root Directories: Never execute broad recursive scans against /usr, /etc, or /var/lib. Operating systems often maintain identical binary libraries or configuration skeletons across multiple software packages intentionally.
• Exclude Version Control Directories: Always exclude .git folders from scans, as Git’s internal object store naturally retains identical blob hashes across commit trees.