The Challenge of Backups
The standard rsync command is the definitive tool for backing up files in Linux. A basic rsync -av command seamlessly mirrors a source directory to a destination. However, if you run this command every day into a new folder to keep daily snapshots (e.g., a Monday folder, a Tuesday folder), you will quickly run out of disk space.
If your web server has 50GB of data, and only 10MB of text files change on Tuesday, a standard backup will copy the entire 50GB again, wasting massive amounts of storage. This is where the brilliant –link-dest flag comes in. It allows rsync to create “incremental” backups that look like full backups but take up almost zero extra disk space.
How –link-dest Works
When you use the --link-dest flag, you point rsync to yesterday’s backup folder. As rsync copies today’s files to today’s new folder, it compares every file against yesterday’s folder.
- If a file has changed since yesterday,
rsynccopies the new file to today’s folder. - If a file has NOT changed since yesterday,
rsyncdoes not copy the data. Instead, it creates a “hard link” in today’s folder pointing to the physical data located in yesterday’s folder.
To the operating system and the user, today’s folder looks like a complete, standalone 50GB backup. You can navigate through it and open any file. But physically on the disk, it only consumes the 10MB of space required for the files that actually changed.
Step 1: The Initial Baseline Backup
Before you can use --link-dest, you must have an initial baseline backup to compare against.
Assume your source data is in /var/www/html/ and your backup drive is mounted at /backup/.
Run a standard rsync to create the “Monday” baseline:
rsync -av /var/www/html/ /backup/monday/
This will take a long time and consume the full 50GB of space.
Step 2: Creating the Incremental Hard-Link Backup
It is now Tuesday. You want to create a Tuesday backup, but you want to hard-link against the Monday backup to save space.
The syntax requires you to specify the --link-dest path relative to the new destination (or you can just use absolute paths to avoid confusion).
Run the following command:
rsync -av --link-dest=/backup/monday/ /var/www/html/ /backup/tuesday/
Breaking down the command:
- -av: Archive mode (preserves permissions and dates) and verbose output.
- –link-dest=/backup/monday/: Tells rsync to look inside the monday folder. If a file in the source matches a file in here, create a hard link instead of copying data.
- /var/www/html/: The live source data.
- /backup/tuesday/: The brand-new destination folder for today’s backup.
This command will execute incredibly fast. If you run du -sh /backup/tuesday/, the system will report it as 50GB. However, if you run df -h to check the actual physical drive usage, you will see that almost no new space was consumed.
Step 3: Automating the Process
To make this practical, you need to automate the folder naming using date variables in a bash script. A standard daily backup script looks like this:
#!/bin/bash
SOURCE="/var/www/html/"
BACKUP_DIR="/backup"
TODAY=$(date +%Y-%m-%d)
YESTERDAY=$(date -d "yesterday" +%Y-%m-%d)
rsync -av --link-dest=$BACKUP_DIR/$YESTERDAY/ $SOURCE $BACKUP_DIR/$TODAY/
By scheduling this script in cron, you will generate a new folder every single day containing a complete snapshot of your data, but your hard drive will only fill up at the rate of your daily changes.