The Limitations of the cp and scp Commands
When managing Linux servers, transferring files from one location to another is a daily task. The standard cp (copy) command is fine for local files, and scp (secure copy) is traditionally used for transferring files between a local machine and a remote server over SSH.
However, scp has a massive flaw when dealing with large directories or backups: it is completely “dumb”. If an scp transfer of a 50GB directory fails halfway through, restarting the command forces it to overwrite everything and start from the beginning. It cannot detect which files are already present on the destination server.
This is where rsync (remote sync) becomes indispensable. rsync is a powerful, highly efficient utility that compares the source and destination directories. It only transfers the specific files—or even the specific blocks within a file—that have changed. This drastically reduces bandwidth usage and transfer times, making it the industry standard for Linux server backups and file synchronization.
Basic rsync Syntax
The basic structure of the rsync command is identical to cp and scp:
rsync [options] [source] [destination]
The source and destination can be local directories, or they can be remote paths formatted as user@host:/path/to/directory.
Essential rsync Options
To use rsync effectively, you must understand its flags. The most common combination used by system administrators is -avz.
- -a (archive): This is a macro flag that preserves permissions, ownership, timestamps, and symbolic links. It also ensures the transfer is recursive (copies all subdirectories).
- -v (verbose): Outputs the names of the files being transferred to the terminal so you can monitor progress.
- -z (compress): Compresses file data during the transfer. This reduces network bandwidth but increases CPU usage on both servers. It is excellent for slow connections but can actually slow down transfers over gigabit LAN networks.
- –delete: (Use with extreme caution) This tells rsync to delete files in the destination directory that no longer exist in the source directory, creating an exact mirror.
- -P: Combines
--progress(shows a progress bar for each file) and--partial(allows resuming of interrupted transfers).
How to Synchronize Files Locally
Before attempting a remote transfer, it is good practice to understand local synchronization.
To back up a directory named website_data to a mounted backup drive at /mnt/backups, use:
rsync -av /var/www/website_data /mnt/backups/
The Importance of the Trailing Slash
In rsync, a trailing slash on the source directory is critical.
/var/www/website_data/(with slash): Copies the contents of the directory into the destination./var/www/website_data(without slash): Copies the directory itself into the destination, creating/mnt/backups/website_data.
How to Synchronize Files Between Remote Servers
To use rsync over a network, it relies on SSH for secure data transport. Therefore, you must have SSH access to the remote server.
Pushing Files to a Remote Server
To push your local website_data directory to a remote backup server (IP: 192.168.1.100), run:
rsync -avz /var/www/website_data [email protected]:/backups/
You will be prompted for the admin user’s SSH password. If you have configured SSH key authentication, the transfer will begin immediately without a password prompt.
Pulling Files from a Remote Server
If you are logged into your local machine and want to pull a database backup from the remote production server, simply reverse the source and destination:
rsync -avzP [email protected]:/var/backups/db_dump.sql /local/path/
By adding the -P flag here, you will see a detailed progress bar for the large SQL file, and if your connection drops, running the exact same command again will resume the download from where it left off.
How to Test with a Dry Run
Because rsync is capable of overwriting and deleting files (especially if you use the --delete flag or misunderstand the trailing slash rule), making a typo can be catastrophic.
Always use the –dry-run (or -n) flag when running a complex command for the first time. This simulates the transfer without actually copying or deleting any data.
rsync -avz --delete --dry-run /source/ /destination/
Review the verbose output carefully. If it shows the correct files being copied and deleted, you can remove the --dry-run flag and execute the command for real.