When you need to compress massive log files or database dumps on a Linux server, traditional tools like gzip or bzip2 prioritize file size over speed. If you are compressing a 50GB file, gzip can take hours, completely saturating your CPU and stalling your backup scripts. If your priority is absolute, blistering speed—processing data at over 500 MB/s per core—you must abandon traditional tools and use the lz4 command.
Understanding the LZ4 Algorithm
LZ4 is a lossless data compression algorithm heavily used by enterprise systems, the Linux kernel, and massive data platforms like Hadoop. It does not compress files as tightly as gzip (meaning the final file will be slightly larger), but its compression and decompression speeds are orders of magnitude faster. It is designed to compress data at RAM speed, making it the perfect tool for streaming backups or compressing real-time log files.
How to Compress a File Using lz4
Most modern Linux distributions include the LZ4 package in their primary repositories. If the command fails, you can install it via sudo apt install lz4 (Ubuntu) or sudo dnf install lz4 (Fedora).
To compress a massive database dump file, run:
lz4 database_dump.sql database_dump.sql.lz4Unlike gzip, which deletes the original file by default, lz4 safely preserves your original database_dump.sql file and generates a completely new, compressed .lz4 copy alongside it. This non-destructive default behavior is incredibly useful for testing backup scripts.
If you absolutely need a slightly smaller file size and are willing to sacrifice a few seconds of speed, you can increase the compression ratio (from level 1 up to level 9). For maximum compression, run:
lz4 -9 database_dump.sql database_dump.sql.lz4How to Decompress (Extract) an LZ4 File
The decompression speed of LZ4 is its most impressive feature, often exceeding several gigabytes per second, meaning it extracts files practically instantaneously.
To extract your compressed file back to its original state, you must pass the -d (decompress) flag.
lz4 -d database_dump.sql.lz4 extracted_dump.sqlAgain, this is non-destructive. It reads the .lz4 file, builds the extracted_dump.sql file, and leaves the compressed archive intact. If you want the tool to delete the archive after extraction (mimicking gzip‘s behavior), you must append the --rm flag.