How to Break Large Files into Smaller Chunks Using the split Command in Linux

When you attempt to transfer a massive 50GB database dump or a giant log file over a slow SFTP connection to a remote Linux server, the process is incredibly fragile. If your network connection drops after 49GB, you lose everything and must start the entire hours-long transfer completely over from scratch. To protect against network drops and bypass strict file size upload limits, you should use the Linux split command to break massive files into smaller, easily manageable chunks.

How to Split a File by Size

The split utility does exactly what its name implies: it reads a large input file and physically slices it into multiple smaller output files based on the parameters you define. It does not compress or alter the data; it simply cuts it.

To break a massive 10GB file (named database.sql) into smaller, 1GB chunks, open your terminal and run:

split -b 1G database.sql chunk_

Let’s break down this syntax:

• -b 1G: This flag tells the command to slice the file into exact 1-Gigabyte byte chunks. You can also use M for Megabytes (e.g., -b 500M).
• database.sql: The massive input file you want to break apart.
• chunk_: This is the prefix for your newly created files.

When you run this command, Linux will instantly generate files named chunk_aa, chunk_ab, chunk_ac, and so on, until the entire 10GB file has been processed. You can now transfer these ten 1GB files individually. If the network drops while transferring chunk_ad, you only have to re-upload that single 1GB piece, saving you massive amounts of time.

How to Split a File by Line Count

If you are dealing with a massive CSV file containing millions of customer records, splitting by raw gigabytes might cut a customer’s name in half at the exact byte boundary, corrupting the database. For text files, you must split by line count, ensuring that every record remains perfectly intact.

To split a massive CSV file so that each smaller file contains exactly 10,000 lines, use the -l (lines) flag:

split -l 10000 massive_customers.csv customer_batch_

This guarantees that no text strings or database rows are severed in the middle of a word.

How to Reassemble the Chunks

Once you have successfully transferred all the chunk_ files to your remote destination server, you must stitch them back together into the massive original file. You do this using the standard cat (concatenate) command, utilizing a wildcard (*) to grab all the chunks in alphabetical order.

cat chunk_* > database_restored.sql

The cat command reads the files in perfect alphabetical sequence (aa, ab, ac) and streams them into the new database_restored.sql file, resulting in a bit-for-bit perfect replica of your original massive file.