How to Remove Duplicate Lines Using the uniq Command in Linux

When you merge two massive server logs or combine multiple email subscriber lists into a single Linux text file, you will inevitably generate thousands of redundant, duplicate lines. Keeping these duplicates artificially inflates your file size and can completely break automated scripts that expect clean, unique data arrays. You cannot manually hunt down identical lines in a 50,000-line document. To force the Linux kernel to mathematically identify and permanently delete redundant data, you must use the uniq command.

The Sorting Prerequisite

The uniq command is incredibly powerful, but it has one critical, unbending architectural flaw: it does not scan the entire document at once. It only compares a line of text to the exact line sitting immediately above it and the exact line sitting immediately below it. If it sees two identical lines stacked directly on top of each other, it deletes one. If the identical lines are separated by even a single different line, the engine will completely miss the duplicate.

Therefore, you must perfectly sort your file before the uniq command can function properly.

You can achieve this by combining the sort command with the uniq command using a standard bash pipe (|):

sort master_list.txt | uniq

This combined command instantly reorganizes every line in the file alphabetically (forcing all identical lines to stack directly on top of each other) and then immediately passes that sorted data into the uniq engine, which rips out all the redundant clones.

Generating Unique and Duplicate Reports

By default, uniq simply outputs a clean, deduped list. However, you can use specialized flags to force the engine to act as a highly specific diagnostic tool.

To isolate ONLY the duplicate data:
If you are auditing a database and want to see exactly which email addresses were submitted multiple times, you want to ignore the clean data entirely. Append the -d (duplicate) flag.

sort master_list.txt | uniq -d

This forces the engine to strip away every single unique line, outputting a strict list containing only the lines that were duplicated.

To count the exact number of duplications:
If you want to know exactly how many times a specific error code appeared in a log, append the -c (count) flag.

sort server_log.txt | uniq -c

This command outputs the clean, deduped list, but it mathematically prepends a hard numerical count to the beginning of every single line (e.g., 5 CRITICAL_ERROR_404), instantly showing you exactly how many redundant clones were deleted.