How to Filter Repeated Lines Using the uniq Command in Linux

When you combine multiple massive server logs or append data from several CSV files together, you almost always generate duplicate lines of data. If you are preparing this data for ingestion into a database or statistical analysis, duplicate records will heavily skew your results. While the sort command can organize the data, you need a dedicated tool to actually strip out the redundant information. In Linux, the uniq command is designed exclusively to filter out and omit repeated lines of text.

How the uniq Command Works

The uniq command analyzes a text file line by line. When it finds a line that is absolutely identical to the line immediately preceding it, it filters the duplicate out and only prints a single instance of the line to the standard output.

Crucial Requirement: The uniq command only compares adjacent lines. If duplicate lines are spread randomly throughout a document (e.g., duplicate data on line 5 and line 500), uniq will completely ignore them. Therefore, you must almost always run the sort command first, piping the mathematically ordered output directly into uniq.

Basic Duplicate Filtering

To completely strip a file of all duplicate lines, combine sort and uniq using a bash pipe (|).

sort raw_emails.txt | uniq

The sort command organizes every email address alphabetically, forcing all identical emails to sit directly next to each other. The uniq command then scans the sorted list, collapsing all the adjacent duplicates into a single line. The resulting clean list is printed to your terminal. To save this clean list, redirect the output to a new file:

sort raw_emails.txt | uniq > clean_emails.txt

Finding Only the Duplicates

Sometimes you don’t want to clean the file; you want to actively investigate the duplication to find out which users are spamming a system or which errors are recurring. You can instruct uniq to suppress all the unique, normal lines and only print the lines that are repeated. You do this using the -d (duplicate) flag.

sort application_log.txt | uniq -d

The output will show you a clean list containing only the specific strings of text that appeared more than once in the original document.

Counting Occurrences

If you need to know exactly how many times a specific duplicate occurred, you can use the -c (count) flag. This is incredibly powerful for basic log analysis.

sort access_log.txt | uniq -c

The output will prefix every single unique line with a numerical count indicating exactly how many times it appeared in the original sorted file.

    4 192.168.1.50
    1 192.168.1.66
   12 192.168.1.104