When working with large text files, log archives, or lists of IP addresses in Linux, you will frequently encounter duplicate entries. Manually scanning thousands of lines of text to delete repeated words is impossible.
Instead, you can use the uniq command to automatically filter out duplicate lines and clean up your data instantly.
The Crucial Rule: You Must Sort First
The uniq command has one major limitation: it only detects duplicate lines if they are directly next to each other (adjacent). If “apple” is on line 1, and “apple” is on line 10, the uniq command will not notice the duplicate.
Therefore, you almost always need to pipe your text through the sort command first, which forces all identical lines to group together.
How to Remove Duplicates
Let’s say you have a file named list.txt. To sort the file and remove all duplicates, you pipe the commands together like this:
sort list.txt | uniq
This will print a clean, deduplicated list directly to your terminal. If you want to save that clean list to a brand new file instead of just looking at it on the screen, use the redirect arrow (>):
sort list.txt | uniq > clean_list.txt
How to Only Show Duplicate Lines
Sometimes you don’t want to remove the duplicates; you want to find them. For example, you might want to know if the same error message appeared multiple times in a log file.
To force the command to print only the lines that were duplicated, use the -d (duplicate) flag:
sort list.txt | uniq -d
How to Count the Duplicates
If you want to know exactly how many times each item appeared in your list, use the -c (count) flag. This is incredibly useful for finding the most active IP addresses hitting a web server or identifying the most common errors in a log file.
sort list.txt | uniq -c
This will output the list with a number next to each line indicating how many times it occurred.
Case Sensitivity and uniq
By default, uniq is strictly case-sensitive. It considers “Apple” and “apple” to be completely different words. If you want to ignore case and treat them as duplicates, you must use the -i (ignore case) flag:
sort list.txt | uniq -i