How to Remove Duplicate Lines Using the uniq Command in Linux

When you dump raw data from an unformatted database (like a massive server log or a chaotic email list) into a Linux text file, it is often riddled with thousands of duplicate entries. If you try to manually scroll through a 50,000-line file looking for duplicate IP addresses, you are wasting massive amounts of time. To force the Linux kernel to mathematically analyze the file structure and violently strip out every single duplicate line, you must use the uniq command.

How the uniq Command Works

The uniq command is a highly specific filtering engine. It rips through a text file, mathematically comparing every single line to the exact line directly beneath it. If line 2 is a byte-for-byte identical clone of line 1, the engine violently destroys line 2, outputting only a single, perfectly unique instance of the data.

CRITICAL WARNING: The uniq engine is highly flawed by design: it only detects adjacent duplicates. If line 1 says “Server_A” and line 5 says “Server_A”, the engine will completely ignore the duplicate because they are not touching each other. To guarantee a perfect purge, you must mathematically force the file into alphabetical order using the sort command before you feed it into uniq.

Assume you have a chaotic file named raw_ips.txt.

sort raw_ips.txt | uniq

The pipe (|) forces the raw data through the sort engine first, placing every identical IP address directly next to each other. The data is then slammed into the uniq engine, which effortlessly annihilates every single duplicate, outputting a mathematically pristine list to your terminal screen.

Advanced Data Reporting

The uniq command does not just delete data; it can generate highly detailed forensic reports about the duplicates it finds.

If you want to know exactly how many times a specific IP address hammered your server, you can inject the -c (count) flag.

sort raw_ips.txt | uniq -c

The engine will output a perfectly clean table, prepending a strict numerical count directly to the front of each line:

   1 192.168.1.15
  45 192.168.1.50
   2 192.168.1.99

This instantly tells you that the IP ending in .50 is aggressively spamming your network, allowing you to instantly deploy a firewall block.

If you want to execute an absolute purge and only see the lines that are 100% unique (meaning they did not have a single duplicate in the entire file), use the -u flag:

sort raw_ips.txt | uniq -u