How to Find and Remove Duplicate Lines Using the uniq Command in Linux

When you dump a massive, chaotic log file or a highly unstructured list of IP addresses from a Linux server, the data is often plagued with identical, repeated entries. Manually scanning thousands of lines to identify and delete duplicates is mathematically inefficient. To force the Linux kernel to algorithmically parse the data matrix and aggressively collapse identical adjacent strings into single lines, you must deploy the uniq command.

Executing the Deduplication Engine

The uniq command is a highly focused text-filtering engine. It ingests a data stream, analyzes it line by line, and if it detects that Line 2 is mathematically identical to Line 1, it completely vaporizes Line 2 from the output stream.

CRITICAL ARCHITECTURAL WARNING: The uniq engine is strictly adjacent. It only compares a line to the line immediately before it or after it. If your file contains duplicates that are separated by other data (e.g., duplicate IP addresses on line 5 and line 500), uniq will completely fail to detect them. You must mathematically sort the file first to force all duplicates into adjacent clusters.

Executing the Global Collapse

Imagine you have a file named ip_list.txt containing chaotic, unsorted data. To execute a flawless deduplication, you must chain the sort and uniq engines together using a pipeline.

Open your terminal and type:

sort ip_list.txt | uniq

The exact millisecond you press Enter, the sort engine reorganizes the entire file alphabetically, clustering all identical lines together. The data is instantly piped into the uniq engine, which rips through the clusters, vaporizes all redundant lines, and outputs a pristine, mathematically unique list to standard output.

Analyzing the Duplicate Matrix

If you do not want to delete the duplicates, but rather forensically audit exactly how many times a specific line was repeated, inject the -c (count) flag into the engine.

sort ip_list.txt | uniq -c

The engine will now output the collapsed list, but it will mathematically prefix every single line with an integer proving exactly how many times that exact string appeared in the original file matrix (e.g., 5 192.168.1.100).