How to Use GNU Datamash in Linux to Perform Command-Line Statistical and Numerical Aggregations

In Linux system administration and data engineering, log files, system metrics, and server reports are frequently structured as tabular text streams (such as whitespace, comma, or tab-delimited values). While traditional UNIX utilities like awk, cut, and sort are exceptional at slicing and filtering rows, performing mathematical and statistical aggregations—such as calculating group-by sums, medians, quartiles, or standard deviations—requires complex, multi-line awk scripts that are difficult to write and maintain.

GNU Datamash is an official GNU command-line utility built specifically to bridge this gap. It allows developers to perform quick numerical, textual, and statistical operations directly on piped streams without writing Python or R scripts. This guide demonstrates how to install, configure, and use GNU Datamash for advanced command-line data processing.

Installing GNU Datamash

GNU Datamash is available in the official package repositories of all major Linux distributions:

On Ubuntu / Debian

sudo apt update && sudo apt install datamash -y

On Fedora / RHEL

sudo dnf install datamash -y

On Arch Linux

sudo pacman -S datamash

Core Statistical Operations

1. Calculating Basic Descriptive Statistics

To calculate the sum, arithmetic mean, minimum, and maximum of a single column of numbers (e.g. memory consumption values):

cat memory_usage.txt | datamash sum 1 mean 1 min 1 max 1

2. Computing Medians and Percentiles

Unlike awk, which requires sorting and array indexing in memory to compute medians, Datamash computes order statistics natively:

cat response_times_ms.txt | datamash median 1 q1 1 q3 1 sstdev 1

This command outputs the median, first quartile (25th percentile), third quartile (75th percentile), and sample standard deviation in a single clean pass.

3. Performing Group-By Aggregations

Suppose you have a file named web_traffic.tsv where Column 1 is the HTTP status code and Column 2 is the response payload size in bytes. To calculate the total bandwidth and average payload size per status code:

# Note: input must be sorted by the grouping column
sort -k1,1 web_traffic.tsv | datamash -g 1 sum 2 mean 2 count 2

The -g 1 flag instructs Datamash to group records by Column 1, producing aggregated metrics per unique HTTP code.

Supported Operations Reference

Category	Operations	Description
Basic Numeric	sum, min, max, absmin, absmax	Arithmetic totals and bounding values
Central Tendency	mean, median, mode	Averages and most common occurrences
Dispersion	sstdev, pstdev, svar, pvar, mad	Sample/population standard deviations and variances
Order Statistics	q1, q3, iqr, percentile:N	Quartiles, interquartile ranges, arbitrary percentiles
Textual Operations	count, unique, collapse, countunique	String grouping, comma-separated collapsing, counts