How to Install and Use Miller (mlr) in Linux for Fast CSV, TSV, and JSON Processing

Modern data pipelines, system logging facilities, and backend diagnostics generate vast quantities of tabular and structured data formatted as CSV, TSV, JSON, and JSON Lines. While traditional Unix utilities like awk, sed, cut, and sort are legendary for unstructured text manipulation, they are inherently field-positional rather than header-aware. If a CSV file adds or reorders a column, your shell scripts break instantly. Conversely, heavy data science environments like Python’s Pandas require significant runtime overhead for quick command-line audits. Miller (mlr) bridges this gap. It acts like awk, sed, cut, join, and sort for name-indexed tabular data, allowing you to slice, dice, transform, and convert data effortlessly from the Linux terminal.

What Is Miller and Why Use It?

Miller is an open-source, multi-format data processing utility written in Go (and formerly C). Unlike positional text processors, Miller understands structured data formats natively:

• Header and Key-Aware: References columns and keys by name (e.g., $status, $latency_ms) rather than positional indices (e.g., $4, $7).
• Multi-Format Translation: Reads and writes seamlessly across CSV, TSV, JSON, JSON Lines (JSONL), key-value pairs (DKVP), and pretty-printed ASCII tables.
• Rich Domain-Specific Language (DSL): Provides built-in mathematical, statistical, string manipulation, and datetime functions.
• Streaming Architecture: Processes multi-gigabyte files record-by-record with a minimal memory footprint.

Installing Miller on Linux

Miller is available in the official repositories of nearly all major Linux distributions, as well as via pre-compiled binaries.

Method 1: Installing via APT (Ubuntu and Debian)

sudo apt update
sudo apt install miller -y

Method 2: Installing via DNF or Pacman (Fedora and Arch Linux)

# Fedora / RHEL
sudo dnf install miller -y

# Arch Linux
sudo pacman -S miller

Method 3: Downloading Pre-Built Binaries from GitHub

MLR_VER=$(curl -s "https://api.github.com/repos/johnkerl/miller/releases/latest" | grep -Po '"tag_name": "v\K[^"]*')
curl -Lo mlr.tar.gz "https://github.com/johnkerl/miller/releases/download/v${MLR_VER}/miller-${MLR_VER}-linux-amd64.tar.gz"
tar -xzf mlr.tar.gz
sudo mv miller-${MLR_VER}-linux-amd64/mlr /usr/local/bin/
rm -rf mlr.tar.gz miller-${MLR_VER}-linux-amd64

Confirm that the binary is available in your shell:

mlr --version

Core Syntax and Verbs in Miller

The standard execution pattern for Miller is:

mlr --[format-flags] verb [verb-options] input_file

Common format flags include --csv (standard comma-separated), --tsv (tab-separated), --json (standard JSON), and --jsonl (JSON Lines).

Essential Everyday Transformations

1. Pretty-Printing and Inspecting Tabular Data

Convert an unreadable, wide CSV file into a beautiful ASCII table directly in your terminal:

mlr --icsv --opprint cat server_metrics.csv | head -n 15

2. Filtering Rows Based on Conditional Criteria

Filter records dynamically based on column values without fragile regular expressions:

# Select all transactions where response time exceeds 500ms and status is not 200
mlr --csv filter '$response_time > 500 && $status != 200' access_logs.csv

3. Selecting and Reordering Columns (cut)

# Retain only specific fields by name
mlr --csv cut -f timestamp,client_ip,status access_logs.csv

# Exclude sensitive columns
mlr --csv cut -x -f internal_token,ssn customer_data.csv

4. Converting Between Formats (CSV to JSON)

Transform formats instantly without writing Python or jq scripts:

# Convert CSV to pretty JSON
mlr --icsv --ojson cat users.csv > users.json

# Convert JSON Lines to CSV
mlr --ijsonl --ocsv cat telemetry.jsonl > telemetry.csv

Advanced Operations: Grouping, Aggregation, and Calculations

Computing Summary Statistics

Calculate statistical aggregates (mean, minimum, maximum, count) grouped by category:

mlr --csv stats1 -a count,mean,min,max -f latency_ms -g region datacenter_pings.csv

Deriving New Columns with Math Expressions (put)

Use the put verb to compute derived columns dynamically:

# Convert milliseconds to seconds and flag slow queries
mlr --csv put '$duration_sec = $duration_ms / 1000; $is_slow = ($duration_sec > 2.0)' queries.csv

Joining Two Files on a Shared Key

Perform relational database-style inner or left joins across separate files directly in the terminal:

mlr --csv join -j user_id -f accounts.csv orders.csv

Adding Miller to your Linux command-line arsenal eliminates the brittle friction of positional parsing, bringing reliable, header-aware data transformation straight to your everyday shell pipelines.