The Challenge of JSON in the Terminal
In modern cloud and DevOps environments, almost every API, configuration file, and logging system relies on JSON (JavaScript Object Notation). While JSON is highly readable for humans when properly formatted, interacting with it programmatically in a bash script is notoriously difficult.
Traditional Linux text-processing tools like grep, awk, and sed process data line-by-line using regular expressions. Because JSON is a hierarchical, deeply nested data structure where whitespace and line breaks are irrelevant, attempting to parse it with awk or grep leads to incredibly fragile scripts that break the moment the JSON formatting changes.
To solve this, developers created jq. Described as the “sed for JSON data,” jq is a lightweight, command-line JSON processor. It understands the actual structure of the JSON object, allowing you to slice, filter, map, and transform JSON data with absolute precision.
Step 1: Pretty-Printing JSON
The most fundamental use of jq is formatting unreadable, minified JSON into a colorized, human-readable structure.
If you execute a curl command to a REST API, the response is often a massive, single-line string of data. By piping the output into jq with a simple dot . (which represents the root object), you instantly format the output.
curl -s https://api.github.com/users/torvalds | jq '.'The output will be perfectly indented and color-coded, making it immediately readable.
Step 2: Extracting Specific Values
To extract specific keys from a JSON object, you provide the exact path to the key. In jq, you traverse the object using dots.
Given a file server.json:
{
"hostname": "web-prod-01",
"specs": {
"cpu": 8,
"ram": "32GB"
}
}To extract the amount of RAM, you trace the path through the “specs” object:
cat server.json | jq '.specs.ram'The output will be "32GB". Note that jq preserves the quotes because it outputs valid JSON strings. If you want the raw text without quotes (useful when assigning to a bash variable), use the -r (raw) flag:
RAM=$(cat server.json | jq -r '.specs.ram')Step 3: Iterating Over Arrays
JSON frequently returns arrays of objects. jq provides an array iterator [] to unpack the array and process each object individually.
Consider an API response containing a list of users:
{
"users": [
{ "id": 1, "name": "Alice", "admin": true },
{ "id": 2, "name": "Bob", "admin": false }
]
}To print just the names of every user, you first access the array .users, then unpack it with [], and finally select the .name key:
cat response.json | jq '.users[].name'Output:
"Alice"
"Bob"Step 4: Filtering Data with Select
jq is a fully functional programming language that allows you to filter data based on logical conditions using the select() function.
Using the previous array of users, suppose you only want to extract the names of users where "admin" is true. You can pipe the unpacked array into the select function, evaluate the condition, and then extract the name:
cat response.json | jq '.users[] | select(.admin == true) | .name'This powerful chaining syntax operates much like a standard bash pipe |, but operates exclusively on the internal JSON objects.
Step 5: Transforming and Constructing New JSON
Beyond extracting data, jq can build entirely new JSON structures on the fly. This is incredibly useful when reading data from one API and formatting it to POST to a different API.
If you want to read the user array and output a brand new, simplified JSON object containing only the ID and Name, you construct the new object syntax inside jq:
cat response.json | jq '.users[] | { user_id: .id, full_name: .name }'Output:
{
"user_id": 1,
"full_name": "Alice"
}
{
"user_id": 2,
"full_name": "Bob"
}Conclusion
Parsing JSON with traditional regex tools is an anti-pattern that leads to broken scripts and unreliable automation. By integrating jq into your shell scripting workflow, you gain the ability to manipulate, filter, and transform complex JSON payloads with absolute precision, making it a mandatory skill for modern cloud engineers and system administrators.