How to Use the mpstat and pidstat Commands in Linux to Diagnose Per-Core CPU Saturation and Thread Latency

System administrators and DevOps engineers frequently encounter situations where overall server CPU utilisation appears moderate, yet mission-critical applications suffer severe response-time latency. Traditional tools such as uptime or global CPU metrics in top obscure granular hardware saturation by calculating broad cluster-wide averages. A server with 64 cores displaying 12.5% total CPU load might actually have eight threads completely pinning eight specific cores at 100%, causing fatal queueing delays for upstream requests.

To pinpoint per-core imbalances, storage wait bottlenecks, and thread-level contention, Linux provides the sysstat performance monitoring suite. Among its most potent utilities are mpstat (multiprocessor statistics) and pidstat (per-process and per-thread statistics). This guide provides a detailed operational breakdown of utilising mpstat and pidstat to isolate processor saturation and thread latency in Linux production environments.

---

Architectural Mechanics: Kernel CPU Accounting

Linux tracks CPU time slices through low-level kernel counters maintained in virtual file systems:

• /proc/stat: Aggregates system-wide processor ticks across user mode, system mode, hardware/software interrupts, and idle loops since system boot. mpstat samples these counters over a specified time interval to compute differential percentage rates.
• /proc/[pid]/stat: Tracks execution time slices and page faults incurred by specific process identifiers.
• /proc/[pid]/task/[tid]/stat: Tracks granular scheduler metrics for individual POSIX threads running beneath a parent process. pidstat reads this directory structure to identify thread-level starvation and context switching.

Tool	Primary Data Source	Granularity	Primary Diagnostic Role
mpstat	/proc/stat	Per-Core / Per-Socket / Global	Identifies uneven core saturation, interrupt thrashing, and hypervisor steal time.
pidstat	/proc/[pid]/*	Per-Process / Per-Thread	Isolates thread-level CPU usage, I/O wait latency, and involuntary context switching.

---

Step 1: Installing and Verifying the sysstat Suite

The mpstat and pidstat binaries are packaged together within the sysstat package across major enterprise Linux distributions:

# On Ubuntu / Debian systems:
sudo apt update && sudo apt install -y sysstat

# On RHEL / AlmaLinux / Rocky Linux / Fedora:
sudo dnf install -y sysstat

# Verify installation and active versions:
mpstat -V
pidstat -V

---

Step 2: Diagnosing Per-Core Imbalances with mpstat

Running mpstat without arguments displays CPU metrics averaged since boot, which is generally useless for active incident triage. Always pass an interval (in seconds) and a sample count.

Monitoring All Cores Individually

# Sample all CPU cores every 2 seconds for 5 iterations
mpstat -P ALL 2 5

Interpreting the Output Columns:

Metric Column	Performance Description	Threshold & Operational Signal
%usr	Percentage of CPU time consumed by un-niced user-space applications.	Sustained high values indicate application-level compute tasks or query loops.
%nice	CPU time running user-space processes with positive nice values.	Reflects low-priority background batch processing.
%sys	Percentage of CPU time spent executing kernel system calls and drivers.	High values (>20%) point to excessive I/O syscalls, memory allocation locks, or context overhead.
%iowait	CPU time idle while waiting for an outstanding disk or network storage I/O request.	High values (>10%) indicate that storage throughput or latency is throttling compute threads.
%irq	CPU time servicing hardware interrupts.	Elevated rates indicate massive packet arrival on network interface cards without hardware offload.
%soft	CPU time servicing software interrupts (network softirqs, timer loops).	Persistent values on specific cores suggest network receive packet steering (RPS) imbalance.
%steal	CPU time involuntarily lost by a virtualised guest while the hypervisor serviced another VM.	Values >2% indicate noisy-neighbour contention on cloud hypervisors (AWS EC2, Azure, GCP).
%idle	Percentage of CPU time spent without running tasks.	Values near 0% confirm hardware saturation on that specific core.

Diagnosing Hardware Interrupt Affinity

To verify if high interrupt traffic is overwhelming specific physical cores:

# Display hardware and software interrupt statistics per CPU
mpstat -I CPU 2 3

If core 0 or 1 shows thousands of HI/s (hardware interrupts per second) while remaining cores show zero, network interface card (NIC) interrupts are pinning a single core. This bottleneck requires configuring smp_affinity or enabling the irqbalance daemon.

---

Step 3: Tracking Process and Thread Latency with pidstat

Once mpstat reveals which cores are saturated, use pidstat to correlate thread behaviour with that consumption.

1. Identifying Top CPU Consuming Processes

# Sample process CPU usage every 2 seconds for 3 intervals, sorting active tasks
pidstat 2 3

2. Breaking Down Thread-Level Latency (-t Flag)

Multi-threaded engines (such as Java JVMs, PostgreSQL, or Nginx worker pools) distribute load across dozens of threads. Inspecting only the parent process masks individual rogue threads:

# Inspect all threads for a specific process (e.g., PID 4092)
pidstat -t -p 4092 2 5

The output introduces the TID (Thread Identifier) column alongside the parent PID. If a single TID sustains 100% %CPU while sibling threads sit idle, an application thread has locked into an infinite loop or regular expression backtracking fault.

3. Analysing Context Switching and Thread Preemption (-w Flag)

Context switches incur heavy latency penalties by flushing CPU L1/L2 caches and translation lookaside buffers (TLB). pidstat categorises switches into two critical types:

# Monitor context switches per process every 2 seconds
pidstat -w 2 5

Switch Type	Metric Column	Underlying Cause	Remediation Target
Voluntary	cswch/s	The thread willingly surrendered the CPU because it blocked on I/O, a lock, or a sleep call.	Normal behaviour during network polling or file reads; excessive rates reflect lock contention.
Involuntary	nvcswch/s	The Linux kernel scheduler forcibly preempted the thread because its allocated time slice expired.	High values indicate severe CPU starvation; there are more runnable threads than physical cores available.

4. Correlating I/O Delays at the Process Level (-d Flag)

When mpstat displays elevated %iowait, identify which process is saturating the disk pipeline:

# Display per-process read/write transfer rates and cancelled write bytes
pidstat -d 2 5

• kB_rd/s: Kilobytes per second read from disk.
• kB_wr/s: Kilobytes per second written to storage controllers.
• iodelay: Number of clock ticks spent by the process waiting for synchronous disk I/O to complete.

---

Integrated Triage Workflow: Combining mpstat and pidstat

During an active production latency incident, execute this structured diagnostics pipeline:

# 1. Step: Check per-core distribution for 3 seconds
mpstat -P ALL 1 3

# 2. Step: If a specific core (e.g. CPU 3) is pinned, locate top threads running on it
pidstat -t 1 3 | awk '$9 == "3" || NR<4'

# 3. Step: Check if the application is suffering from thread starvation or lock spinning
pidstat -w -p <TARGET_PID> 1 5

# 4. Step: Check if I/O latency is dragging down the thread
pidstat -d -p <TARGET_PID> 1 5

---

Troubleshooting Common CPU Latency Faults

Observed Phenomenon	Root Cause	Diagnostic & Remediation Action
Single core at 100% %usr; all other cores 95% idle	Single-threaded application architecture or single hotspot lock in multi-threaded software	Execute pidstat -t -p <PID> to isolate the thread ID. Profile the thread stack using perf top -t <TID> or generate a thread dump for analysis.
High %sys (>30%) across all cores with low %usr	Excessive kernel syscall overhead, frequent file descriptor polling, or extreme page allocation churn	Run perf top to identify kernel routines. Trace syscall frequency using strace -c -p <PID> to determine if the process is hammering futex, epoll_wait, or read.
Surge in nvcswch/s on application threads	CPU over-subscription; thread pool configured with significantly more active workers than physical cores	Tune application thread pool size (e.g., Tomcat, Gunicorn, Puma) to match physical core counts (typically Cores * 1.5 or Cores * 2 for I/O bound workloads).
High %soft on CPU 0 under heavy network traffic	Network driver interrupts not distributed across CPU cores (single-queue NIC)	Enable Receive Packet Steering (RPS) via /sys/class/net/<interface>/queues/rx-0/rps_cpus or configure multi-queue RSS via ethtool -L <interface> combined <N>.
Unexplained latency spikes with elevated %steal	Cloud provider hypervisor over-committing physical CPUs to multiple virtual machines	Audit steal time trends. If %steal consistently exceeds 3%, migrate the workload to dedicated host instances or contact cloud support.

---

Operational Best Practices for Linux Performance Engineering

• Do Not Rely Exclusively on Global Load Averages: A load average of 4.0 on a 16-core system appears safe, but if all four tasks are non-interruptible disk waits (D-state) on the same database volume, latency will degrade exponentially. Cross-reference load with mpstat -P ALL.
• Automate Historical sysstat Archival: Ensure the background sysstat collector service is enabled (systemctl enable --now sysstat). This automatically records 10-minute historical intervals into /var/log/sysstat/saDD, enabling post-incident analysis with sar.
• Isolate Latency-Sensitive Tasks via CPU Pinning: For real-time services, assign critical threads to dedicated cores using taskset -c 2,3 <command> or systemd CPU affinity directives (CPUAffinity=2,3) to prevent cache thrashing.