How to Visualise System Scheduling and CPU States in Linux Using perf timechart

Diagnosing erratic latency spikes, multithreaded synchronization bottlenecks, and power-management overhead in Linux systems requires an understanding of how processes interleave across physical CPU cores over time. While command-line profilers like top, htop, or basic perf stat provide aggregated numerical averages, they obscure transient microsecond-scale anomalies—such as lock convoys, core migrations, prolonged CPU sleep states (C-states), and runqueue delays. Understanding these phenomena requires temporal visualization.

The Linux performance analysis suite includes perf timechart, a specialized forensic subsystem designed to capture kernel scheduler tracepoints and processor power events, compiling them into high-resolution, interactive Scalable Vector Graphics (SVG) timelines. System engineers can open these SVG visualisations in any modern web browser or vector editor to inspect CPU execution bars, task scheduling latency, polling loops, and CPU frequency transitions across every hardware core with nanosecond precision.

Understanding perf timechart Visual Architecture

The perf timechart tool parses low-level kernel trace events into distinct, coordinated graphical tracks:

• CPU Power and Frequency Tracks: Rendered along the top of the chart, these tracks display processor C-state transitions (idle states such as C1, C2, C3) and P-state dynamic frequency scaling (DVFS) changes, illustrating whether cores were throttled or sleeping when a workload stalled.
• Per-Process Execution Bars: Each running process and thread receives a horizontal lane. Blue solid bars denote active CPU execution, while grey segments indicate blocked or waiting states.
• Scheduler Latency Indicators: Yellow and red highlight bands represent scheduling delay—the duration a thread spent in the runnable state waiting for a physical CPU core to become free.
• Polling Loops: Highlighted in distinct orange cross-hatching, showing when applications waste CPU cycles spinning on mutexes rather than yielding via kernel futexes.

Prerequisites and Kernel Tracing Support

Capturing timecharts requires administrative root privileges and the Linux perf userspace package matching your running kernel version.

Install the required performance utilities and verify tracepoint availability:

# Install Linux performance tools
sudo apt update
sudo apt install -y linux-tools-common linux-tools-$(uname -r)

# Verify power and scheduler tracepoints
sudo perf list "power:*" "sched:*"

Confirm that critical kernel events—including power:cpu_idle, power:cpu_frequency, sched:sched_switch, and sched:sched_wakeup—are enabled and accessible in your kernel build.

Recording System Workloads with perf timechart record

The perf timechart record command attaches to kernel tracepoints and writes timestamped execution records to a local trace file.

To record system-wide scheduler and power activity for 5 seconds during a benchmark run:

# Record system-wide events for 5 seconds
sudo perf timechart record -- sleep 5

To trace a specific compute-heavy application alongside its background worker threads:

# Profile a specific application during execution
sudo perf timechart record -- ./data-processing-engine --threads=8

To record CPU frequency and idle states alongside scheduler events, append the -P (power events) flag:

# Capture full power states and scheduler metrics simultaneously
sudo perf timechart record -P -- sleep 10

The capture commits all events to a binary file named perf.data in the current working directory.

Generating the Scalable Vector Graphics Timeline

Once recording completes, convert the binary trace file into an SVG timeline using the perf timechart parser:

# Generate standard output.svg vector chart
sudo perf timechart

The utility parses the event stream and writes output.svg. On multi-core servers with hundreds of active processes, the resulting timeline can be vast. You can filter and customize chart generation to focus on specific latency thresholds:

# Only render processes that ran for at least 1 millisecond
sudo perf timechart --runtime 1.0

# Highlight tasks experiencing scheduling delays greater than 2 milliseconds
sudo perf timechart --highlight "postgres"

The --highlight argument paints matching task bars in contrasting colors, making it easy to track primary worker threads amidst noisy background system daemons.

Analysing the Generated SVG Timeline in a Web Browser

Transfer output.svg to your local desktop workstation or open it directly in a modern web browser:

# Example: serve via Python HTTP server for remote inspection
python3 -m http.server 8080

When reviewing the interactive timeline, inspect three critical operational zones:

• Gap Analysis: Look for unexpected blank gaps in process execution lanes. Gaps between thread runs indicate blocking I/O calls, page faults, or lock contention.
• Runqueue Latency Bands: Note where yellow scheduler delay bars precede blue execution blocks. Wide delay bands confirm that physical cores were oversubscribed, forcing threads to wait for CPU time.
• C-State Wakeup Penalties: Correlate the moment a process wakes up with the CPU power track. If the core was residing in a deep idle state (e.g. C6/C7), waking the physical core requires tens of microseconds, delaying instruction execution.

Tuning Systems Based on Timechart Findings

Use the visual insights from perf timechart to eliminate scheduling bottlenecks:

• Disable Deep C-States: If latency-critical threads suffer from idle wake-up penalties, set processor.max_cstate=1 or use cpupower idle-set -d <state> to lock cores into low-latency operating states.
• Pin Worker Threads: If threads bounce across cores between execution bursts, bind processes to dedicated NUMA nodes using numactl --cpunodebind=0 --membind=0 to preserve hardware L3 cache locality.
• Tune CFS Latency: Adjust sysctl kernel.sched_migration_cost_ns to prevent the scheduler from aggressively migrating threads between cores.

By transforming abstract trace logs into intuitive visual timelines, perf timechart enables Linux engineers to rapidly resolve complex multi-core scheduling contention and optimize workload responsiveness.