How to Trace System Call Latency and Outliers in Linux Using syscount

System calls (syscalls) represent the critical bridge between user space applications and kernel-level hardware operations, handling file I/O, process management, memory allocation, and networking. When applications experience unexplained slowdowns or CPU spikes, aggregate CPU metrics often fail to reveal which specific kernel operations are responsible. The syscount tool, part of the BPF Compiler Collection (BCC), provides comprehensive syscall profiling to identify call frequency, execution latency, and slow outlier transactions in real time.

Understanding How syscount Profiles Kernel Syscalls

The syscount utility leverages extended Berkeley Packet Filters (eBPF) to attach low-overhead probes to the kernel’s raw_syscalls:sys_enter and raw_syscalls:sys_exit tracepoints. By recording precise hardware timestamps on entry and exit, syscount calculates the exact elapsed time spent executing each individual syscall inside the kernel, allowing administrators to quantify both high-frequency calls and rare, high-latency blocking calls.

Installing the Required Tracing Packages

To utilize syscount, ensure the BCC toolkit and kernel headers are installed:

# On Debian or Ubuntu:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

# On RHEL, AlmaLinux, or Fedora:
sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Debian/Ubuntu environments, the command is accessible as syscount-bpfcc, while on RHEL distributions it resides under /usr/share/bcc/tools/syscount.

Executing syscount to Measure System-Wide Latency

To inspect total execution time rather than simple invocation counts, run syscount with the latency flag (-L):

sudo syscount-bpfcc -L -d 10

This command monitors system activity for 10 seconds (-d 10) and outputs a sorted breakdown ordered by cumulative latency:

SYSCALL                   COUNT         TIME (uS)
epoll_wait                14250         12450820
futex                     8910          3451020
read                      45120         1820410
write                     32140         945120
nanosleep                 1200          600150

Targeting Specific Processes and Detecting Outliers

When profiling a specific problematic service (such as a database or web server), target its Process ID (PID) to isolate application-specific overhead:

# Profile PID 2489 for 15 seconds, sorting by cumulative execution time:
sudo syscount-bpfcc -p 2489 -L -d 15

To capture slow outlier syscalls exceeding a defined execution duration, combine the filter with a latency threshold:

# Flag syscalls taking longer than 10 milliseconds (10,000 microseconds):
sudo syscount-bpfcc -p 2489 -L -m 10000

Interpreting Results for Performance Optimisation

Analysing the output yields clear engineering directives:

• Excessive read/write Latency: Indicates disk storage bottlenecks, saturated file system queues, or synchronous I/O blocks that could benefit from memory caching or asynchronous I/O (io_uring).
• High futex Duration: Signifies intense lock contention across multithreaded worker pools, pointing to synchronization bottlenecks within the application code.
• Frequent epoll_wait Delays: Typically normal for idle event-driven services waiting on incoming client connections.

Using syscount provides quantitative kernel-level telemetry, empowering systems engineers to pinpoint bottlenecks and optimise Linux application performance with surgical precision.