How to Trace and Audit System Calls in Real Time in Linux Using syscount-bpfcc

System calls (syscalls) form the fundamental interface between user-space applications and the Linux kernel. Whenever software reads from a socket, allocates memory, or accesses storage, it transitions into kernel space. Diagnosing performance degradation often requires understanding which syscalls dominate execution time. While classical diagnostic tools such as strace capture this data, they intercept every process instruction using ptrace, causing catastrophic performance overhead (frequently 10x to 100x slowdowns). The BPF Compiler Collection (BCC) provides syscount-bpfcc, an eBPF-powered utility that aggregates syscall frequencies and latencies directly within kernel space with negligible performance impact.

Why eBPF Makes syscount Superior to strace

Classical tracing mechanisms rely on context-switching between user and kernel boundaries for every single system call. Under heavy I/O or web server workloads handling tens of thousands of requests per second, running strace can cause production services to stall or crash.

In contrast, syscount-bpfcc attaches low-overhead tracepoints to kernel syscall handlers (sys_enter and sys_exit):

• In-Kernel Summarisation: Counters and latency histograms are updated directly in efficient BPF hash maps inside kernel space.
• Zero User-Space Overhead: No data is copied to user space until tracing concludes or the display interval elapses.
• Production Safe: Safely deployed on high-concurrency database servers, container hosts, and critical network nodes.

Installing the BCC Tools Suite on Major Linux Distributions

To use syscount-bpfcc, your system requires kernel headers and the BCC utilities package.

On Debian and Ubuntu

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

Note: On Debian/Ubuntu systems, BCC commands append the -bpfcc suffix to avoid namespace collisions (e.g., syscount-bpfcc).

On Fedora, RHEL, and AlmaLinux

sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Red Hat enterprise distributions, tools reside in /usr/share/bcc/tools/syscount or are symlinked to /usr/sbin/syscount.

Executing Real-Time System Call Audits

Because eBPF interacts directly with core kernel data structures, syscount commands must be executed with root privileges.

1. Measuring System-Wide Syscall Frequencies

To gather a global overview of kernel interactions across all running processes, execute syscount-bpfcc without flags and allow it to run for several seconds before terminating with Ctrl + C:

sudo syscount-bpfcc

The resulting output tabulates total invocations per syscall:

Tracing syscalls, printing top 10... Ctrl+C to quit.
^C
SYSCALL             COUNT
futex               84521
epoll_wait          42109
read                31204
write               18450
nanosleep            9320
close                4112
openat               4098
poll                 3211
recvmsg              2450
sendmsg              1890

2. Profiling Syscall Latency Instead of Raw Count

High call frequency does not always indicate a bottleneck; a lightweight call running 100,000 times may consume less CPU time than five blocking disk operations. Use the -L flag to sort results by cumulative latency:

sudo syscount-bpfcc -L

This displays cumulative nanoseconds or milliseconds spent inside kernel routines, immediately revealing whether threads are bottlenecked by nanosleep, epoll_wait, or blocking fsync writes.

3. Isolating a Specific Process by PID

To profile a particular daemon—such as an Nginx worker, Redis instance, or PostgreSQL backend—filter by Process ID using the -p parameter:

sudo syscount-bpfcc -p 18452 -i 5

The -i 5 argument outputs updated summary tables every 5 seconds without stopping the tracer, providing continuous monitoring during load testing.

4. Counting Syscalls Performed by Specific Binaries

If you want to monitor all instances of a program regardless of how many individual child processes it spawns, filter by process name using -c:

sudo syscount-bpfcc -c mysqld

5. Filtering Down to a Specific Syscall

When investigating file descriptor exhaustion or network socket leaks, target individual syscalls using the -e argument:

sudo syscount-bpfcc -e openat -p 4021

Interpreting Results to Resolve Production Bottlenecks

Analysing syscount-bpfcc output enables immediate architectural insights:

• Excessive openat / close cycles: Software is repeatedly opening files without caching handles; implement application connection pooling or persistent file streams.
• Disproportionate futex latency: Threads are experiencing severe lock contention in multithreaded applications; review synchronisation primitives and thread worker pools.
• High read / write counts with low bytes: Applications are issuing unbuffered I/O requests; wrap streams in user-space buffers (such as Java BufferedInputStream or Python io.BufferedReader).

Leveraging syscount-bpfcc ensures precise, non-intrusive visibility into kernel workloads without impacting system stability.