On high-core multi-socket servers running database engines, high-frequency network proxies, or virtualised cloud instances, CPU utilisation alone rarely exposes architectural scalability bottlenecks. When multiple threads simultaneously attempt to acquire kernel synchronization primitives—such as spinlocks, read-write locks, or mutexes—threads spend valuable CPU cycles spinning or sleeping in wait queues. The lockstat tool, included in the BPF Compiler Collection (BCC), instruments kernel locking routines using eBPF probes, measuring lock hold times, contention frequency, and caller call stacks without requiring kernel recompilations.
Understanding Kernel Synchronization Primitives and Contention Overhead
The Linux kernel employs distinct locking primitives depending on execution context and latency sensitivity:
- Spinlocks (
spinlock_t): Used primarily in interrupt contexts and short critical sections where sleeping is prohibited. If another core holds the lock, the waiting CPU core executes busy-wait loops (“spins”) at 100% CPU utilisation until the lock is released. High spinlock contention wastes immense processor capacity. - Mutexes (
struct mutex): Used in process contexts for longer operations. When contended, threads attempt an optimistic spin; if unsuccessful, they relinquish the core and enter sleep states until awakened by the lock holder, incurring context-switch latency. - Read-Write Semaphores (
rw_semaphore): Allow concurrent shared read access while enforcing exclusive locks for write modifications.
When multiple CPU cores contend for shared data structures—such as socket tables, memory dentry caches, or filesystem inode locks—throughput plateaus or degrades as core counts increase. lockstat identifies which specific kernel locks are saturated and pinpoint which kernel call paths are causing the bottlenecks.
Installing BCC Tools on Linux Servers
Ensure that BCC packages and kernel development headers corresponding to your active kernel version are installed:
# On Debian and Ubuntu
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)
# On Enterprise Linux (RHEL, Rocky Linux, Fedora)
sudo dnf install bcc-tools kernel-devel-$(uname -r)
Following installation, the utility is available as /usr/sbin/lockstat-bpfcc or /usr/share/bcc/tools/lockstat.
Running lockstat to Profile Lock Contention
Running lockstat with administrative privileges monitors lock acquisition events and outputs a ranked statistical table upon receiving an interrupt signal (Ctrl+C) or when a duration timer expires:
sudo /usr/sbin/lockstat-bpfcc 5
This command profiles kernel locking for 5 seconds and summarises the most contended locks:
Tracing lock events... Hit Ctrl-C to end.
Lock: 0xffff8881048b0a00
Caller: inet_csk_accept+0x142/0x290
Type: spinlock
Acquired: 18402
Contended: 1240
Total Wait (us): 15420
Max Wait (us): 84
Lock: 0xffff88812c40a180
Caller: d_alloc_parallel+0x68/0x420
Type: mutex
Acquired: 8401
Contended: 412
Total Wait (us): 48910
Max Wait (us): 240
Interpreting the output columns:
- Caller: The kernel function offset where lock acquisition was initiated.
- Acquired: Total number of times the lock was successfully obtained.
- Contended: Number of times the calling thread could not obtain the lock immediately and was forced to wait or spin.
- Total Wait (us): Cumulative time in microseconds spent waiting for the lock across all profiling threads.
- Max Wait (us): The worst-case single wait latency spike recorded during the sampling window.
Advanced Profiling: Filtering by Lock Type and Call Stacks
To avoid information overload on large multi-core architectures, apply targeted command-line filters:
1. Isolating Mutex Contention
To restrict tracing exclusively to sleeping mutexes while filtering out transient spinlocks, use the -t flag:
sudo /usr/sbin/lockstat-bpfcc -t m 10
2. Isolating Spinlock Saturation
To focus specifically on CPU-wasting spinlocks where cores burn clock cycles in busy-wait loops, specify spinlock profiling:
sudo /usr/sbin/lockstat-bpfcc -t s 10
3. Capturing Full Kernel Call Stacks
To identify the entire execution path leading to a contended lock rather than inspecting only the immediate caller, append the -s flag to record full kernel call frames:
sudo /usr/sbin/lockstat-bpfcc -s 5 5
This outputs the complete stack trace for the top 5 most contended locks, enabling kernel developers and systems engineers to diagnose whether locking contention is originating from filesystem cache updates, network socket backlog locks, or memory allocation contention.