How to Trace Thread Creation and Latency in Linux Using threadsnoop

Modern Linux servers, microservice containers, and multithreaded applications (such as Java JVMs, Go runtimes, Python event loops, and database engines) depend on light-weight POSIX threads (pthreads) to achieve concurrency. While creating threads allows applications to process tasks in parallel across multi-core processors, excessive or unbounded thread creation introduces severe performance penalties. Spawning new threads requires allocating stack memory, initializing task structures (task_struct), and triggering kernel scheduler context switches.

When applications mismanage thread pools or spawn ad-hoc threads inside rapid request loops, systems experience memory fragmentation, thread thrashing, and high thread creation latency. Because ephemeral thread creations occur in sub-millisecond intervals, conventional process sampling tools like ps or top miss them completely. The BPF Compiler Collection (BCC) provides threadsnoop, an eBPF tracing utility designed to intercept and record thread creations via pthread_create() in real time. This guide shows how to trace thread creation latency and identify short-lived threads in Linux using threadsnoop.

How threadsnoop Operates via eBPF

The threadsnoop utility attaches an extended Berkeley Packet Filter (eBPF) uprobe (user-space probe) to the GNU C Library (glibc) function pthread_create(). Its operational flow proceeds as follows:

1. Whenever any user-space process invokes pthread_create(), the uprobe intercepts the invocation.
2. The eBPF program records a high-resolution timestamp, the calling Process ID (PID), the parent thread ID, and the memory address of the thread start routine function.
3. When the thread creation completes, threadsnoop measures the total creation latency in microseconds ($\mu\text{s}$).
4. It outputs a stream of events detailing which processes are spawning threads, how frequently they are spawned, and how long the initialisation took.

Prerequisites and Tool Installation

The utility requires Linux kernel 4.9 or newer with BPF enabled, along with elevated administrative permissions.

Installing on Debian and Ubuntu

1. Update the local APT package index:

sudo apt update

2. Install the BCC tools package and Linux kernel headers:

sudo apt install bpfcc-tools linux-headers-$(uname -r)

The executable is available as threadsnoop-bpfcc under /usr/sbin/.

Installing on RHEL, Fedora, and Rocky Linux

Install the package using dnf:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

The binary is located at /usr/share/bcc/tools/threadsnoop.

Basic Thread Tracing with threadsnoop

To begin monitoring thread creations across the entire system, execute threadsnoop with root privileges:

sudo threadsnoop-bpfcc

Whenever an application initializes a new POSIX thread, threadsnoop prints an event record immediately:

TIME(ms)   PID     COMM             START_FUNC         LAT(us)
0.000      1820    mysqld           0x55d14210a480         42
12.481     3410    java             0x7f10b2401820         85
12.512     3410    java             0x7f10b2401820         78
45.109     8912    python3          0x7fa0e8912040        124
105.812    9102    worker           0x5590c1209b10       1450

Interpreting the Output Columns

• TIME(ms): Elapsed milliseconds since tracing commenced, or a wall-clock timestamp depending on configuration flags.
• PID: The Process ID of the application creating the thread.
• COMM: The executable command name of the parent application.
• START_FUNC: The user-space virtual memory address of the entry function executed by the new thread.
• LAT(us): The thread creation latency in microseconds ($\mu\text{s}$) measured from the entry of pthread_create() to its return.

Advanced Filtering and Diagnostics

On high-throughput application servers, monitoring every thread creation can generate high volumes of trace records. Use command-line arguments to isolate specific applications and investigate latency anomalies.

Filtering by Process ID

To audit thread creation behaviour for a single problematic application (such as an Apache Cassandra node or a JVM container), restrict tracing to its specific PID using the -p parameter:

sudo threadsnoop-bpfcc -p 3410

This allows you to verify whether your application is maintaining a static thread pool or continuously spawning and destroying worker threads under load.

Detecting Thread Creation Latency Spikes

In high-performance microservices, thread creation should ideally complete in under 100µs. If the LAT(us) column consistently displays values exceeding 1,000µs (1 millisecond) or higher, the system is experiencing thread creation latency bottlenecks. Common causes include:

• Memory Pressure and Virtual Memory Limits: The kernel is taking too long to allocate the default thread stack memory (typically 8MB per thread under glibc defaults).
• PID/Task Limit Saturation: The system is approaching kernel.pid_max or systemd TasksMax thresholds.
• Lock Contention: High lock contention inside glibc memory allocation arenas during concurrent thread spawning.

Resolving Virtual Addresses to Function Names

The START_FUNC column displays a memory address. To map that hexadecimal address to an actual function name in your application’s source code, use the addr2line or nm utilities:

addr2line -e /usr/local/bin/worker -f -C 0x5590c1209b10

This outputs the exact C/C++ or Rust source file and line number (e.g. worker_pool.cpp:142) that initiated the thread start routine, enabling rapid code-level remediation.

Best Practices and Thread Pool Optimisation

• Migrating to Static Thread Pools: If threadsnoop reveals an application spawning thousands of threads per minute, refactor the application to use a persistent thread pool (e.g. Java ThreadPoolExecutor or Go goroutine worker pools) to amortise creation latency.
• Adjusting Stack Sizes: The default glibc thread stack size (8MB) wastes address space when thousands of threads are created. Reduce thread stack size using pthread_attr_setstacksize() if threads only execute lightweight tasks.
• Uprobe Overhead Considerations: Because threadsnoop attaches user-space uprobes to glibc, it introduces slightly more overhead than kernel-only kprobes. While completely safe for debugging, avoid running unfiltered system-wide traces on production servers for extended periods.