How to Trace TCP Round-Trip Time Latency Histograms in Linux Using tcprtt

Kernel-Level TCP Round-Trip Time Analysis with eBPF

Diagnosing network latency issues in production Linux environments often requires measuring socket-level round-trip times (RTT) rather than relying on synthetic Internet Control Message Protocol (ICMP) ping probes. ICMP echoes test the path between two hosts, but they do not traverse the kernel Transmission Control Protocol (TCP) stack, do not experience congestion control window adjustments, and bypass application-level socket queues. The tcprtt utility, provided within the BPF Compiler Collection (BCC), attaches extended Berkeley Packet Filter (eBPF) probes directly to kernel TCP tracepoints to capture actual round-trip times experienced by established connections.

Instead of logging individual packet traces that overwhelm system buffers, tcprtt aggregates round-trip time measurements into log2 or linear histograms directly inside kernel BPF memory maps. This enables systems engineers to analyse microsecond-level latency distributions, identify network jitter outliers, and evaluate TCP performance on high-throughput database servers, web proxies, and microservices without noticeable CPU overhead.

Under the hood, tcprtt hooks into kernel functions such as tcp_rcv_established() and reads the TCP socket control block (struct tcp_sock). By sampling the smoothed round-trip time (srtt_us) or the instantaneous RTT calculated when acknowledging segments, it produces high-fidelity latency telemetry across any local or remote port combination.

Installing the BPF Compiler Collection on Linux

Because tcprtt relies on eBPF bytecode compilation and kernel tracepoints, the host system must run a modern Linux kernel (version 4.9 or later, with 5.x/6.x recommended) compiled with CONFIG_BPF=y and CONFIG_BPF_SYSCALL=y. Additionally, the corresponding kernel headers and BCC runtime packages must be installed.

On Debian and Ubuntu systems, install the toolset using the APT package manager:

# Update package lists and install BCC tools along with Linux headers
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On Red Hat Enterprise Linux, Rocky Linux, or Fedora installations, install BCC tools via DNF:

# Install BCC tools and kernel developer headers on RHEL/Fedora
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

Once installed, binaries are accessible under /usr/sbin/ or /usr/share/bcc/tools/ depending on distribution packaging. On Debian and Ubuntu distributions, the binary is appended with the -bpfcc suffix (e.g. tcprtt-bpfcc). To standardise command execution, establish a symlink or append BCC paths to your environment:

# Create convenient symlink if using Debian/Ubuntu packages
sudo ln -s /usr/sbin/tcprtt-bpfcc /usr/local/bin/tcprtt

Tracing Global TCP Round-Trip Time Distributions

Executing tcprtt without filtering arguments captures round-trip time samples across all active TCP sockets traversing the host network namespace. When terminated with Ctrl+C, the tool prints an ASCII histogram categorising latency brackets in microseconds:

# Trace global TCP RTT distribution across all local sockets
sudo tcprtt

The resulting terminal output displays the distribution of observed round-trip times:

Tracing TCP RTT... Hit Ctrl-C to end.
^C
     msecs               : count     distribution
         0 -> 1          : 842      |****************************************|
         2 -> 3          : 215      |**********                              |
         4 -> 7          : 47       |**                                      |
         8 -> 15         : 12       |                                        |
        16 -> 31         : 3        |                                        |
        32 -> 63         : 0        |                                        |

By default, tcprtt displays intervals in milliseconds. To inspect ultra-low-latency local or intra-datacenter connections where sub-millisecond granularity is necessary, pass the -u flag to render the histogram in microseconds:

# Display round-trip times with microsecond resolution
sudo tcprtt -u

Reviewing the histogram shape immediately indicates latency stability. A tight distribution concentrated in the lowest buckets signifies low jitter and optimal link performance, whereas a bimodal distribution or long tail extending into higher buckets reveals packet retransmissions, bufferbloat, or routing delays.

Filtering by Local and Remote Ports

In multi-tenant or multi-service environments, tracing all TCP traffic generates noisy histograms that obscure specific application performance. tcprtt provides targeted filtering flags to isolate inbound listener ports or outbound client connections.

To measure round-trip times for an inbound HTTP/HTTPS reverse proxy or web server listening on port 443:

# Trace RTT exclusively for local port 443 (inbound HTTPS traffic)
sudo tcprtt -l 443

Conversely, to evaluate latency when communicating with an external database cluster (such as PostgreSQL running on port 5432):

# Trace RTT for outbound connections targeting remote port 5432
sudo tcprtt -r 5432

You can combine port filtering with microsecond resolution to monitor backend API microservices:

# Measure microsecond RTT for remote Redis connections on port 6379
sudo tcprtt -u -r 6379

Filtering by IP Address and Subnet

When troubleshooting network routing discrepancies between specific data centres or cloud zones, restrict tracing to a single remote IP address using the -a parameter:

# Trace TCP latency to a specific remote peer IP address
sudo tcprtt -a 192.168.10.45

To inspect communication across an entire Classless Inter-Domain Routing (CIDR) subnet, combine the IP parameter with CIDR bitmask syntax:

# Trace TCP latency across a specific internal subnet
sudo tcprtt -a 10.240.0.0/16

This capability allows infrastructure engineers to verify whether cross-region Virtual Private Cloud (VPC) peerings or Software-Defined Networks (SDN) introduce asymmetric latency penalties for specific worker nodes.

Periodic Sampling and Automated Monitoring Intervals

Instead of capturing a single aggregate histogram over the entire process lifetime, tcprtt can emit periodic snapshots at defined time intervals. Supplying an interval in seconds and an optional count argument configures automated sampling:

# Output RTT histogram snapshots every 5 seconds for 6 iterations
sudo tcprtt 5 6

To capture time-series latency shifts during simulated traffic spikes or benchmark tests, pass the -T flag to prefix each histogram with a high-precision wall-clock timestamp:

# Include timestamps with 10-second periodic RTT reports
sudo tcprtt -T 10

This timestamped format facilitates correlating latency spikes with background system events, such as storage backups, garbage collection cycles, or scheduled cron jobs.

Isolating Latency Outliers with Custom Bucket Sizing

While power-of-two (log2) histograms provide a wide dynamic range, they group broader time intervals into higher buckets. When fine-grained analysis of high-latency tails is required, tcprtt allows switching to linear histograms or customising bucket ranges.

Pass the -i flag to specify linear bucket intervals in milliseconds or microseconds:

# Generate linear histogram with 5-millisecond bucket steps
sudo tcprtt -i 5

To record round-trip times only for sockets whose RTT exceeds a specific threshold (e.g. filtering out normal fast packets and only recording connections exceeding 50 milliseconds):

# Trace only connections with RTT exceeding 50ms (50000 microseconds)
sudo tcprtt -u -m 50000

By filtering fast traffic out of the data collection pipeline, engineers can pinpoint intermittent degradation events, diagnose TCP window stalls, and optimise kernel networking parameters without sifting through millions of standard packet traces.