How to Trace TCP Socket TCP_INFO tcpi_busy_time in Linux Using eBPF

In high-performance Linux network performance profiling, measuring how long an active connection actually spends transmitting data versus sitting idle is critical for evaluating connection efficiency. A long-lived database connection or streaming gRPC channel may remain open for hours, yet only actively transmit packets during brief bursts. Coarse metrics like connection age or total bytes transferred cannot differentiate between a connection that was saturated for 10 seconds versus one that trickle-fed data over an hour.

The Linux kernel tracks the cumulative microseconds that a TCP socket had unacknowledged data in flight inside struct tcp_info as tcpi_busy_time. Tracing tcpi_busy_time using extended Berkeley Packet Filter (eBPF) and bpftrace allows systems engineers to calculate true socket duty cycles and detect underutilized or stalled connections in real time.

Understanding tcpi_busy_time in the Linux Kernel

Inside the Linux networking subsystem (defined in include/uapi/linux/tcp.h), struct tcp_info provides granular time counters:

struct tcp_info {
    /* ... state fields ... */
    __u64   tcpi_busy_time;      /* Time (us) spent with packets in flight */
    __u64   tcpi_rwnd_limited;   /* Time (us) stalled by receive window */
    __u64   tcpi_sndbuf_limited; /* Time (us) stalled by send buffer */
    __u32   tcpi_data_segs_out;
};

The tcpi_busy_time counter increments whenever tp->packets_out > 0. By comparing tcpi_busy_time against total socket lifetime (now - start_time), engineers calculate the Socket Duty Cycle (SDC):

Socket Duty Cycle = (tcpi_busy_time / total_lifetime_us) * 100%

A duty cycle near 100% indicates continuous saturating transmission, whereas a low duty cycle (<5%) signals an idle connection pool or infrequent RPC polling.

Authoring the bpftrace Socket Duty Cycle Probe

The following bpftrace script probes tcp_close() to generate socket efficiency reports summarizing busy time upon connection termination:

#!/usr/bin/env bpftrace

#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_close
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;
    $tp = (struct tcp_sock *)$sk;

    $daddr = ntop($inet->inet_daddr);
    $saddr = ntop($inet->inet_saddr);
    $dport = bswap($inet->inet_dport);
    $busy_ms = $tp->busy_time / 1000;
    $rwnd_lim_ms = $tp->rwnd_limited / 1000;

    if ($busy_ms > 100) {
        time("%H:%M:%S ");
        printf("CLOSED %s -> %s:%d | Busy: %d ms | RcvWnd Limited: %d ms | Data Segs: %d\n",
            $saddr, $daddr, $dport, $busy_ms, $rwnd_lim_ms, $tp->data_segs_out);
    }
}

Running the Trace and Diagnosing Transmission Efficiency

Execute the bpftrace probe on your Linux server:

sudo bpftrace trace_busy_time.bt

When connections terminate, bpftrace outputs structured telemetry:

05:14:02 CLOSED 10.0.1.12 -> 10.0.4.99:5432 | Busy: 4520 ms | RcvWnd Limited: 12 ms | Data Segs: 84000
05:14:15 CLOSED 10.0.1.12 -> 192.168.1.50:8080 | Busy: 12400 ms | RcvWnd Limited: 8900 ms | Data Segs: 14000

Notice the second entry: of the 12.4 seconds the connection spent busy, 8.9 seconds (over 70%) were spent throttled by the remote receiver’s window (rwnd_limited), pointing to severe backpressure on the client.

Production Applications and Monitoring

Tracking tcpi_busy_time and tcpi_rwnd_limited via eBPF enables systems teams to:

1. Differentiate between network bottleneck latency and client-side processing bottlenecks.
2. Right-size persistent connection pool capacities by auditing average socket duty cycles.
3. Identify clients that keep connections open indefinitely while rarely transmitting data.