How to Trace TCP Socket TCP_INFO tcpi_delivery_rate_app_limited in Linux Using eBPF

Diagnosing throughput degradation across high-bandwidth Linux server environments requires determining whether performance bottlenecks stem from the underlying network fabric or application starvation. The kernel struct tcp_info maintains a bitfield member named tcpi_delivery_rate_app_limited, which indicates whether a connection was transmission-limited by the socket buffer rather than transport congestion. Tracing this flag using eBPF provides direct observability into application runtime behaviour.

Understanding the Application-Limited State

Modern Linux TCP congestion control algorithms, notably BBR (Bottleneck Bandwidth and RTT), track socket pacing and delivery rates continuously. If an application fails to push data into the write queue fast enough to saturate the estimated pipe capacity, the kernel marks the socket sample as app_limited. Analysing this metric prevents network engineers from misdiagnosing server-side processing delays as transport packet loss.

Developing the eBPF Kernel Probe

The following BCC Python script attaches a kprobe to tcp_rate_check_app_limited to log real-time socket transitions directly from kernel memory:

from bcc import BPF

bpf_source = """
#include <net/sock.h>
#include <linux/tcp.h>

struct app_event_t {
    u32 pid;
    u32 saddr;
    u32 daddr;
    u16 sport;
    u16 dport;
    u8 is_app_limited;
};

BPF_PERF_OUTPUT(events);

int trace_tcp_rate(struct pt_regs *ctx, struct sock *sk) {
    if (sk->sk_family != AF_INET)
        return 0;

    struct tcp_sock *tp = tcp_sk(sk);
    struct app_event_t event = {};

    event.pid = bpf_get_current_pid_tgid() >> 32;
    event.saddr = sk->sk_rcv_saddr;
    event.daddr = sk->sk_daddr;
    event.sport = sk->sk_num;
    event.dport = bpf_ntohs(sk->sk_dport);
    event.is_app_limited = tp->app_limited ? 1 : 0;

    events.perf_submit(ctx, &event, sizeof(event));
    return 0;
}
"""

b = BPF(text=bpf_source)
b.attach_kprobe(event="tcp_rate_check_app_limited", fn_name="trace_tcp_rate")

def print_event(cpu, data, size):
    event = b["events"].event(data)
    status = "APPLICATION_STARVED" if event.is_app_limited else "PIPE_SATURATED"
    print(f"PID {event.pid} | {event.sport} -> {event.dport} | State: {status}")

b["events"].open_perf_buffer(print_event)
print("Tracing tcpi_delivery_rate_app_limited events... Ctrl-C to stop.")
while True:
    b.perf_buffer_poll()

Operational Verification and Kernel Tuning

Execute the tracing agent on your target Linux machine with appropriate administrative capabilities:

sudo python3 trace_app_limited.py

When an application demonstrates persistent APPLICATION_STARVED flags alongside sub-optimal throughput, inspect the following runtime parameters:

• Application Buffer Sizing: Verify that user-space buffers align with the negotiated TCP window scales via setsockopt(SO_SNDBUF).
• Event Loop Latency: Assess asynchronous worker loops for blocking system calls that prevent prompt buffer population.
• Sysctl Verification: Audit net.ipv4.tcp_wmem settings to ensure auto-tuning parameters do not artificially throttle headroom.

By leveraging eBPF to trace socket delivery dynamics, system administrators can isolate root causes and optimise software architecture with uncompromising accuracy.