How to Trace TCP Pacing Rates and FQ Qdisc Latency in Linux Using eBPF

Modern Linux TCP congestion control algorithms such as BBR (Bottleneck Bandwidth and RTT) rely heavily on kernel socket pacing to smooth outbound packet bursts and prevent network bufferbloat. By coupling the sch_fq (Fair Queueing) packet scheduler with socket-level rate calculations, the kernel spaces packet transmissions evenly across round-trip intervals. Tracing these pacing dynamics using eBPF exposes queuing stalls, timer drift, and misconfigured qdisc parameters in high-throughput production environments.

Understanding Kernel TCP Pacing and sch_fq Architecture

TCP socket pacing operates at two coordinated layers within the Linux networking subsystem:

• Socket Rate Calculation: Algorithms like BBR continuously compute the pacing rate stored in sk->sk_pacing_rate, measuring target throughput in bytes per second.
• Fair Queueing Queueing Discipline: The sch_fq packet scheduler inspects outbound skb->tstamp headers, enqueuing packets into per-flow red-black trees and releasing them only when the targeted delivery deadline elapses.
• High-Resolution Timers (hrtimer): The queueing discipline arms kernel hrtimers to wake the transmission queue, ensuring precise transmission timing even when burst sizes fluctuate.

Verifying Fair Queueing Configuration on Network Interfaces

Prior to tracing, ensure the network interface uses the fq qdisc rather than default classless schedulers like pfifo_fast or fq_codel:

# Check active qdisc configuration on the primary network interface
tc qdisc show dev eth0

# If fq is not active, apply Fair Queueing with pacing support
sudo tc qdisc replace dev eth0 root fq pacing

Query socket-level pacing parameters using ss to observe real-time rate allocation:

# Inspect socket pacing rate and delivery rate for active connections
ss -tin '( dport = :443 or sport = :443 )'

Tracing sk_pacing_rate Dynamic Adjustments with bpftrace

To inspect real-time changes to socket pacing rates during congestion avoidance phases, attach an eBPF tracepoint to tcp_update_pacing_rate:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_update_pacing_rate
{
    $sk = (struct sock *)arg0;
    $pacing_rate = $sk->sk_pacing_rate;
    $max_pacing_rate = $sk->sk_max_pacing_rate;

    printf("%-8d %-16s PacingRate: %lu Bps (Max: %lu Bps)\n",
           pid, comm, $pacing_rate, $max_pacing_rate);
}'

This script samples pacing rate recalculations, pinpointing when transport congestion collapses scheduled throughput.

Measuring FQ Queueing Delays with eBPF kretprobes

Network latency spikes often occur when packets spend excessive duration trapped in the FQ scheduler due to timer inaccuracies or CPU throttling. Measure the enqueue-to-dequeue duration across fq_enqueue and fq_dequeue:

sudo bpftrace -e '
#include <linux/skbuff.h>

kprobe:fq_enqueue
{
    $skb = (struct sk_buff *)arg0;
    @enqueue_time[$skb] = nsecs;
}

kprobe:fq_dequeue
{
    $skb = (struct sk_buff *)retval;
    if ($skb != 0 && @enqueue_time[$skb]) {
        $delta_us = (nsecs - @enqueue_time[$skb]) / 1000;
        @fq_latency_us = hist($delta_us);
        delete(@enqueue_time[$skb]);
    }
}

END
{
    printf("\nFQ Queueing Latency Histogram (microseconds):\n");
    print(@fq_latency_us);
    clear(@enqueue_time);
}'

Interpreting Trace Results and Performance Tuning

Evaluate the generated histogram and socket telemetry against established thresholds:

• Sub-100 Microsecond Latencies: Normal for standard kernel scheduling, confirming responsive timer execution.
• Millisecond-Level Spikes: Indicates interrupt latency, thread contention, or timer jitter caused by aggressive CPU c-states or power-saving governor profiles.
• Zero Pacing Rates: Indicates socket stalling caused by application buffer backpressure or window starvation.