How to Trace TCP Zero-Copy Send Latency in Linux Using eBPF

High-throughput network applications—such as NVMe-over-Fabrics targets, Kafka message brokers, and content delivery reverse proxies—frequently bottleneck on memory bus bandwidth when copying packet buffers between user space and kernel space. The Linux kernel’s MSG_ZEROCOPY socket flag enables user space processes to bypass intermediate data copies, allowing network interface cards (NICs) to DMA payload memory directly from application buffers. However, tracking asynchronous completion notifications (SO_EE_ORIGIN_ZEROCOPY) introduces subtle latency overheads. Tracing zero-copy send paths with eBPF exposes these delays.

Understanding Linux Kernel MSG_ZEROCOPY Mechanics

The zero-copy transmission pipeline requires careful socket coordination:

• Memory Pinning: When an application transmits data using sendmsg(..., MSG_ZEROCOPY), the kernel locks user space memory pages in place inside skb_zerocopy_iter_stream(), bypassing kernel sk_buff memory allocation.
• Hardware Transmission: The NIC reads memory directly across the PCIe bus and dispatches packets onto the physical network wire.
• Completion Notification: Once the NIC completes transmission, the driver raises a completion interrupt, and the kernel queues a notification into the socket’s error queue (sock_queue_err_skb()). The application must drain this queue via recvmsg(MSG_ERRQUEUE) to unlock its memory pages.

Tracing Zero-Copy Invocations in Real Time with bpftrace

Using bpftrace, intercept the kernel zero-copy send handler to monitor throughput:

sudo bpftrace -e '
#include <net/sock.h>
#include <linux/tcp.h>

kprobe:tcp_sendmsg_locked {
    $flags = arg2;
    if ($flags & 0x4000000) { // MSG_ZEROCOPY flag
        @zerocopy_sends[comm] = count();
    } else {
        @standard_sends[comm] = count();
    }
}

interval:s:10 {
    print(@zerocopy_sends);
    print(@standard_sends);
    clear(@zerocopy_sends);
    clear(@standard_sends);
}'

This script tabulates active processes utilizing zero-copy transmission relative to traditional copied buffers.

Measuring Socket Notification and Page Unlock Latency

To inspect how long application buffers remain locked awaiting NIC hardware acknowledgment:

sudo bpftrace -e '
kprobe:sock_zerocopy_alloc {
    @start[arg0] = nsecs;
}

kprobe:sock_zerocopy_callback {
    $dur_us = (nsecs - @start[arg0]) / 1000;
    if ($dur_us > 500) {
        printf("Zero-Copy Stalled: ubuf=%p delay_us=%d\n", arg0, $dur_us);
    }
    delete(@start[arg0]);
}'

Identifying delayed completion notifications highlights NIC queue contention or undersized error queues, ensuring storage services maintain peak zero-copy throughput.