How to Trace TCP Zero-Copy Send Completion Latency in Linux Using eBPF

In high-throughput storage engines, media streaming pipelines, and distributed databases, memory copying between user-space buffers and kernel sk_buff structures represents a substantial CPU overhead. The Linux kernel’s MSG_ZEROCOPY flag allows applications to transmit data directly from user-space memory to network interface cards without intermediate memory copies. However, zero-copy networking introduces asynchronous completion notifications: user applications must read the socket’s error queue (via recvmsg(MSG_ERRQUEUE)) to confirm when hardware transmission completes before recycling buffers. Using eBPF, systems engineers can trace zero-copy completion latency and identify socket queue bottlenecks.

Understanding TCP MSG_ZEROCOPY Mechanics

When an application invokes send(sock, buf, len, MSG_ZEROCOPY):

• The kernel pins the user memory pages in place and passes references directly to the network device driver.
• Once the network card dispatches the frame or the TCP stack receives acknowledgements, the kernel posts a completion notification (SO_EE_ORIGIN_ZEROCOPY) to the socket’s error queue.
• If the application fails to harvest these completion notifications promptly, pinned pages accumulate, consuming virtual memory and stalling future transmissions.

Tracing Zero-Copy Completions with bpftrace

Deploy a real-time eBPF script targeting __sock_zerocopy_put() to measure the duration between packet dispatch and completion notification:

# Install bpftrace on Ubuntu/Debian
sudo apt-get update && sudo apt-get install -y bpftrace linux-headers-$(uname -r)

# Trace zero-copy completion times and socket error queue events
sudo bpftrace -e '
kprobe:__sock_zerocopy_put {
    $uarg = arg0;
    @zerocopy_completions[comm] = count();
}
kprobe:tcp_sendmsg_locked {
    $flags = arg3;
    if ($flags & 0x4000000) { // MSG_ZEROCOPY flag
        @zerocopy_sends[comm] = count();
    }
}'

Comparing the frequency of @zerocopy_sends against @zerocopy_completions reveals whether application worker threads are processing error queue notifications efficiently.

Monitoring Kernel Zero-Copy Statistics with nstat

Inspect system-level socket zero-copy statistics to ensure hardware drivers are not falling back to software copies:

# View live zero-copy counters
nstat -az | grep -E "TCPZeroCopy|ExtTCPZeroCopy"

# Monitor socket buffer error queue depth
cat /proc/net/sockstat

Tuning Socket Thresholds for High-Throughput Storage

To maximize efficiency when employing zero-copy networking in production:

1. Payload Size Thresholds: Only apply MSG_ZEROCOPY to transmissions exceeding 10 Kilobytes. For small writes (under 4KB), the CPU overhead of page pinning and error-queue synchronization exceeds the cost of a traditional memory copy.
2. Opt-In Socket Option: Ensure sockets explicitly register capability before transmission:

   int one = 1;
   setsockopt(sockfd, SOL_SOCKET, SO_ZEROCOPY, &one, sizeof(one));

3. Batch Error Queue Reads: Drain multiple completion notifications per recvmsg() call using epoll edge-triggered notifications on EPOLLERR to minimize context switches.