How to Trace TCP Socket Orphan Inodes and Memory Limits in Linux Using eBPF

In high-throughput Linux web servers and reverse proxies, an “orphaned” TCP socket represents an active network connection that is no longer attached to any user-space file descriptor (process). This occurs when an application process calls close() on an active socket that still has outgoing data in its transmit buffer, or when a client ungracefully disconnects while the server waits for FIN-ACK exchanges. When orphan socket counts exceed kernel limits (governed by net.ipv4.tcp_max_orphans), the Linux kernel emits Out of socket memory warnings and forcibly resets connections via RST packets. Using eBPF, Linux engineers can monitor orphan lifecycles and diagnose memory exhaustion in real time.

Understanding Linux TCP Orphan Socket Mechanics

The Linux kernel tracks orphaned socket allocations through dedicated memory accounting structures:

• tcp_close(): When userspace closes a socket with unacknowledged egress data, the kernel detaches the socket from the file descriptor and increments the global orphan counter (tcp_orphan_count).
• tcp_max_orphans: Maximum number of orphaned sockets allowed across the system. If this threshold is breached, subsequent orphaned sockets are immediately reset, discarding data.
• net.ipv4.tcp_mem: System-wide memory limits (measured in 4KB memory pages) defining low, pressure, and high memory boundaries for all active TCP sockets combined.
• SOCK_DEAD Flag: The kernel marks orphaned sockets with the SOCK_DEAD state, leaving them under kernel timer supervision (such as the FIN-WAIT-2 or keepalive timer).

Tracing Socket Orphan Allocations with bpftrace

Deploy a bpftrace one-liner to intercept socket closures that transition into orphaned status, recording the process name and destination endpoint:

sudo bpftrace -e '
#include <net/sock.h>
#include <net/tcp.h>

kprobe:tcp_close
{
    $sk = (struct sock *)arg0;
    $inet = (struct inet_sock *)$sk;
    
    // Check if transmit queue has unacknowledged data upon close
    $tp = (struct tcp_sock *)$sk;
    if ($tp->write_seq != $tp->snd_una) {
        printf("ORPHAN CREATED: %s (PID %d) | %s:%d -> %s:%d | Unacked bytes: %u
",
               comm, pid,
               ntop($inet->inet_saddr), bswap($inet->inet_sport),
               ntop($inet->inet_daddr), bswap($inet->inet_dport),
               ($tp->write_seq - $tp->snd_una));
    }
}'

When an application abruptly terminates client sessions before TCP transmit buffers flush, bpftrace highlights the socket and unacknowledged byte volume:

ORPHAN CREATED: gunicorn (PID 3140) | 10.0.1.8:8080 -> 198.51.100.4:52110 | Unacked bytes: 14480
ORPHAN CREATED: nginx (PID 1822) | 10.0.1.8:443 -> 203.0.113.88:61204 | Unacked bytes: 28960

Tracing Kernel Out of Socket Memory Events with BCC

To detect when orphaned socket limits cause kernel resets, compile an eBPF script targeting tcp_out_of_memory:

from bcc import BPF
import time

bpf_code = """
#include <net/sock.h>
#include <net/tcp.h>

BPF_HISTOGRAM(orphan_drops);

int kprobe__tcp_out_of_memory(struct pt_regs *ctx, struct sock *sk) {
    u32 zero = 0;
    orphan_drops.increment(zero);
    return 0;
}
"""

b = BPF(text=bpf_code)
print("Monitoring TCP Out of Memory and Orphan resets... Hit Ctrl+C to stop.")

try:
    while True:
        time.sleep(5)
        table = b["orphan_drops"]
        for k, v in table.items():
            if v.value > 0:
                print(f"[CRITICAL ALERT] TCP Out of Memory events detected: {v.value}")
        table.clear()
except KeyboardInterrupt:
    print("
Detached.")

Kernel Tuning and Orphan Mitigation Strategies

When high orphan counts or socket memory pressure occurs on production servers:

1. Increase System Maximum Orphans:

   sudo sysctl -w net.ipv4.tcp_max_orphans=262144

2. Accelerate FIN-WAIT-2 Orphan Teardown:

   Reduce the orphan timeout from the default 60 seconds to 15 seconds to free kernel memory buffers rapidly:

   sudo sysctl -w net.ipv4.tcp_fin_timeout=15

3. Expand TCP Memory Page Thresholds:

   # Set low, pressure, and high limits in 4KB pages
   sudo sysctl -w net.ipv4.tcp_mem="786432 1048576 1572864"

4. Application Connection Draining: Configure web servers (such as Nginx lingering_close on; or Node.js graceful shutdown hooks) to allow transmission buffers to drain before closing socket descriptors.