Domain Name System (DNS) resolution bottlenecks frequently introduce hidden latency into modern microservices, web servers, and distributed Linux applications. While traditional network monitoring tools capture packets traversing physical network interfaces, they often fail to expose the exact application processes initiating queries or the processing time consumed within user-space resolvers. In Linux systems relying on the standard GNU C Library (glibc), applications resolve hostnames into IP addresses via standard library calls such as getaddrinfo() and gethostbyname(). Tracing these functions at the kernel-user boundary using eBPF (Extended Berkeley Packet Filter) allows systems engineers to pinpoint which processes request external lookups, record the hostnames being queried, and measure round-trip resolution latency in milliseconds.
Understanding eBPF Hostname Tracing with gethostlatency
The gethostlatency utility is part of the standard BCC (BPF Compiler Collection) toolchain. It attaches uprobes (user-space dynamic probes) to the POSIX resolver functions provided by the shared standard library (libc.so.6). Specifically, it instruments the entry and return points of getaddrinfo(), gethostbyname(), and gethostbyname2().
When an application invokes one of these resolver routines, gethostlatency captures the invoking Process ID (PID), the process name, and the queried hostname string from function arguments. Upon function return, the tool calculates the elapsed duration between function entry and exit. This provides granular timing data encompassing internal cache checks (such as systemd-resolved or nscd), local socket transmission, recursive upstream resolver query delays, and final response decoding.
Installing the BCC Tracing Tools
Because gethostlatency relies on the Linux eBPF subsystem and kernel headers, appropriate tracing packages must be installed based on your Linux distribution.
On Debian and Ubuntu systems, install the BCC packages and kernel development headers using apt:
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)
On RHEL, Rocky Linux, or Fedora installations, enable required repositories and install BCC via dnf:
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)
Once installed, BCC binaries are typically located in /usr/sbin (often suffixed with -bpfcc on Ubuntu) or in /usr/share/bcc/tools/ on enterprise distributions. Verify executable availability using the following command:
which gethostlatency || which gethostlatency-bpfcc || ls -l /usr/share/bcc/tools/gethostlatency
Tracing Global Hostname Queries Across All Processes
To trace every DNS lookup occurring across the entire operating system, execute gethostlatency with root privileges. By default, the tool outputs real-time query events in tabular format:
sudo /usr/sbin/gethostlatency-bpfcc
When system processes generate network lookups, gethostlatency displays continuous telemetry similar to the following trace output:
TIME PID COMM LATms HOST
12:14:02 2104 curl 4.21 api.github.com
12:14:05 3452 python3 22.65 db-cluster.internal.net
12:14:08 8910 node 148.12 auth.service.production
12:14:11 1023 systemd-timesyn 2.15 0.debian.pool.ntp.org
The standard output provides vital operational metrics:
- TIME: The local system timestamp when the DNS query returned.
- PID: The process identifier that issued the resolution call.
- COMM: The executable command name executing the lookup.
- LATms: The total duration in milliseconds spent waiting for the resolution request to complete.
- HOST: The exact domain name or hostname requested by the caller.
Filtering DNS Traces by Specific Process IDs
During active incident triage, tracing the entire system may generate excessive diagnostic noise. To isolate lookups generated by a single application instance, use the -p option followed by the target Process ID:
sudo /usr/sbin/gethostlatency-bpfcc -p 3452
This filters all uprobes so that only resolver invocations from PID 3452 are captured and printed to the terminal, allowing developers to profile database drivers, web frameworks, or microservice components without unrelated background OS activity.
Adding High-Resolution Timestamps
When correlating resolution latency with external application logs, distributed APM traces, or packet captures in Wireshark, standard wall-clock time may lack required precision. The -T option prepends full time stamps, while the -d option reports relative microsecond or millisecond deltas:
sudo /usr/sbin/gethostlatency-bpfcc -T
This outputs full time fields suitable for structured ingestion into centralized observability pipelines:
TIME(s) PID COMM LATms HOST
2026-11-08 12:18:22 4102 nginx 1.82 static.example.org
2026-11-08 12:18:24 4102 nginx 2.04 cdn.example.org
Diagnosing Latency Anomalies and Resolver Bottlenecks
Tracing resolver latency using eBPF reveals architectural misconfigurations that standard Ping or Dig commands cannot diagnose:
Sequential Search Domain Lookups
If an entry in /etc/resolv.conf contains multiple domains in the search directive (such as search dev.internal test.internal example.com), applications querying non-fully qualified names will attempt sequential lookups. In gethostlatency, this manifests as multiple consecutive queries with incremental latency delays before the actual target resolves successfully.
Uncached Repetitive Queries
Applications that do not implement local DNS caching or connection pooling will repeatedly invoke getaddrinfo() for every outbound HTTP request. When high throughput processes generate thousands of identical lookups per minute, gethostlatency highlights frequent query cycles with 1–10ms latency that collectively degrade CPU performance and network stack throughput.
Recursive Upstream Failures and Timeouts
Queries exhibiting latency spikes exceeding 1,000ms or 5,000ms indicate unresponsive primary nameservers. When the primary DNS server fails to respond, the local stub resolver waits for the configured timeout before failing over to secondary nameservers listed in /etc/resolv.conf, generating massive latency penalties in user-facing workflows.
Verifying Resolver Library Path Compatibility
In environments where applications link against non-standard C libraries (such as Alpine Linux running musl libc) or utilise isolated containerised namespaces, BCC uprobes must target the exact shared object used by the executable. You can verify which library provides resolver symbols using ldd:
ldd /usr/bin/curl | grep libc.so
If the target application runs inside a distinct container mount namespace, execute gethostlatency from the host operating system. Because eBPF operates globally inside the kernel, uprobes placed on the host libc.so.6 capture calls from containers sharing standard glibc binaries, providing non-intrusive container visibility without modifying production images.