How to Trace DNS Lookup Latency and Hostname Queries in Linux Using gethostlatency

Domain Name System (DNS) resolution bottlenecks frequently introduce hidden latency into modern microservices, web servers, and distributed Linux applications. While traditional network monitoring tools capture packets traversing physical network interfaces, they often fail to expose the exact application processes initiating queries or the processing time consumed within user-space resolvers. In Linux systems relying on the standard GNU C Library (glibc), applications resolve hostnames into IP addresses via standard library calls such as getaddrinfo() and gethostbyname(). Tracing these functions at the kernel-user boundary using eBPF (Extended Berkeley Packet Filter) allows systems engineers to pinpoint which processes request external lookups, record the hostnames being queried, and measure round-trip resolution latency in milliseconds.

Understanding eBPF Hostname Tracing with gethostlatency

The gethostlatency utility is part of the standard BCC (BPF Compiler Collection) toolchain. It attaches uprobes (user-space dynamic probes) to the POSIX resolver functions provided by the shared standard library (libc.so.6). Specifically, it instruments the entry and return points of getaddrinfo(), gethostbyname(), and gethostbyname2().

When an application invokes one of these resolver routines, gethostlatency captures the invoking Process ID (PID), the process name, and the queried hostname string from function arguments. Upon function return, the tool calculates the elapsed duration between function entry and exit. This provides granular timing data encompassing internal cache checks (such as systemd-resolved or nscd), local socket transmission, recursive upstream resolver query delays, and final response decoding.

Installing the BCC Tracing Tools

Because gethostlatency relies on the Linux eBPF subsystem and kernel headers, appropriate tracing packages must be installed based on your Linux distribution.

On Debian and Ubuntu systems, install the BCC packages and kernel development headers using apt:

sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On RHEL, Rocky Linux, or Fedora installations, enable required repositories and install BCC via dnf:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

Once installed, BCC binaries are typically located in /usr/sbin (often suffixed with -bpfcc on Ubuntu) or in /usr/share/bcc/tools/ on enterprise distributions. Verify executable availability using the following command:

which gethostlatency || which gethostlatency-bpfcc || ls -l /usr/share/bcc/tools/gethostlatency

Tracing Global Hostname Queries Across All Processes

To trace every DNS lookup occurring across the entire operating system, execute gethostlatency with root privileges. By default, the tool outputs real-time query events in tabular format:

sudo /usr/sbin/gethostlatency-bpfcc

When system processes generate network lookups, gethostlatency displays continuous telemetry similar to the following trace output:

TIME      PID    COMM              LATms HOST
12:14:02  2104   curl               4.21 api.github.com
12:14:05  3452   python3           22.65 db-cluster.internal.net
12:14:08  8910   node             148.12 auth.service.production
12:14:11  1023   systemd-timesyn    2.15 0.debian.pool.ntp.org

The standard output provides vital operational metrics:

  • TIME: The local system timestamp when the DNS query returned.
  • PID: The process identifier that issued the resolution call.
  • COMM: The executable command name executing the lookup.
  • LATms: The total duration in milliseconds spent waiting for the resolution request to complete.
  • HOST: The exact domain name or hostname requested by the caller.

Filtering DNS Traces by Specific Process IDs

During active incident triage, tracing the entire system may generate excessive diagnostic noise. To isolate lookups generated by a single application instance, use the -p option followed by the target Process ID:

sudo /usr/sbin/gethostlatency-bpfcc -p 3452

This filters all uprobes so that only resolver invocations from PID 3452 are captured and printed to the terminal, allowing developers to profile database drivers, web frameworks, or microservice components without unrelated background OS activity.

Adding High-Resolution Timestamps

When correlating resolution latency with external application logs, distributed APM traces, or packet captures in Wireshark, standard wall-clock time may lack required precision. The -T option prepends full time stamps, while the -d option reports relative microsecond or millisecond deltas:

sudo /usr/sbin/gethostlatency-bpfcc -T

This outputs full time fields suitable for structured ingestion into centralized observability pipelines:

TIME(s)            PID    COMM         LATms HOST
2026-11-08 12:18:22 4102   nginx         1.82 static.example.org
2026-11-08 12:18:24 4102   nginx         2.04 cdn.example.org

Diagnosing Latency Anomalies and Resolver Bottlenecks

Tracing resolver latency using eBPF reveals architectural misconfigurations that standard Ping or Dig commands cannot diagnose:

Sequential Search Domain Lookups

If an entry in /etc/resolv.conf contains multiple domains in the search directive (such as search dev.internal test.internal example.com), applications querying non-fully qualified names will attempt sequential lookups. In gethostlatency, this manifests as multiple consecutive queries with incremental latency delays before the actual target resolves successfully.

Uncached Repetitive Queries

Applications that do not implement local DNS caching or connection pooling will repeatedly invoke getaddrinfo() for every outbound HTTP request. When high throughput processes generate thousands of identical lookups per minute, gethostlatency highlights frequent query cycles with 1–10ms latency that collectively degrade CPU performance and network stack throughput.

Recursive Upstream Failures and Timeouts

Queries exhibiting latency spikes exceeding 1,000ms or 5,000ms indicate unresponsive primary nameservers. When the primary DNS server fails to respond, the local stub resolver waits for the configured timeout before failing over to secondary nameservers listed in /etc/resolv.conf, generating massive latency penalties in user-facing workflows.

Verifying Resolver Library Path Compatibility

In environments where applications link against non-standard C libraries (such as Alpine Linux running musl libc) or utilise isolated containerised namespaces, BCC uprobes must target the exact shared object used by the executable. You can verify which library provides resolver symbols using ldd:

ldd /usr/bin/curl | grep libc.so

If the target application runs inside a distinct container mount namespace, execute gethostlatency from the host operating system. Because eBPF operates globally inside the kernel, uprobes placed on the host libc.so.6 capture calls from containers sharing standard glibc binaries, providing non-intrusive container visibility without modifying production images.

Get the best tech tips delivered straight to your inbox.

Join thousands of readers mastering Apple, Google, Microsoft, and Linux.