How to Use numactl and numastat in Linux to Optimize NUMA Node Memory Allocation and Thread Affinity

Modern enterprise servers powering cloud virtualisation, high-frequency trading platforms, and large in-memory databases (such as PostgreSQL, MySQL, and Redis) rely on multi-socket processors with high core densities (e.g. AMD EPYC and Intel Xeon Scalable processors). In these architectures, physical memory is partitioned across multiple distinct hardware nodes, forming a Non-Uniform Memory Access (NUMA) topology. When a CPU core accesses memory modules physically attached to its own local socket, latency is minimal. However, when a thread must access memory connected to an adjacent socket, data must cross interconnect buses—such as AMD Infinity Fabric or Intel Ultra Path Interconnect (UPI)—imposing a severe latency penalty.

Without deliberate thread and memory placement, the Linux kernel scheduler may migrate compute threads to CPU Node 1 while leaving memory buffers allocated on Node 0. This “remote memory access” scenario degrades throughput and causes unpredictable latency spikes. System engineers resolve this by profiling memory miss ratios using numastat and binding memory allocations and CPU thread affinity using numactl.

Architectural Overview: The NUMA Memory Hierarchy

A NUMA system segments physical CPU cores and DDR memory channels into discrete topological domains:

Memory Access Mode	Interconnect Traversed	Access Latency Profile	Throughput Impact
Local Memory Access	Direct socket memory channels (DDR4/DDR5).	Minimal (~60–80 ns).	Maximum throughput; zero interconnect bus congestion.
Remote Memory Access	Cross-socket UPI or Infinity Fabric links.	High latency (~140–220 ns; 2x–3x slower).	Throughput bottlenecked by interconnect link saturation.
Interleaved Allocation	Round-robin distribution across all nodes.	Balanced, predictable average latency.	Prevents single-node memory saturation for large distributed datasets.
Strict Membind	Locked strictly to designated NUMA node(s).	100% local access guaranteed.	Fails with Out-of-Memory (OOM) if designated node runs out of physical RAM.

Installing NUMA Diagnostic Tools and Inspecting Hardware Topology

The numactl and numastat utilities are bundled within the numactl package on Debian/Ubuntu systems.

Step 1: Install the NUMA Administration Package

# Install numactl on Ubuntu / Debian
sudo apt update
sudo apt install -y numactl hwloc

Step 2: Inspect Server NUMA Hardware Layout

Query the operating system’s detected NUMA nodes, CPU core mappings, and physical memory distributions:

# Display hardware topology and distance matrix
numactl --hardware

The command outputs the total number of available nodes, which CPU logical processors belong to each node (e.g. node 0 cpus: 0 1 2 3 … 31), the total and free RAM per node, and the node distance matrix. A distance of 10 represents local socket memory, while a distance of 20 or 32 indicates remote interconnect hops.

Profiling Cross-Node Memory Leaks with numastat

The numastat utility parses kernel statistics exported through /sys/devices/system/node/node*/numastat to identify performance-degrading remote memory lookups.

Step 1: Run Global NUMA Statistics

# Display system-wide NUMA allocation counters in megabytes
numastat -m

Inspect key metrics across nodes:

• numa_hit: Memory allocated successfully on the CPU’s local node (ideal scenario).
• numa_miss: Memory intended for another node that was forced onto this node due to exhaustion.
• numa_foreign: A thread intended to allocate memory locally, but was redirected to a remote node. High numbers indicate severe cross-node penalties.
• interleave_hit: Pages successfully allocated using round-robin interleaving policies.

Step 2: Profile a Specific Production Process

Identify whether a running database process (e.g. PostgreSQL PID 4821) is straddling multiple nodes inefficiently:

# Inspect per-node memory footprint for a specific process ID
numastat -c postgres

If the process consumes 30 GB on Node 0 and 28 GB on Node 1 while its CPU affinity is locked exclusively to Node 0, half of all memory queries are suffering cross-bus latency.

Controlling Thread and Memory Allocation Using numactl

The numactl utility launches processes with explicit NUMA memory allocation policies and CPU core affinity masks.

Policy 1: Strict Local Node Binding (Membind)

For latency-critical, single-socket workloads (such as high-frequency trading gateways or cache engines), restrict both CPU threads and memory allocations strictly to Node 0:

# Bind execution to CPU Node 0 and memory to Node 0 only
numactl --cpunodebind=0 --membind=0 /usr/bin/redis-server /etc/redis/redis.conf

If Node 0 runs out of memory, allocations will fail rather than falling back to remote memory, preserving strict determinism.

Policy 2: Preferred Node Binding with Automatic Fallback

If you prefer allocating on Node 1 but wish to avoid Out-of-Memory crashes under peak load, configure --preferred:

# Attempt allocation on Node 1, falling back to other nodes if exhausted
numactl --cpunodebind=1 --preferred=1 /usr/bin/node /opt/app/server.js

Policy 3: Memory Interleaving for Large Databases

For applications that consume massive amounts of memory exceeding single-node capacities (e.g. 512 GB database buffers on a multi-terabyte system), round-robin interleaving avoids hotspotting a single memory controller:

# Interleave memory allocations evenly across all available NUMA nodes
numactl --interleave=all /usr/lib/postgresql/16/bin/postgres -D /var/lib/postgresql/16/main

Policy 4: Core-Level Thread Pinning (Physcpubind)

Bind compute tasks directly to specific hardware core ranges to eliminate CPU context migration:

# Pin process to physical CPU cores 0 through 15
numactl --physcpubind=0-15 --membind=0 /opt/analytics/worker

Configuring Kernel Automatic NUMA Balancing via sysctl

The Linux kernel includes an automated background scanner (AutoNUMA) that attempts to detect memory pages accessed by threads on remote nodes and migrate those pages across nodes dynamically.

Step 1: Check AutoNUMA Balancing State

# Query active NUMA balancing status (1 = enabled, 0 = disabled)
sysctl kernel.numa_balancing

Step 2: Optimize AutoNUMA for Dedicated Workloads

While AutoNUMA is beneficial for generic multi-tenant virtualisation, its background page scanning consumes CPU overhead and introduces page-fault jitter into high-throughput databases. For systems where applications are already pinned via numactl, disable automated kernel balancing:

# Disable automated kernel page migration overhead
sudo sysctl -w kernel.numa_balancing=0

# Make persistent across reboots
echo "kernel.numa_balancing = 0" | sudo tee -a /etc/sysctl.d/99-numa.conf

Troubleshooting NUMA Bottlenecks and Allocation Failures

Consult this diagnostic reference table to debug NUMA-related memory performance degradations:

Symptom	Underlying Root Cause	Remediation Procedure
Process terminated by Linux Out-of-Memory (OOM) killer despite abundant free system RAM.	Process was launched with --membind on a specific node, and that specific node ran out of memory.	Switch from --membind to --preferred to allow fallback to other nodes, or increase physical memory assigned to the constrained node.
libnuma: warning: node 1 not available	Attempted to bind to a NUMA node that does not exist in the physical system or is disabled in BIOS.	Check active nodes via numactl --hardware. Ensure NUMA is enabled in system BIOS (e.g. disable Node Interleaving / UMA mode in motherboard firmware).
Severe performance degradation after system operates for several days.	Zone reclaim mode or memory fragmentation preventing clean local node page allocations.	Inspect zone reclaim settings via sysctl vm.zone_reclaim_mode. Set to 0 to allow remote allocation instead of aggressive local page reclamation.
Database queries experience random 50ms latency spikes.	AutoNUMA background scanner generating minor page faults while remapping memory pages.	Disable kernel auto-balancing: sudo sysctl -w kernel.numa_balancing=0 and enforce static process binding via numactl.

Mastering NUMA memory topology, numastat profiling, and numactl allocation policies ensures deterministic low-latency performance and unlocks maximum hardware throughput across enterprise multi-socket Linux servers.