How to Use numactl in Linux to Optimise NUMA Memory Allocation, CPU Node Binding, and Memory Interleaving

Architectural Mechanics of Non-Uniform Memory Access (NUMA) in Linux

Modern enterprise multi-socket servers and high-core-count processors (such as AMD EPYC and Intel Xeon Scalable architectures) do not treat physical random-access memory (RAM) as a uniform, equidistant pool. Instead, hardware employs Non-Uniform Memory Access (NUMA). Under NUMA, CPU cores and memory controllers are grouped into discrete hardware domains termed NUMA Nodes.

When a processor core accesses RAM physically wired to its local integrated memory controller, latency is minimal (typically 60 to 80 nanoseconds). However, when a thread scheduled on Node 0 must read or write data resident in RAM wired to Node 1, memory requests must traverse inter-socket fabric bridges—such as Intel Ultra Path Interconnect (UPI) or AMD Infinity Fabric. This Remote Node Access incurs substantial latency penalties (120 to 180+ nanoseconds) and consumes high-speed interconnect bus bandwidth, causing jitter and latency spikes in database engines (e.g., PostgreSQL, Redis), in-memory analytics, and high-frequency trading workloads.

Memory Allocation Policy	Kernel Invocation Flag	Allocation Behaviour	Optimal Workload Context
Local (Default)	--localalloc	Allocates RAM on the node where the requesting thread is currently executing	General-purpose multi-threaded workloads
Membind	--membind=nodes	Restricts allocations strictly to designated nodes; fails if node memory exhausts	High-performance pinned databases, isolated containers
Preferred	--preferred=node	Attempts allocation on specified node; falls back gracefully to alternate nodes	Workloads prioritizing a specific socket without OOM risks
Interleave	--interleave=nodes	Rounds-robin page allocations uniformly across all designated NUMA nodes	Shared-memory data structures, large streaming buffers

Auditing NUMA Hardware Topology and Inter-Node Distance

Before pinning processes or enforcing memory allocation policies, systems engineers must audit the server’s physical and logical topology using numactl and numastat.

# Install the numactl utility suite on Debian/Ubuntu
sudo apt-get update && sudo apt-get install -y numactl

# Install on RHEL, AlmaLinux, or Rocky Linux
sudo dnf install -y numactl

# Inspect NUMA hardware topology, core mappings, and distances
numactl --hardware

A dual-socket system returns detailed node maps accompanied by the Node Distance Matrix:

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
node 0 size: 130942 MB
node 0 free: 112450 MB
node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
node 1 size: 131072 MB
node 1 free: 118204 MB
node distances:
node   0   1 
  0:  10  21 
  1:  21  10 

In this matrix, a distance score of 10 represents baseline local memory latency. A distance score of 21 indicates that accessing memory across the inter-socket bridge incurs a 2.1x latency overhead.

Inspecting Memory Locality with numastat

The numastat command interrogates /sys/devices/system/node/node*/numastat, providing real-time telemetry regarding memory allocation success:

# View system-wide allocation hits and misses
numastat

# Inspect memory footprint broken down per running process
numastat -c redis-server

Metric	Diagnostic Significance
numa_hit	Count of pages successfully allocated on the local node where the thread executed.
numa_miss	Count of pages intended for another node that fell back locally.
numa_foreign	Count of allocations intended for this node that were forced to execute remotely due to memory pressure.
local_node / other_node	Ratio comparing successful local memory execution versus penalised cross-socket bus fetches.

Binding Processes and Memory via numactl Invocations

The numactl wrapper launches processes with deterministic CPU core and memory node bindings, preventing operating system thread migrations.

1. Restricting Execution and Memory to a Single Node:

For latency-sensitive databases (such as Redis or MySQL), bind execution exclusively to CPU Node 0 and mandate that all memory allocations reside within Node 0’s RAM banks:

# Launch process bound strictly to Node 0 CPU cores and Node 0 memory
numactl --cpunodebind=0 --membind=0 /usr/bin/redis-server /etc/redis/redis.conf

2. Fine-Grained Physical Core Pinning:

To avoid sharing cache lines or hyperthreaded execution pipelines with secondary processes, pin a process directly to specific physical CPU cores (e.g., cores 0, 2, 4, and 6):

# Pin to specific physical core IDs with local memory allocation
numactl --physcpubind=0,2,4,6 --localalloc ./high_throughput_worker

3. Enforcing Memory Interleaving for Large In-Memory Caches:

When an application allocates a massive dataset that exceeds the RAM capacity of a single NUMA node (e.g., an in-memory graph database spanning 200GB), leaving memory allocation to default local heuristics causes Node 0 to exhaust its memory while Node 1 sits idle. Interleaving distributes pages uniformly across all sockets:

# Interleave memory pages round-robin across nodes 0 and 1
numactl --interleave=0,1 ./graph_database_engine

Tuning Automatic NUMA Balancing in the Linux Kernel

The Linux kernel includes an automated subsystem known as AutoNUMA (kernel.numa_balancing). When active, the kernel periodically scans task virtual memory addresses, revoking page access permissions to generate intentional minor page faults. By analysing which CPU core triggered the fault, the kernel dynamically migrates the memory page closer to the consuming thread.

While AutoNUMA optimizes generic virtualization clusters, it can induce severe latency jitter in deterministic, pre-pinned enterprise workloads due to continuous page fault scanning.

# Inspect current AutoNUMA state (1 = Enabled, 0 = Disabled)
cat /proc/sys/kernel/numa_balancing

# Disable AutoNUMA to eliminate page fault jitter for pre-pinned applications
sudo sysctl -w kernel.numa_balancing=0

# Persist setting across reboots in /etc/sysctl.d/99-numa.conf
echo "kernel.numa_balancing=0" | sudo tee /etc/sysctl.d/99-numa.conf

Troubleshooting NUMA Faults and Memory Exhaustion

Observed Fault	Probable Root Cause	Technical Diagnostic Method	Remediation Protocol
OOM Killer Terminates Process	Process launched with --membind=0 exhausted Node 0 memory while Node 1 had free capacity	Check dmesg | grep -i oom; examine per-node free memory in numactl -H	Switch from --membind to --preferred=0 to permit graceful cross-node fallback
High System Latency / Unexplained Jitter	Kernel numa_balancing scanning pages and moving memory across sockets	Review CPU utilization in perf top; check for high change_prot_numa activity	Disable automatic NUMA balancing via sudo sysctl -w kernel.numa_balancing=0
numactl: node 1 not available	System is a single-socket workstation or UMA emulation active in motherboard BIOS	Run lscpu | grep NUMA; verify hardware supports multi-node architecture	Verify dual-socket hardware; disable UMA (Uniform Memory Access) mode in BIOS/UEFI
Cross-Node Traffic Dominating Interconnect	Threads migrated by OS scheduler to secondary socket while data remained on primary node	Monitor with numastat -c <process>; look for mounting numa_foreign counts	Bind both CPU threads and memory simultaneously using numactl --cpunodebind=N --membind=N