In highly standardized cloud environments, the Linux kernel’s Completely Fair Scheduler (CFS) does an excellent job of balancing workloads across available CPU cores. However, for ultra-low latency applications—such as High-Frequency Trading (HFT) platforms, real-time audio processing, or DPDK-accelerated packet inspection—CFS is actively detrimental.
If the kernel decides to migrate an active trading thread from CPU Core 2 to CPU Core 5, the thread loses all its L1 and L2 cache data. This CPU context switch introduces microscopic but devastating latency spikes. Furthermore, on multi-socket servers, migrating a thread to a core on a different physical processor socket forces it to access memory across the QPI/UPI link, causing massive Non-Uniform Memory Access (NUMA) latency penalties.
To achieve absolute deterministic performance, systems engineers must strip control away from the kernel scheduler using CPU Core Pinning (Affinity) and NUMA Node Binding. This guide explains how to isolate CPU cores and bind processes to specific silicon.
Understanding NUMA and Core Isolation
Before pinning processes, you must understand your hardware topology. A modern dual-socket server has two physical CPUs. Each CPU has its own directly attached RAM. This pairing (CPU + attached RAM) is a NUMA Node.
- Local Memory Access: A core on Node 0 accessing RAM on Node 0 is fast (approx 70ns).
- Remote Memory Access: A core on Node 0 accessing RAM on Node 1 is slow (approx 120ns).
To verify your NUMA topology, install and run numactl:
sudo apt install numactl
numactl --hardware
The output will show which physical CPU cores belong to Node 0, and which belong to Node 1.
Step 1: Isolating CPU Cores at Boot (isolcpus)
Simply telling an application to run on Core 4 is insufficient. If the Linux kernel decides to run a background Cron job or process network interrupts on Core 4, it will preempt your latency-sensitive application.
You must completely banish the Linux kernel scheduler from the target cores. You do this at boot time using the isolcpus kernel parameter.
Open your GRUB configuration file:
sudo nano /etc/default/grub
Locate the GRUB_CMDLINE_LINUX_DEFAULT line and append the isolcpus parameter. For example, to completely isolate Cores 4, 5, 6, and 7:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"
Note: We also add nohz_full (disables the kernel tick timer on those cores) and rcu_nocbs (moves RCU callbacks to other cores) to achieve absolute zero interruption.
Update GRUB and reboot the server:
sudo update-grub
sudo reboot
After the reboot, Cores 4-7 are invisible to standard system processes. If you run htop, you will see those cores sitting at 0.0% utilization, completely silent.
Step 2: Pinning a Process with taskset
Now that we have pristine, silent CPU cores, we can manually force our latency-sensitive application to run on them.
You use the taskset utility to define the CPU affinity mask of a process.
To launch a new application (e.g., hft-engine) and pin it exclusively to isolated Core 4:
taskset -c 4 ./hft-engine
To pin an already running process (PID 1234) to Core 4 and 5:
taskset -cp 4,5 1234
Because Cores 4 and 5 are isolated from the kernel scheduler via GRUB, hft-engine now possesses 100% undisputed access to the silicon. No context switches will ever occur.
Step 3: Binding Memory with numactl
CPU pinning solves cache locality, but we must also solve memory locality (the NUMA problem).
If we pinned our application to Core 4 (which is on NUMA Node 0), but the application allocates memory on NUMA Node 1, performance will degrade.
We use numactl to launch the application, combining both CPU pinning and memory allocation policies.
numactl --physcpubind=4-7 --membind=0 ./hft-engine
This command performs two critical actions simultaneously:
--physcpubind=4-7: Restricts the execution threads strictly to physical Cores 4 through 7.--membind=0: Forces the Linux kernel to allocate all RAM for this process exclusively from the memory banks physically attached to NUMA Node 0. If Node 0 runs out of memory, the process will OOM (Out Of Memory) crash rather than falling back to the slower Node 1 RAM, which is exactly the strict deterministic behavior required for HFT.
Step 4: Handling Interrupt Requests (IRQs)
There is one final source of latency: hardware interrupts (IRQs). If the NIC receives a packet, it triggers an IRQ, forcing a CPU core to pause execution and handle the network stack.
By default, a daemon called irqbalance spreads these interrupts across all cores.
You must stop irqbalance and manually configure the IRQ SMP affinity to ensure hardware interrupts are never routed to your isolated cores (4-7).
sudo systemctl stop irqbalance
sudo systemctl disable irqbalance
You then manually write hexadecimal CPU masks into /proc/irq/<irq_number>/smp_affinity to route network interrupts strictly to your “housekeeping” cores (0-3), keeping your isolated cores mathematically silent.
Conclusion
Relying on the default Linux scheduler is fatal in microsecond-sensitive environments. By combining kernel boot parameters (isolcpus) to banish background noise, taskset to enforce strict execution affinity, and numactl to guarantee local physical memory allocation, engineers can forge an execution environment that delivers absolute, deterministic, ultra-low latency performance.