How to Profile KVM Guest VM-Exits and Virtualization Latency in Linux Using perf kvm

When running virtualization hosts with Linux Kernel-based Virtual Machine (KVM) and QEMU, understanding guest CPU overhead is critical for high-density compute environments. Virtual machines execute user and kernel code natively on physical host CPUs until an operation requires hypervisor intervention—such as handling physical hardware interrupts, accessing unmapped memory (EPT/NPT violations), executing privileged instructions (like CPUID or HLT), or performing memory-mapped I/O (MMIO). When this occurs, the hardware processor forces a transition from guest mode to host mode, known as a VM-exit.

Excessive VM-exits introduce substantial CPU latency and degrade I/O throughput. The Linux kernel performance subsystem includes a dedicated profiling module, perf kvm, capable of tracing hardware virtualization events, measuring VM-exit frequencies, calculating guest transition latency, and pinpointing guest operating system bottlenecks directly from the command line.

Understanding KVM VM-Exits and Performance Impact

Hardware virtualization extensions—Intel VT-x (VMX) and AMD-V (SVM)—define rigorous boundaries between the guest virtual CPU (vCPU) and the host root mode. While KVM optimises modern execution using Extended Page Tables (EPT) and Virtual Processor IDs (VPID), certain operations invariably trigger exits:

• EPT Violation / NPT Page Faults: Triggered when a guest attempts to read or write a memory address that has not yet been backed by physical host RAM pages, requiring host shadow page population.
• I/O Instruction and MMIO Trapping: Triggered when virtual network cards, storage controllers, or legacy serial ports exchange data, forcing the KVM module to intercept and hand off packets to host tap interfaces or virtio queues.
• Interrupt Injection and Halts: When a virtual CPU executes a HLT instruction to save power during idle loops, or when virtual APIC interrupts require delivery.

High exit frequencies (often tens of thousands per second) create severe cache invalidation and instruction pipeline stalls on dense virtualization hosts.

Prerequisites and Kernel Tracing Configuration

Profiling KVM requires root privileges, the linux-tools-common package matching your running kernel version, and enabled KVM kernel tracepoints.

Install the required performance utilities and verify tracepoint availability:

# Install Linux perf tools
sudo apt update
sudo apt install -y linux-tools-common linux-tools-$(uname -r)

# Verify available KVM kernel tracepoints
sudo perf list "kvm:*"

Confirm that KVM hardware modules are loaded and verify the processor virtualization architecture:

# Check active KVM module (kvm_intel or kvm_amd)
lsmod | grep -i kvm

# Verify hypervisor event tracing permissions
sudo sysctl kernel.perf_event_paranoid=1

Recording KVM Events with perf kvm stat record

To analyse what causes a running virtual machine to exit to the host, use perf kvm stat record. This command hooks into host KVM tracepoints to record entry and exit timestamps across all running VMs.

Capture hypervisor events host-wide for 10 seconds:

# Record KVM hypervisor exit events for 10 seconds
sudo perf kvm stat record -- sleep 10

To isolate analysis to a specific QEMU/KVM virtual machine process, identify its Process ID (PID) and restrict the capture:

# Identify PID of target QEMU instance
pgrep -fl qemu-system-x86_64

# Record trace events exclusively for the specified VM process
sudo perf kvm stat record -p 14285 -- sleep 15

The capture writes trace records to perf.data.guest or the default perf.data file in the current working directory.

Analysing VM-Exit Reasons and Latencies

Once recording completes, process the trace file to generate an aggregated report of exit causes, total elapsed time, and maximum latency spikes:

# Generate exit reason summary sorted by total execution time
sudo perf kvm stat report --event=vmexit --sort=time

The resulting output produces an exhaustive diagnostic table:

KVM event: vmexit

VM-EXIT Reason             Samples  Samples%  Time%   Min Time (us)  Max Time (us)  Avg Time (us)
-------------------------  -------  --------  -----   -------------  -------------  -------------
EPT_VIOLATION               14523    42.15%   58.20%      1.24          148.30          8.21
HLT                          8912    25.88%   21.14%      0.82           45.12          4.85
IO_INSTRUCTION               6140    17.83%   12.45%      0.45           22.60          4.15
MSR_WRITE                    3210     9.32%    5.81%      0.30           14.20          3.70
CR_ACCESS                    1750     5.08%    2.40%      0.28            8.50          2.80

Key indicators to evaluate:

• High EPT_VIOLATION Time: Indicates excessive memory allocation or page faulting in the guest. Resolving this often involves configuring Transparent Huge Pages (THP) or dedicated 1 GB / 2 MB hugepages on the host.
• High IO_INSTRUCTION Count: Indicates that guest drivers are using emulated IDE/SATA or legacy network adapters rather than para-virtualised virtio-blk, virtio-scsi, and virtio-net devices.
• Spike in Max Time: Identifies host thread scheduling contention or lock contention on host physical CPUs.

Inspecting Live Real-Time Hypervisor Activity with perf kvm top

For immediate interactive troubleshooting when a guest VM experiences an unexplained freeze or CPU spike, launch the real-time top interface:

# Monitor live KVM exit events across all host vCPUs
sudo perf kvm stat live

The terminal displays dynamic counters refreshed every second, showing instant exit rates per second, average handling time, and active VM-exit reasons, enabling administrators to observe the immediate impact of guest workload changes.

Optimising KVM Configurations Based on Profile Insights

Apply the findings gathered from perf kvm to harden the hypervisor configuration:

• Enable Host Transparent Huge Pages: Reduce EPT violations by backing virtual RAM with hugepages:

  echo always | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

• Configure vCPU Pinning: Prevent CPU cache thrashing by binding guest vCPUs to dedicated physical host cores using virsh vcpupin.
• Enable APIC Virtualisation: Confirm hardware APIC registration (apicv or avic) is active in module options to eliminate synthetic timer exits:

  cat /sys/module/kvm_intel/parameters/enable_apicv

Systematic profiling with perf kvm empowers engineers to eliminate virtualization overhead and deliver bare-metal performance for critical virtualized infrastructure.