How to Configure Linux SR-IOV (Single Root I/O Virtualization) for KVM Network Hardware Pass-Through

In high-performance virtualized environments (such as Telco NFV clouds or High-Frequency Trading hypervisors), the standard software-based virtual bridges (like Linux br0 or Open vSwitch) introduce unacceptable latency and CPU overhead. Every packet traversing a software bridge requires context switching between the hypervisor and the Virtual Machine (VM), limiting throughput to a fraction of wire speed.

The industry standard solution for zero-overhead virtual networking is SR-IOV (Single Root I/O Virtualization). SR-IOV allows a single physical PCIe network card (the Physical Function, or PF) to securely slice itself into multiple lightweight virtual network cards (Virtual Functions, or VFs) at the hardware level. The hypervisor can then pass these Virtual Functions directly through to KVM Virtual Machines. The VM’s network driver talks directly to the hardware silicon, entirely bypassing the Linux hypervisor’s software network stack.

This guide explains how to enable IOMMU, instantiate Virtual Functions, and attach them to KVM Virtual Machines using virsh.

Understanding SR-IOV Architecture

SR-IOV relies on two critical hardware capabilities:

1. SR-IOV Capable NICs: The network card firmware must support creating Virtual Functions (e.g., Intel X710, Mellanox ConnectX).
2. IOMMU (Input-Output Memory Management Unit): The motherboard and CPU must support VT-d (Intel) or AMD-Vi. IOMMU ensures that when a Virtual Machine is given direct access to a Virtual Function, that VF can only perform Direct Memory Access (DMA) to the specific RAM allocated to that VM. Without IOMMU, a compromised VM could use the physical network card to read the host hypervisor’s memory.

Step 1: Enabling IOMMU in the Kernel

Before configuring the network card, you must enable IOMMU in the Linux bootloader.

Open your GRUB configuration file (e.g., /etc/default/grub) and append the IOMMU parameters to GRUB_CMDLINE_LINUX.

For Intel CPUs:

GRUB_CMDLINE_LINUX="... intel_iommu=on iommu=pt"

For AMD CPUs:

GRUB_CMDLINE_LINUX="... amd_iommu=on iommu=pt"

(The iommu=pt flag enables “pass-through” mode, which improves host performance by only enforcing IOMMU isolation for devices explicitly passed to VMs).

Update GRUB and reboot the hypervisor:

sudo grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
sudo reboot

After rebooting, verify that IOMMU is active by checking the kernel ring buffer:

dmesg | grep -e DMAR -e IOMMU

Step 2: Instantiating Virtual Functions (VFs)

Once IOMMU is enabled, identify your SR-IOV capable network interface (e.g., enp3s0f0). You instantiate Virtual Functions by writing the desired number of VFs to the PCI device’s sysfs interface.

To create 4 Virtual Functions on enp3s0f0:

echo 4 | sudo tee /sys/class/net/enp3s0f0/device/sriov_numvfs

You can verify the creation of the VFs using the ip link command:

ip link show dev enp3s0f0

The output will display the Physical Function along with the new vf 0, vf 1, vf 2, and vf 3, each with their own generated MAC addresses.

Step 3: Making VFs Persistent Across Reboots

Writing to sysfs is volatile; the VFs will disappear upon reboot. To make them persistent, it is best practice to configure them via a systemd service or via a udev rule.

Create a udev rule (e.g., /etc/udev/rules.d/99-sriov.rules):

ACTION=="add", SUBSYSTEM=="net", ENV{ID_NET_NAME_MAC}=="enp3s0f0", ATTR{device/sriov_numvfs}="4"

Step 4: Assigning the VF to a KVM Virtual Machine

With the VFs created on the host, you can now attach one to a running or stopped KVM Virtual Machine. The cleanest way to do this in libvirt is to define a “Hostdev” network.

Create an XML file named sriov-net.xml:

<network>
  <name>sriov-pool</name>
  <forward mode='hostdev' managed='yes'>
    <pf dev='enp3s0f0'/>
  </forward>
</network>

Import and start the network in libvirt:

virsh net-define sriov-net.xml
virsh net-autostart sriov-pool
virsh net-start sriov-pool

Now, edit the XML of your Virtual Machine (e.g., virsh edit my-vm) and add an interface pointing to this SR-IOV pool:

<interface type='network'>
  <source network='sriov-pool'/>
  <mac address='52:54:00:11:22:33'/>
  <model type='virtio'/> <!-- Ignored for hostdev, but required for XML validation -->
</interface>

When you start the VM, libvirt will automatically detach one of the available VFs from the host kernel, bind it to the vfio-pci driver, and pass it directly into the VM. Inside the guest OS, the network card will appear as a physical PCI device, completely unaware that it is virtualized.

Step 5: Configuring Hardware VLAN Tagging

Because SR-IOV bypasses the host software bridge, you cannot use standard Linux VLAN tagging (802.1Q) on the host bridge to isolate tenant traffic. Instead, you must instruct the physical network card firmware to tag the packets for the specific VF.

To force vf 0 to tag all outgoing packets with VLAN 100 in hardware:

sudo ip link set enp3s0f0 vf 0 vlan 100

Conclusion

Linux SR-IOV represents the pinnacle of virtualized network performance. By leveraging IOMMU hardware isolation and slicing physical network ASICs into discrete Virtual Functions, administrators can achieve bare-metal throughput and microsecond latency for KVM guests, entirely eliminating the software overhead of traditional hypervisor bridging.