How to Use the rdma Command in Linux to Inspect and Manage InfiniBand and RoCE Devices

In modern high-performance computing (HPC), artificial intelligence (AI) model training clusters, and low-latency storage architectures (such as NVMe over Fabrics), standard TCP/IP networking introduces significant CPU overhead and latency due to operating system kernel context switching and memory buffer copying. To bypass these bottlenecks, enterprise infrastructure relies heavily on Remote Direct Memory Access (RDMA). RDMA enables network adapters to transfer data directly into or out of application memory across systems without involving either host’s operating system kernel. RDMA implementations typically deploy over dedicated InfiniBand fabrics or enterprise Ethernet via RoCE (RDMA over Converged Ethernet, v1 and v2).

Historically, system engineers inspected RDMA hardware using fragmented diagnostic tools from the legacy infiniband-diags package (such as ibstat or ibv_devinfo). However, modern Linux distributions unify RDMA subsystem management directly within the standard iproute2 suite through the rdma command. Mirroring the syntax structure of ip, bridge, and tc, the rdma utility provides an authoritative, scriptable interface to audit physical adapters, inspect link-layer state, monitor active Queue Pairs (QPs), and track hardware drop statistics. This comprehensive guide covers the Linux RDMA architecture, demonstrates the core command verbs of rdma, and explains how to diagnose fabric performance.

RDMA Architectural Foundations: InfiniBand vs. RoCE

RDMA technologies deliver microsecond-level latency and full wire-speed throughput by offloading protocol processing to hardware Host Channel Adapters (HCAs):

RDMA Technology	Physical Fabric	Network Layer / Encapsulation	Primary Use Case
Native InfiniBand (IB)	Dedicated InfiniBand switches & cabling	Native IB protocol with credit-based flow control	Supercomputing, MPI clusters, extreme low latency.
RoCEv1	Standard Ethernet (Layer 2)	Ethertype 0x8915 (non-routable across subnets)	Isolated single-switch high-speed storage clusters.
RoCEv2	Converged Ethernet (Layer 3)	UDP port 4791 encapsulation (fully routable)	Distributed AI clusters (NVIDIA GPU Direct), cloud scale.
Soft-RoCE (RXE)	Any standard Linux Ethernet NIC	Software emulation via the rdma_rxe kernel driver	Development, functional testing, and CI environments.

Inspecting RDMA Hardware Adapters with rdma dev

The rdma dev object manages physical RDMA hardware devices registered with the kernel’s OpenFabrics (OFA) subsystem. To view all active RDMA devices on your system, execute:

rdma dev show

On an enterprise server equipped with Mellanox/NVIDIA ConnectX adapters, the output displays registered hardware handles:

1: mlx5_0: node_type ca fw 28.39.1002 node_guid 0002:c903:002b:e340 sys_image_guid 0002:c903:002b:e340
2: mlx5_1: node_type ca fw 28.39.1002 node_guid 0002:c903:002b:e341 sys_image_guid 0002:c903:002b:e340

The key attributes include:

• Device Handle: The canonical subsystem identifier (e.g., mlx5_0).
• node_type: Indicates device class (ca for Channel Adapter / HCA, or switch).
• fw: The active firmware revision loaded on the network adapter.
• node_guid: The globally unique 64-bit hardware identifier of the RDMA adapter.

Inspecting Detailed Adapter Capabilities

To view supported hardware capabilities, memory registration maximums, and protocol features, add the -d (detail) flag:

rdma dev show -d

For automated orchestration scripts, rdma supports native JSON output via the -j flag:

rdma -j dev show

Auditing Port Status and Network Mapping with rdma link

Unlike standard Ethernet interfaces, an RDMA adapter may host multiple physical ports or bind to existing Linux network devices (netdevs). The rdma link object exposes these port bindings:

rdma link show

A typical output mapping RDMA ports to Linux network interfaces looks like this:

link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev enp1s0f0np0
link mlx5_1/1 state DOWN physical_state DISABLED netdev enp1s0f1np1

This command immediately clarifies the relationship between the RDMA device (mlx5_0/1) and the standard Linux network device (enp1s0f0np0). To establish an RDMA session, both state must report ACTIVE and physical_state must report LINK_UP.

Auditing Memory Regions and Queue Pairs with rdma res

RDMA communication operates via Queue Pairs (QPs), which consist of a Send Queue and a Receive Queue, coupled with Completion Queues (CQs) and pinned Memory Regions (MRs). The rdma res object allows administrators to audit hardware resource utilisation across all processes:

rdma res show

Terminal outputs summary statistics per adapter:

link mlx5_0/1 pd 12 cq 48 qp 32 cm_id 8 mr 64 ctx 14

To inspect individual Queue Pairs, identify their current state, and reveal the exact Linux process (PID) consuming RDMA resources, query the QP resource sub-object:

rdma res show qp

Terminal displays the active Queue Pair table:

link mlx5_0/1 lqpn 248 rqpn 512 type RC state RTS rq-psn 12401 sq-psn 49202 pid 4092 comm nvmeof_tgt
link mlx5_0/1 lqpn 250 rqpn 514 type RC state RTS rq-psn 18942 sq-psn 31092 pid 4092 comm nvmeof_tgt

This reveals critical diagnostic data:

• lqpn / rqpn: Local and Remote Queue Pair Numbers.
• type: The transport service type (e.g., RC for Reliable Connection, UD for Unreliable Datagram).
• state: The operational QP state (RTS indicates Ready To Send; INIT or RTR indicate incomplete handshakes).
• pid / comm: The exact process ID and process name holding the connection (e.g., nvmeof_tgt).

Tracking Fabric Performance and Drop Counters with rdma stat

Hardware packet drops in RDMA environments severely degrade throughput because RDMA assumes a lossless network fabric. The rdma stat command provides access to hardware performance counters:

rdma stat show

To monitor counters on a specific link continuously:

rdma stat show link mlx5_0/1

Inspect output counters for indicators of network congestion:

• roce_adp_retrans: Adaptive retransmission events caused by dropped packets or sequence gaps.
• roce_slow_restart: Flow control throttling incidents where Priority Flow Control (PFC) or Explicit Congestion Notification (ECN) paused transmissions.
• rx_prio_pause: The number of PFC pause frames received from the upstream network switch.

Configuring System-Wide RDMA Network Namespaces

In containerised environments running Docker, Podman, or Kubernetes, RDMA devices can operate in one of two namespace sharing modes: shared (default, where devices appear across all network namespaces) or exclusive (where an RDMA device can be isolated inside a single container). To check and configure this behavior:

rdma system show

To enforce strict multi-tenant isolation, switch the system to exclusive mode:

sudo rdma system set netns exclusive

Once set to exclusive mode, administrators can assign specific RDMA interfaces directly into a container’s network namespace using rdma dev set mlx5_0 netns <PID>.

Mastering the rdma command within iproute2 provides Linux system engineers with an indispensable, modern administrative framework to validate connectivity, audit hardware resources, and troubleshoot latency in mission-critical RDMA deployments.