rdma-verbs

RDMA Verbs

Purpose

Guide agents through RDMA programming with libibverbs: one-sided vs two-sided operations, RC/UC/UD transports, device setup (ibv_get_device_list, protection domains, memory registration, completion queues, queue pairs), work requests and completions, RoCE vs InfiniBand, perftest benchmarking, and Rust rdma-sys bindings.

When to Use

• Building ultra-low-latency storage or database networking
• Bypassing CPU for remote memory access (one-sided RDMA)
• Setting up RoCE on Ethernet fabrics
• Benchmarking network fabric with perftest tools
• Integrating RDMA into MPI or custom RPC systems
• Debugging RDMA connection and completion errors

Workflow

1. RDMA concepts

RDMA stack
├── Application (libibverbs)
├── Kernel RDMA driver (mlx5, rdma_rxe)
├── NIC/HCA hardware
└── Fabric (InfiniBand or RoCE/Ethernet)

Operation types
├── Two-sided: Send/Recv (both sides participate)
└── One-sided: RDMA Read/Write (remote CPU not involved)

Transports:

Type	Reliable	Connection	Use
RC (Reliable Connected)	Yes	1:1 QP pair	General purpose
UC (Unreliable Connected)	No	1:1	Multicast-like
UD (Unreliable Datagram)	No	Many:Many	MPI, discovery

2. Device discovery

# List RDMA devices
ibv_devices
ibv_devinfo

# RoCE link status
rdma link show
ibstat

# Perftest prerequisites
modprobe ib_umad

3. Minimal libibverbs setup

#include <infiniband/verbs.h>
#include <stdio.h>
#include <stdlib.h>

int main(void) {
    int num_devices;
    struct ibv_device **dev_list = ibv_get_device_list(&num_devices);
    if (!dev_list || num_devices == 0) {
        fprintf(stderr, "No RDMA devices\n");
        return 1;
    }

    struct ibv_context *ctx = ibv_open_device(dev_list[0]);
    struct ibv_pd *pd = ibv_alloc_pd(ctx);

    char buf[4096];
    struct ibv_mr *mr = ibv_reg_mr(pd, buf, sizeof(buf),
        IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE);

    struct ibv_cq *cq = ibv_create_cq(ctx, 10, NULL, NULL, 0);

    struct ibv_qp_init_attr qp_attr = {
        .send_cq = cq,
        .recv_cq = cq,
        .cap = { .max_send_wr = 10, .max_recv_wr = 10,
                 .max_send_sge = 1, .max_recv_sge = 1 },
        .qp_type = IBV_QPT_RC,
    };
    struct ibv_qp *qp = ibv_create_qp(pd, &qp_attr);

    printf("QP num %u, MR lkey %u rkey %u\n",
           qp->qp_num, mr->lkey, mr->rkey);

    ibv_destroy_qp(qp);
    ibv_dereg_mr(mr);
    ibv_destroy_cq(cq);
    ibv_dealloc_pd(pd);
    ibv_close_device(ctx);
    ibv_free_device_list(dev_list);
    return 0;
}