Why Is InfiniBand Used in HPC?

In a high-performance computing (HPC) cluster, real-time communication between kernels is required for multi-node parallel computing, which demands a computing network with high bandwidth and low latency. However, the traditional communication method of Ethernet TCP/IP involves transferring data to the operating system kernel before copying it to the memory of the target computer, resulting in high latency and CPU resource consumption. RDMA (remote direct memory access) can transfer data directly from one computer’s memory to another without going through the operating system kernel. This direct memory access method can avoid interference from the operating system kernel, thereby reducing latency and CPU usage.

Protocols that Support RDMA:

The current network protocols that support RDMA are:

InfiniBand (IB)

A new generation network protocol that supports RDMA from the beginning. As this is a new networking technology, it requires network cards and switches that support this technology.

RDMA over Converged Ethernet (RoCE)

A network protocol that allows RDMA to be performed over Ethernet. This allows RDMA to be used on a standard Ethernet infrastructure (switches), but requires a special NIC that supports RoCE.

Internet Wide Area RDMA Protocol (iWARP)

A network protocol that allows RDMA to be performed over TCP. This allows RDMA to be used on a standard Ethernet infrastructure (Ethernet switches), but requires a NIC that supports iWARP.

InfiniBand Overview

The original intention of the development of InfiniBand was to network server buses, so in addition to having strong network performance, InfiniBand also directly inherits the high bandwidth and low latency of buses. The DMA (Direct Memory Access) technology used in bus technology is implemented in InfiniBand in the form of RDMA (Remote Direct Memory Access). RDMA services can transfer data across networks between processors, and data is transmitted directly between temporary memories without the need for operating system intervention or data copying. RDMA reduces latency by reducing the need for bandwidth and processor overhead, which is achieved by deploying a reliable transport protocol in the hardware of the NIC and supporting zero-copy network technology and kernel memory bypass.

The InfiniBand system consists of channel adapters (CA), switches, routers, cables, and connectors. CAs are divided into Host Channel Adapters (HCA) and Target Channel Adapters (TCA). In principle, IBA switches are similar to other standard network switches, but they must meet the high-performance and low-cost requirements of InfiniBand. InfiniBand routers are used to segment large networks into smaller subnets and connect them with routers. HCA is a device point that connects IB end nodes, such as servers or storage devices, to the IB network. TCA is a special form of channel adapter that is often used in embedded environments, such as storage devices.

Key Advantages of InfiniBand

There are several advantages of InfiniBand, including standardized protocols, high speed, Remote Direct Memory Access (RDMA), transport offload, network partitioning, and Quality of Service (QoS).

Standardization

The IBTA, established in 1999 with over 300 members, jointly designed the IBA open standard. IBA supports storage protocols such as SCSI RDMA Protocol (SRP) and iSCSI Extensions for RDMA (iSER).

Speed

InfiniBand’s transmission rate has reached 168Gbps (12xFDR), far exceeding the 10Gbps of 10 Gigabit Ethernet and the 100Gbps of 100 Gigabit Ethernet.

Memory

InfiniBand-enabled servers use Host Channel Adapters (HCA) to convert the protocol to the PCI-X or PCI-E bus inside the server. HCAs have RDMA capabilities, which transfer data directly in server memory through a virtual addressing scheme without involving the kernel of the operating system, making it suitable for clusters.

Transport offload

RDMA implements transport offload, which routes data packet processing from the operating system to the chip level, greatly reducing the processor’s processing load.
Network partitioning: Supports programmable partition keys and routing.

Quality of Service

Multilayer QoS guarantees to meet the diverse QoS requirements of service requesters.

Conclusion

With the continuous development of InfiniBand, it has become the best solution to replace Gigabit/10 Gigabit Ethernet and will undoubtedly become the preferred high-speed interconnect network. Its integration with Ethernet and iSCSI will become even closer. The IBTA has made predictions for the development of InfiniBand, indicating that InfiniBand FDR, EDR, and HDR will have a rapid increase in market demand in the next three years. InfiniBand also has broad application prospects in GPU, solid-state drives, and clustered databases in the future.



Related Resources:
What Is InfiniBand and How Is It Different from Ethernet?
NADDOD High-Performance Computing (HPC) Solution
InfiniBand Network Technology for HPC and AI: In-Network Computing
InfiniBand Trend Review: Beyond Bandwidth and Latency
What Is InfiniBand Network and Its Architecture?