.png)
RDMA
RDMA, known as Remote Direct Memory Access, is a technology used for high-performance network communications.RDMA technology allows a computer to directly access the memory of a remote computer without the need to copy data between the local and remote computers.
Compared with traditional network communication methods, RDMA technology has the advantages of lower latency, higher bandwidth and lower CPU utilization, which can significantly improve the performance and efficiency of network communication. Currently the main network protocols that support RDMA are:
1. InfiniBand (IB): IB is a set of high-performance network communication standards developed and promoted by the InfiniBand Trade Association (IBTA). version 1.0 of the IB architecture specification was released in 2000, which natively supports RDMA and is probably the earliest RDMA implementation. However, the IB Architecture is not compatible with Ethernet and requires the purchase of an IB switch in addition to an IB-enabled NIC for deployment.
2, RoCE: RDMA over Ethernet, RoCE is an Ethernet-based RDMA standard, also developed by IBTA, which allows remote direct memory access in Ethernet networks. there are two versions of RoCE: RoCE v1 and RoCE v2. RoCE v1 is a link-layer protocol (L2) that enables communication between any two hosts in the same broadcast domain. RoCE v2 is a network layer protocol (L3), which means that RoCE v2 packets can be routed across three layers, so networking can be much larger.
3. Internet Wide Area RDMA Protocol (iWARP): i.e. RDMA over TCP, a network protocol that allows RDMA to be performed over TCP. This allows RDMA to be used on standard Ethernet infrastructures (switches), except that the NIC is required to be a NIC that supports iWARP (if CPU offload is used).
As shown in the figure below:
IB, RoCE and iWARP are all implementations of RDMA technology that can be used in high performance computing, storage, data centers and other areas. They are compared with each other with the following advantages and disadvantages:
Input Costs: InfiniBand has higher input costs and requires specialized hardware and network equipment; RoCE can be deployed using existing Ethernet equipment at lower costs; iWARP requires specialized NICs and protocol stacks, with costs falling in between.
Ease of Use: RoCE and iWARP are easier to use than InfiniBand because they are both based on the Ethernet stack with good interoperability and compatibility, whereas InfiniBand requires the use of a dedicated stack and drivers.
Bandwidth: All three have high bandwidth, but InfiniBand has the highest bandwidth, which can reach up to 400 Gb/s, while RoCE and iWARP generally have bandwidths between 40 Gb/s and 100 Gb/s.
Latency: InfiniBand has the lowest latency, which can reach below 1 microsecond; RoCE and iWARP have lower latency, but usually higher than InfiniBand.
CPU Occupancy: InfiniBand and RoCE have lower CPU occupancy because they both use hardware offloads to reduce the CPU burden; iWARP has higher CPU occupancy because it uses software offloads for RDMA.
Scalability: RoCE and iWARP are more scalable than InfiniBand because they can be scaled using existing Ethernet infrastructure, whereas InfiniBand requires specialized hardware and network equipment.
Overall, the choice of technology depends on factors such as specific application scenarios, input costs and performance requirements. If the application scenario requires very high performance, InfiniBand can be chosen; if cost is the main consideration, RoCE can be chosen; if good interoperability and compatibility are required, RoCE or iWARP can be chosen.
RoCE cards, because they can be deployed using existing Ethernet devices, have gained the attention of more and more manufacturers and are developing most rapidly, and have become the mainstay of shipments for the above three types of cards. EXCEL has released storage acceleration cards and DPUs that support the RoCEv2 protocol.