400G Data Center Network Solution Using Switch, SmartNIC and Optical Transceivers

NADDOD Jason Data Center Architect Mar 15, 2023

The 400G data center network market is developing rapidly. Switch chips with greater throughput, higher-speed optical transceiver modules, and explosive data growth are working together to speed up the development of data center networks. In addition, with the vigorous development of 5G and the rapid growth of video-based data transmission, data centers need to upgrade to higher capacity to meet the rapidly growing demand for data and bandwidth-intensive applications.

All of these factors are driving continued growth in data center traffic and the need for high-capacity data center networking solutions. As the capacity demand of data-intensive applications surges far beyond the current high-speed transmission capability, as an evolving technology, 400G is bound to be promising for data center networks. It has relatively lower operation and maintenance costs (OpEx) and smaller footprint, and can better meet the urgent demand for fiber optic connections.

400G Ethernet Network Overview

In the past few decades, we have witnessed the transformation of Ethernet speed from 1G, 10G/40G to 25G/100G. The Ethernet technology continues to innovate to achieve higher data rates. Before introducing solutions and products of 400G network, we need to understand how 400G is defined and used in data center networking conversations.

400G: 100G/400G and other network speed descriptions refer to the interface serdes rate, that is, the transmission capacity of a single channel. It is usually used to verbally describe the transmission rate of a single interface. Depending on the context, it can sometimes be used interchangeably with 400GbE and 400Gb/s.

400GbE: It refers to the next-generation capacity rate that can be transmitted through a single link in an Ethernet interface. Its 400GbE physical layer, management parameters and media access control (MAC) specifications must comply with the IEEE 802.3bs standard.

400Gb/s: It refers to the speed of data transmission, usually describing the highest transmission rate supported by the device port, or the transmission rate of optical transceivers.

400G Network Working Principle

We may be familiar with the 100G network, which uses 4 NRZ modulated 25G signals to complete 100G transmission on a single interface. There are usually two ways to increase the transmission rate of Ethernet: to increase the transmission rate of a single channel, and to increase the number of channels. And 400G is a technology that not only doubles the single-channel transmission, but also doubles the number of channels, which transmits 8 channels of 50G signals through PAM4 modulation technology to achieve 400G transmission (QSFP-DD).
PAM4 Transmission Principle
PAM4 Transmission Principle

Advantages of Deploying 400G Networks in Data Centers

Higher transmission density: Taking a 100G switch as an example, the port density of the NVIDIA 100G switch SN2700/3700C is 32x 100G per U, while the 400G switch SN4700 has 32x 400G QSFP-DD ports on the 1U panel, which increases the port density by four times in the same height.
Lower transmission power consumption: Based on the policies and regulations on environmental protection in different regions, how to reduce data centers’ carbon footprint has become primary. Compared with 100G, the consumption of 400G is lower, and it also complies with the needs of power consumption reduction and environmental protection.

Strong scalability: 100G QSFP28 ports generally support two transmission modes: 100G to 100G and 100G to 4x25G. 400G switches such as NVIDIA SN4700 support various transmission modes, like 400G to 400G, 400G to 2x200G, 400G to 4x100G, 400G to 8x50G, etc., which can meet different network architectures.
Port Modes Supported by NVIDIA SN4700 Switches
Port Modes Supported by NVIDIA SN4700 Switches

Data Center 400G Network Trends

Trend 1: QSFP-DD.

QSFP-DD is the first type to allow all connectivity cables and modules to use a consistent common form factor, no matter it is copper cable or long-distance fiber cable, without exception. The QSFP-DD Multisource Agreement (MSA) group of 65 organizations has also published version 4.0 of the Common Management Interface Specification (CMIS) for the QSFP-DD form factor. As 400G adoption grows, QSFP-DD will provide the data center market with the smallest 400 modules and the highest port bandwidth density, and CMIS helps ensure that 400G QSFP-DD modules are backward compatible with all former QSFP-based transceivers, including widely deployed 40G/100G/200G products. The powerful performance of QSFP-DD has been widely recognized, and it is most suitable for use as a high-density, high-speed hot-pluggable module in a 400G data center network environment.

Trend 2: Remote Direct Memory Access (RoCE) over Converged Ethernet.

The 400G data network is the key to accelerating the adoption of RoCE technology in data centers. RDMA supports zero-copy networking, which enables network adapters (NICs) and application memory to transfer data directly without copying data between application memory and the operating system’s data buffer. It is particularly worth mentioning that the RoCEv2 protocol supports RDMA across L3 boundaries.

Trend 3: SmartNICs.

The emergence of SmartNIC plays a key role for 400G to meet the new massive bandwidth demand of data centers. The latest versions of SmartNICs offer two 25G/50G/100G/200G ports or one 400G port, along with 50G PAM4 SerDes capable Ethernet connectivity and PCIE 4.0 host connectivity. We believe SmartNICs are integral to increasing network programmability and accelerating application offload, which is critical to improving data center efficiency.

NADDOD Full-stack 400G Data Center Network Solution

Based on the understanding of the development trend of high-speed networks and breakthroughs in optoelectronic device technology, NADDOD provides customers with full-stack 400G data center network solutions.
NADDOD Data Center 400G Network Solution
NADDOD Data Center 400G Network Solution

400G Ethernet Switch

NADDOD uses SN4410/SN4700 on the switch side as leaf/spine switches respectively. The SN4000 series switches based on Spectrum-3 not only support up to 400G transmission rate, but also have advanced network virtualization technology, and support high-performance, single-channel VXLAN routing and IPv6/Multiprotocol Label Switching (MPLS) Segment Routing, and a powerful high-bandwidth data path for RoCE-based and GPUDirect-based applications.
NVIDIA Spectrum-3 SN4000 Switch
NVIDIA Spectrum-3 SN4000 Switch

400G SmartNIC Card

NADDOD uses NVIDIA ConnectX SmartNIC 400G intelligent network card on the NIC side. The SmartNIC uses NVIDIA ASAP2 (Accelerated Switching and Datagram Processing Technology®), which can accelerate network performance and reduce the CPU load when transmitting Internet Protocol (IP) datagrams, thus releasing more processor cycles to execute the application. With unrivaled RoCE performance, ConnectX SmartNICs provide efficient, high-performance Remote Direct Memory Access (RDMA) services for bandwidth- and latency-sensitive applications. You can also choose NVIDIA Bluefield DPU, which combines the power of ConnectX with programmable ARM cores and other hardware offload functions to provide powerful support for software-defined storage, networking, security and workloads management.
NVIDIA ConnectX SmartNIC
NVIDIA ConnectX SmartNIC

400G Transceiver Module

In the connection of the network, the connection between the server and the switch can choose to use NADDOD 200G QSFP-DD to 2x100G QSFP28 cable, and one SN4410 is connected to the 48x100G network card port. The connection of the switch can use NADDOD QDD-400G-SR8 400G SR8 QSFP-DD transceiver to connect two SN4700 switches.
NADDOD 400G Optical Transceiver
NADDOD 400G Optical Transceiver

NADDOD  provides optimal Ethernet and InfiniBand high-performance switches, intelligent network cards, AOC/DAC/optical module product combination solution according to different application scenarios of users. We provide more advantageous and valuable optical network products and overall solutions for data centers, high-performance computing (HPC), edge computing, artificial intelligence (AI) and other application scenarios, with low cost and excellent performance, greatly improving customers’ business acceleration capabilities.

Related Resources:
400G Optical Transceiver Module Types
Typical Connection Solutions of 400G Optical Transceivers and Cables
The Next Station of Data Center Networks—200G vs 400G
400G Optical Module Application Scenarios
400G QSFP-DD FR4 vs 400G QSFP-DD FR8
FAQs for 400G Transceivers and Cables