Silicon-Validated 800G Ethernet Solution

Background Introduction

According to NADDOD's understanding, Broadcom launched the Tomahawk 51.2 Tbps switch chip in August 2022, marking the arrival of the era of 800G Ethernet.

 

The development of high-speed Ethernet can be traced back to 2014 when Arista, Qualcomm, Microsoft, Mellanox, and Google jointly established the Ethernet Alliance, now known as the "Ethernet Technology Alliance." Since then, more than 45 members have adopted this technology.

 

As a leading provider of comprehensive optical network solutions, NADDOD has conducted research for many years and found that the development of 200G Ethernet, 400G Ethernet, and the current 800G Ethernet is driven by the urgent demand for processing and transmitting high-performance workloads in various applications such as high-performance computing, 5G, and deep learning, as well as the total bandwidth required to transfer all these application data between the cloud and the edge.

 

In 2020, the Ethernet Technology Alliance introduced the 800G specification, which has been approved by the IEEE 802.3 standards committee and working group. By using high-speed interfaces, hyperscale data centers can support the continuously growing Ethernet data rates in SoCs designed for computers, switches, retimers, NICs, optical modules, and other devices. The Ethernet standard provides a reference framework for the interoperability interfaces required to design and manufacture such SoCs and their supported devices.

 

The 800G Ethernet controller and PHY solutions can meet the scalable and high-data-rate connectivity requirements of data-intensive applications in various ways. This article introduces a reliable 800G Ethernet implementation solution that utilizes MAC, PCS, and PHY IP, which have undergone silicon validation.

 

Demand for 800G Ethernet

In the past 12 years, switches have undergone continuous development, with their processing capacity growing from 640G to 102.4T. A single channel can support Ethernet speeds ranging from 10G to 100G, and even higher. The number of channels in switches has also increased from 64 to 512. At the same time, the pluggable connectors have evolved from QSFP+ to QSFPDD800/OSFP, as shown in the diagram below. 800G Ethernet represents the latest generation of 51.2T switches and the core technology behind the 800G pluggable modules that support them.

 

800G Ethernet chips with DAC (Direct Attach Copper) can be used for both in-rack communication and as interfaces for standard pluggable optical devices. The standards for 400G pluggable devices have been widely adopted in the optical architecture of the HPC (High-Performance Computing) field, enabling them to deliver Ethernet speeds of up to 400G after being inserted into line card panels.

 

It is crucial that the design of 800G Ethernet adheres to industry-standard form factors and connector interfaces. Furthermore, interoperability must be ensured when replacing other pluggable devices. Comprehensive interoperability provides reliable proof of concept and keeps the design ahead of the game. Once the standard is approved, it will be in a leading position.

 

Next-generation 800G modules will double the bandwidth for each port. The statistics presented at the 2022 OCP (Open Compute Project) Summit, shown in the diagram below, indicate that according to predictions by NADDOD's professional team, 800G pluggable devices will surpass 400G devices by 2025. With the doubling of channel counts, data centers can support 800G Ethernet without completely changing their port configurations, making the transition easier.

 

Analyzing the Powerful Implementation of 800G Ethernet

The implementation of 800G Ethernet is based on 8 channels, with each channel achieving a speed of 100Gb/s. This 800G architecture consists of a new Media Access Control (MAC) and Physical Coding Sublayer (PCS) that essentially reuses two sets of existing 400G Ethernet logic and distributes the data across eight 106.25 Gbps physical channels. By reusing the 400G Ethernet PCS logic, the forward error correction module is preserved, enabling easy compatibility with existing physical layer specifications.

 

However, SoC (System-on-Chip) designers need to develop chips to support 800G Ethernet in order to meet the requirements of all the transmitting and receiving data path units, achieving the lowest latency and higher bandwidth while minimizing issues of excessive power consumption and cost. Maintaining backward compatibility with slower speeds is also crucial, as it ensures that 800G Ethernet and higher-speed Ethernet can be seamlessly adopted and integrated into existing data centers.

 

To provide this capability simultaneously in 200G/400G/800G networks, both the logical and physical domains face multiple challenges. It is highly difficult to achieve faster clock speeds, parallel paths, and complex signaling requirements while meeting the demands for minimum latency, power consumption, and area. To ensure low latency and a low retransmission rate, efficient forward error correction (FEC) codes are required to compensate for higher error rates caused by faster communication speeds.

 

The diagram below illustrates a reference schematic of an 800G chip, featuring an 8-channel 100G SerDes, 800G PCS, MAC, test logic, and application interfaces. One of the key factors in successfully implementing the chip is to have application software interfaces with robust testability and debug capabilities.

 

800G Ethernet implementation is designed to serve as the network interface for next-generation data center networks and other network interfaces. In order to keep up with the increasing demands for CPU, bus, and storage bandwidth, rack servers or blade servers must support 800Gb/s total throughput from Network Interface Cards (NICs). To achieve this, the speed of SerDes needs to reach 100G per channel, and advanced wiring techniques need to be employed. Such techniques require dual-axial copper cables or optical fibers to support speeds of 100 Gb/s per physical channel.

 

As the endpoint bandwidth increases, the uplink links of Top-of-Rack (TOR) switches need to transition from 8-channel, 400 Gb/s SerDes to 8-channel, 800 Gb/s SerDes, ideally while maintaining the same per-channel output capacity and coverage. To maintain overall system performance in scenarios with simultaneous switching and to compensate for increasing crosstalk, designers have started to significantly increase margins in their designs. Carefully optimized package signal position maps can help alleviate any IR drop issues, and package stacking and the flow of signals on the die and package become even more critical than ever.

 

The completion of a 51.2T switch requires multiple vertical and horizontal 800G Ethernet chips. These reusable chip blocks in two directions can be constructed and replicated at various edges of the chip to minimize the overall chip perimeter. Package routing, signal integrity, and power integrity analysis lay the foundation for the basic layout planning of 800G Ethernet chip blocks.

 

The hardening of an 800G Ethernet test chip involves expertise in various domains, aiming to optimize chip edges, SerDes, PCS, and MAC designs. Comprehensive understanding of end-to-end data paths is required for tasks such as retiming, applying pipeline techniques, or optimizing latency, power, and area. Lastly, this also involves front-end and back-end integration using a SoC-level full flow from RTL to GDS, as well as close collaboration using EDA tools for tapeout.

 

The diagram below illustrates a use case of an end-to-end 800G implementation, including a 51.2T switch, 800G pluggable devices, wide-range DAC or active copper cables (for in-rack communication), and 800G pluggable optical devices (for extended coverage).

 

Conclusion

In addition to silicon IP that supports target area, latency, performance, power, and signal integrity, SoC designers also need all necessary documentation and deliverable products to achieve fast integration. With a silicon-validated 800G Ethernet implementation solution, demand users can refer to this solution to successfully realize their own chips and accelerate the Ethernet SoC design implementation process.