How to Migrate Data Centers to 400G/800G?

The adoption of cloud infrastructure and services is driving the demand for higher bandwidth, faster speeds, and lower latency performance. Advanced switch and server technologies are forcing changes in cabling and architecture. Regardless of the market or focus of your facility, you need to consider the changes in enterprise or cloud architecture that may be necessary to support new requirements.

 

This means understanding the trends driving the adoption of cloud infrastructure and services, as well as emerging infrastructure technologies that enable your organization to meet new requirements.

 

When planning for the future, the following points need to be considered:

 

1. Evolution of Data Center Landscape

1.1 Global Data Usage

Of course, at the core of these changes are global trends reshaping consumer expectations and the demand for more and faster communication, such as:

 

• Explosive growth in social media traffic
• Deployment of 5G services achieved through large-scale small-cell densification
• Accelerated deployment of the Internet of Things (IoT) and Industrial IoT (IIoT)
• Shift from traditional office work to remote options

 

1.2 Growth of Hyperscale Providers

Globally, there may be fewer than a dozen true hyperscale data centers, but their impact on the overall data center landscape is enormous. According to research, in just one year in 2020, the total online time worldwide reached 1.25 billion years. Approximately 53% of network traffic was transmitted and processed through hyperscale data centers.

 

1.3 Hyperscale Collaboration with Multi-Tenant Data Center (MTDC)/Co-Location Facilities

As the demand for lower latency performance increases, hyperscale and cloud-scale providers are striving to extend their presence closer to end-users/endpoint devices. Many are partnering with MTDC or hosted data centers, positioning their services at the so-called network "edge." When the edge is physically closer, the value of new low-latency services is amplified by lower latency and network costs.

 

In the hyperscale arena, MTDC and hosted facilities are adjusting their infrastructure and architecture to support the growing scale and traffic demands of increasingly typical hyperscale data centers. At the same time, these largest data centers must continue to flexibly accommodate customer requests for cross-connectivity with cloud providers' entry points.

 

1.4 Spine-Leaf and Fabric Mesh Networks

The demand for low latency, high availability, and ultra-high bandwidth applications is not limited to hyperscale and hosted data centers. All data center facilities now need to reconsider their ability to handle the growing demands of end-users and stakeholders. In response, data center managers are quickly turning to fiber-dense mesh architecture networks. By enabling arbitrary connections, using fiber-dense backbone cables, and introducing new connectivity options, network operators can deliver higher channel speeds. This upgrade supports higher data transfer rates as operators prepare to transition to speeds of 400G per second. This technological upgrade meets the growing demand for internet data, providing faster and more reliable network services. Additionally, these new technologies enhance the transmission and processing of high-volume data, further improving network efficiency and performance.

 

1.5 Enabling Artificial Intelligence (AI) and Machine Learning (ML)

Furthermore, some large data center providers driven by IoT and smart city applications are turning to artificial intelligence and machine learning to help create and refine data models that support near real-time edge computing capabilities. These technologies not only open up possibilities for new application realms (think commercially viable autonomous vehicles) but also require massive datasets (often referred to as data lakes), substantial computational power within the data center, and sufficiently large pipelines to drive the refinement of models at the edge when needed.

 

1.6 Timing the Migration to 400G/800G

If the history of data center development has taught us anything, it's the speed of change. Whether it's bandwidth, fiber density, or channel speed, the trend is towards exponential acceleration, and the transition to 400G is closer than you might imagine. If you're feeling uncertain, just add up the number of 10G (or higher) ports you currently support and imagine upgrading them to 100G, and you'll realize that the demand for 400G (or even higher rates) is not far off.

 

2. What is Driving the Transition to Higher Speeds?

The data center industry is at a turning point. The adoption of 400G is happening rapidly, but soon 800G is expected to surpass 400G in terms of speed. As expected, there is no simple answer to the question of "what is driving the transition to 400G?" There are multiple factors at play, many of which are intertwined. As channel speeds increase, new technologies can drive down the cost per bit.

 

The latest data indicates that 100G channel speeds will be combined with octal switch ports, starting in 2022, to bring 800G options to the market. However, these ports are being utilized in multiple ways, as shown by Light Counting data, as seen in Figure 6 of their optical communications market report. In this data, 400G and 800G are primarily decomposed into 4X or 8X 100G. It is these breakthrough applications that have become early driving factors for these new optical solutions.

 

3. The Transition to 400G

3.1 The Four Pillars of the 400G/800G Migration

When you start considering the "nitty-gritty details" of supporting the transition to 400G, it's easy to get overwhelmed by all the moving parts involved. To help you better understand the key variables to consider, we have divided them into four main areas:

 

• Increasing switch port density
• Optics transceiver technology
• Connector options
• Cabling advancements

 

These four areas collectively represent important components of the migration toolkit. Use them to fine-tune your migration strategy to meet your current and future needs.

 

3.2 Higher Switch Port Density (Radix) - The End of ToR?

As serialization/deserialization (SERDES) provides electrical I/O for switching ASICs at speeds of 10G, 25G, and 50G, the switching speeds are increasing. Once IEEE802.3ck becomes an approved standard, SERDES will also reach 100G. Switching ASICs are also increasing their I/O port density, also known as radix. Higher radix ASICs support more network device connections and offer the potential to eliminate Top-of-Rack (ToR) switches. This, in turn, reduces the total number of switches required for cloud networks (a data center with 100,000 servers can support a two-tier switching with a radix of 512). Higher radix ASICs mean lower capital expenditure (fewer switches), lower operational expenditure (less power and cooling needed for the switches), and improved network performance through lower latency.

 

Closely related to the increase in radix and switching speeds is the shift from Top-of-Rack (ToR) topology to Middle-of-Row (MoR) or End-of-Row (EoR) configurations, along with the benefits of structured cabling methods suitable for many connections between intra-server and MoR/EoR switches. The use of new high-radix switches requires the ability to efficiently manage a large number of server attachments. This, in turn, necessitates new optical transceivers and structured cabling, such as those defined in the IEEE802.3cm standard. The IEEE802.3cm standard supports the advantages of pluggable transceivers that can be used with high-speed server networks in large data centers, defining eight host connections for a QSFP-DD transceiver.

 

3.3 Transceiver Technology

Just as the adoption of 100G was driven by the adoption of the QSFP28 form factor, which provided high density and low power consumption, new transceiver form factors are enabling the leap to 400G and 800G. Current SFP, SFP+, or QSFP+ optical devices are sufficient to achieve 200G link speeds. However, doubling the density of transceivers is necessary to make the jump to 400G.

 

QSFP Double Density (QSFP-DD) and Octal Small Form Factor Pluggable (OSFP) Multi-Source Agreement (MSA) enable a doubling of the ASIC electrical I/O connections in the network. This improvement not only allows for higher aggregated speeds but also increases the total number of ASIC I/O connections, enhancing network performance.

 

A 1U switch form factor with 32 QSFP-DD ports matches with 256 (32x8) ASIC I/O. This enables the establishment of high-speed links (8x100G or 800G) between switches while maintaining maximum connectivity for attached servers.

 

3.4 New Transceiver Formats

As OEMs aim to enter the optimal positions in hyperscale and cloud-scale data centers, the 400G optical market is driven by cost and performance considerations. In 2017, CFP8 became the first-generation 400G transceiver specification used for core routers and DWDM transport client interfaces. CFP8 transceivers are a 400G form factor specified by the CFP MSA. The transceiver size is slightly smaller than CFP2, and the optical devices support CDAUI-16 (16x25G NRZ) or CDAUI-8 (8x50G PAM4) electrical I/O. In terms of bandwidth density, they offer 8 times and 4 times the bandwidth density of CFP and CFP2 transceivers, respectively.

 

The "second generation" 400G form factor transceivers utilize QSFP-DD and OSFP. QSFP-DD transceivers are backward compatible with existing QSFP ports. They build upon the success of the existing optical transceivers QSFP+ (40G), QSFP28 (100G), and QSFP56 (200G).

 

OSFP, like QSFP-DD optical devices, supports 8 channels instead of 4, making it more compact and high-density compared to traditional QSFP and QSFP28 transceivers, with higher bandwidth and lower power consumption.

 

NADDOD is a leading provider of optical network solutions with a strong reputation in product quality and technical service capabilities. They offer 400G and 800G high-speed products and solutions, including optical transceivers and high-speed cables packaged in QSFP-DD and OSFP, as well as end-to-end stack solutions from connectors to networks to servers.

 

3.5 Modulation Schemes

Network engineers have long been using Non-Return to Zero (NRZ) modulation for 1G, 10G, and 25G, along with host-side Forward Error Correction (FEC) to achieve longer-distance transmission. To transition from 40G to 100G, the industry simply turned to parallelizing 10G/25G NRZ modulation while utilizing host-side FEC for longer distances. However, new solutions are needed for achieving speeds of 200G/400G and beyond.

 

As a result, optical network engineers have adopted four-level Pulse Amplitude Modulation (PAM4) for ultra-high bandwidth network architectures. PAM4 is the current solution for 200G/400G. This is largely based on IEEE 802.3, which has already completed new Ethernet standards supporting rates up to 400G (802.3bs/cd/cu) for both multi-mode (MM) and single-mode (SM) applications. There are multiple lane options available to accommodate different network topologies in large data centers.

 

More complex modulation schemes require infrastructure that can provide better echo loss and attenuation.

 

4. OSFP and QSFP-DD

Both OSFP and QSFP-DD form factors have the support of leading data center Ethernet switch vendors and enjoy significant customer support. Enterprises may prefer QSFP-DD as an enhancement to the current QSFP-based optical devices. With the introduction of OSFP-XD, OSFP appears to be driving the horizon of expansion by extending the number of channels to 16 and focusing on future channel rates of 200G.

 

For speeds up to 100G, QSFP has become the preferred solution due to its size, power, and cost advantages over duplex transceivers. QSFP-DD builds upon this successful foundation by offering backward compatibility, allowing QSFP transceivers to be used in switches with the new DD interface.

 

OSFP technology may be better suited for DCI optical links or those that require higher power and more optical I/O. Supporters of OSFP envision the emergence of 1.6T or even 3.2T transceivers in the future.

 

Co-packaged optics (CPO) offer an alternative path to achieving 1.6T and 3.2T. However, CPO would require a new ecosystem that can bring optical devices closer to switch ASICs, enabling higher speeds while reducing power consumption. This track is being developed within the Optical Internetworking Forum (OIF). The OIF is now discussing the technologies that may be most suitable for the "next rate," with many advocating for doubling it to 200G. Other options include more lanes, possibly 32 lanes, as some believe that more lanes and higher lane rates will be necessary to meet network demands at an affordable cost.

 

What is certain is that the wiring infrastructure must have built-in flexibility to support your future network topologies and link requirements. While astronomers have long believed that "every photon counts," as network designers aim to reduce the energy per bit to a few pJ/Bit, conservation at every level matters. High-performance cabling will help reduce network expenses.