High Performance Enterprise Data Center Network Design

With the development of business transformation, the construction of information systems, and the increasing demand for big data and cloud computing resources, the infrastructure network traffic of data centers continues to grow. According to statistics, the compound annual growth rate (CAGR) of global data center traffic from 2016 to 2021 was approximately 23%. Among them, the interconnection between data centers and the traffic within various regions of data centers accounted for about 85% of the total data center traffic. The continuous growth of traffic also drives the development of data center networks towards faster, higher-bandwidth, and lower-latency high-speed networks.

1. Advanced Data Center Network Architecture

The data center network architecture has evolved from the traditional core-aggregation-access architecture to the Spine-Leaf architecture, which fully utilizes network interconnection bandwidth, reduces multi-layer convergence ratios, and is easy to scale. In the Spine-Leaf architecture, each interconnection link has a bandwidth of 100G, and a reasonable network convergence ratio is designed based on business requirements to handle the internal traffic within and between PODs (Points of Delivery) in the data center. The three-layer underlay network in the Spine-Leaf architecture allows for decoupling of the core and access switches. When there is a bottleneck in traffic between the core switch and the aggregation switch or between the aggregation switch and the access switch, horizontal scaling can be achieved by adding uplink links and reducing convergence ratios, with little impact on bandwidth expansion. The overlay network deploys distributed gateways using EVPN-VXLAN technology, enabling flexible and elastic network deployments and allocation of network resources based on business needs.

Based on the design and deployment experience of Internet-scale data center networks, this solution adopts a spine-leaf network architecture and utilizes EVPN-VXLAN technology to achieve network virtualization, providing a flexible and saleable network infrastructure for upper-layer services. The data center network is divided into production network and office network, which are isolated and protected by domain firewalls, and connected to office buildings, laboratories, and regional center exits through network firewalls.

The core switches of the production network and the office network are respectively used for interconnection between PODs and connection to firewall devices, providing up to 1.6Tb/s of inter-POD communication bandwidth and 160G high-speed network egress capacity. The internal horizontal network capacity within each POD is 24Tb, providing high-bandwidth and ultra-low latency network support for high-performance computing clusters (CPU/GPU) and storage clusters, ensuring that the overall network does not experience packet loss due to network performance bottlenecks.

Building cabling is planned based on the Spine-Leaf architecture. The switches within each POD are connected using 100G interconnections and deployed in TOR (Top of Rack) mode, where 2-3 cabinets are combined into one TOR group. TORs are connected to Leafs through 100G links. Each POD's Leaf is divided into two groups and deployed in different network-occupied cabinets, achieving high reliability at the cross-cabinet level within the POD. The overall network structure is clear, and cable deployment and management are more efficient.

2. Future-oriented Equipment Selection

When designing and constructing a data center network, it is important to consider the technological and industry development as well as operational cost expenditure for at least the next five years. This will help optimize the utilization of existing data center resources to serve the core business of the enterprise.

The selection of network switches is a key factor in the overall design of the data center network. Traditional large-scale network designs often choose chassis-based devices to increase the overall capacity of the network system and provide limited scalability. However, this approach has limitations and risks, including:

• Chassis-based devices have limited overall capacity and can no longer meet the growing network scale requirements of data centers.
• Core chassis-based devices are deployed with dual connections, resulting in a high fault radius of up to 50%, which cannot effectively guarantee business security.
• Multi-chip architecture in chassis-based devices leads to serious bottlenecks in traffic processing capacity and network latency.
• Chassis-based device deployment is complex, and the diagnosis and troubleshooting of failures have long cycles, resulting in long business interruption times for upgrades and maintenance.
• To ensure future business expansion, chassis-based devices require reserved slots, which increase upfront investment costs.
• Later expansion is subject to limitations such as vendor binding and weakened bargaining power, significantly increasing the cost of future expansion.

Therefore, in the network equipment selection for this project, NVIDIA indicatively adopts a modular switch network architecture, where switches of different hierarchy levels are unified in model, making it easier for the maintenance team to become familiar with them quickly. This approach also provides operational flexibility for future network architecture adjustments, device reuse, and repair replacements.

By adopting the Spine-Leaf (CLOS) architecture combined with modular switch networking, the initial network investment (Total Cost of Ownership, TCO) is significantly reduced. The Spine-Leaf architecture allows for horizontal scalability. Even if a spine switch goes offline, it only affects 1/8 of the network bandwidth, and the business remains unaffected. For future expansion, additional switches and hierarchy levels can be added based on the scale requirements of the data center, thereby expanding the access capacity and backbone network switching capacity. The entire network can be procured and deployed on-demand, based on service, application, and business requirements.

3. Summary：NADDOD Accelerate Data Center Interconnection

Considering the current trends in business transformation and the increasing demand for big data, most data center network designs adopt advanced Spine-Leaf architecture and leverage EVPN-VXLAN technology to achieve network virtualization. This architecture provides high-bandwidth, low-latency network traffic and offers scalability and flexibility.

In the future, as the manufacturing costs of high-speed 200G/400G/800G optical modules and AOCs/DACs decrease, data center interconnect technologies will continue to evolve. In this context, NADDOD, as a leading provider of optical networking solutions, is committed to building a connected and intelligent world through innovative computing and networking solutions. NADDOD continuously delivers innovative, efficient, and reliable products, solutions, and services, offering optimal switch & AOC/DAC/optical module solutions for data centers, high-performance computing, edge computing, artificial intelligence, and other application scenarios. These solutions enhance customer business acceleration capabilities with low costs and outstanding performance.

NADDOD provides optical modules, AOCs, and DACs supporting both InfiniBand and Ethernet, with options ranging from 100G, 200G, 400G, to 800G, meeting the diverse needs of different data centers. These high-quality interconnect products offer faster and more reliable data transmission solutions for data centers. With NADDOD's professional technical team, rich implementation experience in various application scenarios, and excellent products and solutions, many customers trust and prefer their offerings. These solutions enable the construction of data center networks that can meet future technological demands, provide efficient services, and reduce operational costs and energy consumption.