EVPN-VXLAN in High Performance Network O&M and Deployment

Traditional enterprise data centers are transitioning from 2-layer network designs to actual 3-layer routing. Routing protocols such as OSPF/BGP are capable of handling network connectivity for business needs and ensuring reliability. However, many applications, especially in virtualization, HPC, and storage, still require 2-layer network connectivity. From a business perspective, rapidly evolving applications aim to break free from hardware and network limitations. An ideal network is a scalable, migratable, and highly reliable 2-layer network. VXLAN tunneling technology extends 2-layer networks by creating a virtual 2-layer network above the 3-layer network. The VXLAN data plane requires the use of control plane protocols, and EVPN is chosen here to synchronize network states and tables, meeting the current business requirements.

1. VXLAN Overlay Network Virtualization

Network virtualization partitions a physical network into multiple virtual networks, allowing efficient utilization of network resources within the overall data center infrastructure. Through network resource sharing and isolation between virtual networks, users/businesses can have dedicated virtual networks. VXLAN overlay is currently the most mature and commercially viable network virtualization solution.

VXLAN utilizes standard overlay tunneling encapsulation and extends the control plane using the BGP protocol as specified by IETF standards. This standardized approach provides better cross-vendor compatibility and flexibility in device selection.

VXLAN provides a larger namespace for network (subnet) isolation across the 3-layer network. The VXLAN ID, also known as VNI, can support up to 16 million networks. VLAN isolation and VNI isolation (mapping VLAN to VNI) can be achieved locally on VTEPs, creating an overlay network that combines physical network isolation with virtual networks.

EVPN disseminates layer 2 MAC and layer 3 IP information to each business access switch (VTEP), supporting BUM (Broadcast, Unknown unicast, and Multicast) traffic and providing flood suppression functionality. It also supports pure layer 3 routing. Business communication between VNIs is achieved through layer 3 VNIs, enabling routing between them. Depending on business requirements, both centralized and distributed deployment models are supported.

For this project, there is a need for internal communication between multiple business subnets, so a more flexible distributed gateway deployment is adopted. This fundamentally enables flexible implementation, business migration, and deployment. Underlying network resources are load balanced using Equal-Cost Multipath (ECMP) and other technologies to provide higher east-west bandwidth and shield against single-node network failures, reducing operational risks and difficulties.

2. RoCE over EVPN-VXLAN

With the maturity and standardization of data center network deployment solutions and products, the speed of business deployment has increased, and operational costs have decreased. However, as business demands grow, data center applications require higher levels of computing, storage, and network resources. To match these upper-layer demands and achieve resource scalability, network design needs to provide network virtualization capabilities to support high-performance business requirements. The combination of network virtualization and RoCE technology makes the solutions for large-scale, high-performance data centers more complete.

RoCE (Remote Direct Memory Access over Converged Ethernet) is an Ethernet-based technology that enables efficient data transfer between servers, reducing CPU overhead and network latency. EVPN-VXLAN is a network virtualization technology that constructs virtual networks on top of the physical network by encapsulating business packets in VXLAN packets. This enables flexible network deployment and resource allocation.

By combining RoCE with EVPN-VXLAN, high-throughput, low-latency network transmission can be achieved in large-scale, high-performance data centers, along with scalability. Network virtualization allows the division of physical network resources into multiple virtual networks, providing independent logical network environments for different businesses, as well as flexible resource management and rapid business deployment. This comprehensive solution meets the requirements of data center applications for high-performance networks and provides a more complete solution.

3.Simplified Network Planning,Deployment,Operations

The modern data center networks have evolved from a few dozen devices to large-scale networks with hundreds or thousands of nodes. The complexity of network planning and management has increased significantly. To ensure the normal and reliable operation of the network, the operations and maintenance (O&M) teams need to enhance their capabilities from the aspects of network design, monitoring, and management.

4.Simplified Underlay Network with Unnumbered BGP

As the data center scale expands and the number of access ports increases, in the spine-leaf network topology, routing and load balancing between each layer require the support of External Border Gateway Protocol (EBGP) routing. From the perspective of network functionality, EBGP is already concise and reliable. However, from the deployment and network addressing perspective, designers have to specifically plan a large number of interface addresses for the underlay network. During actual deployment and operation, the large number of interface subnets can easily lead to configuration errors, resulting in underlay network issues that are difficult to control. The adoption of Unnumbered BGP technology eliminates the need to plan IP addresses for physical interfaces, greatly improving efficiency and avoiding risks caused by operational errors.

Unnumbered Interfaces originally referred to interfaces without IP addresses. However, establishing BGP sessions relies on TCP connections, which require unicast IP addresses based on interfaces. To address this issue, NVIDIA leveraged IPv6 Router Advertisement and RFC 5549 specified that for each IPv6 network link, the network will automatically generate a unique IPv6 address (link-local IPv6 address, LLA) on the link. BGP Unnumbered uses extended next hop encoding (ENHE) to allow BGP neighbors to advertise IPv4 addresses using IPv6 link-local addresses as the next-hop address. This eliminates the need to configure interface addresses and enables BGP neighbors to communicate and establish connections using automatically generated link-local IPv6 addresses.

5.WJH: Rapid Fault Localization for Improved Operational Efficiency

Real-time visualization and monitoring of network faults are essential for managing large-scale networks. The technical requirements of modern data centers have demonstrated that simplified network protocols and deep real-time network visualization capabilities are the trend in overall technological development.

Taking the fault snapshot feature WJH provided by Cumulus Linux as an example, let's compare it with the traditional approach to fault handling in operations.

With traditional system monitoring tools, network operators need to deal with a large amount of log collection, coarse-grained statistics, and status information. When a problem occurs, experienced network engineers narrow down the scope step by step based on their experience, sift through relevant information from the massive amount of data, and determine the root cause by considering statistical and status changes. If the problem is caused by configuration errors, the entire troubleshooting process can be extremely challenging because there are no obvious system abnormalities.

Using the WJH feature, based on the capabilities of the switch's switching chip, the switch directly captures abnormal packets and sends them to the network management or third-party monitoring platform as exceptional events, providing packet capture and chip-level problem causes. Whether it is a configuration issue or not, the operations team can directly see the affected business and the cause of the fault, enabling them to quickly take action and resolve the issue. Here, command-line information is used as an example, but the actual presentation format can be through network management or a third-party database.

6.Conclusion and Outlook

Data center infrastructure is transitioning from proprietary and complex systems to repeatable, predictable, and standardized infrastructure. According to Gartner's predictions and trends in enterprise data centers, data centers will gradually become distributed, and multi-data center environments will become the trend. As an important component of data center infrastructure, network technology will continue to build high-speed network connections, improve system access efficiency, enhance user service experience, and drive the high-quality development of data center networks.

As a leading provider of optical networking solutions, NADDOD is committed to building a connected and intelligent world through innovative computing and networking solutions. NADDOD continuously provides customers with innovative, efficient, and reliable products, solutions, and services. It offers optimal switch + AOC/DAC/optical module solutions for applications such as data centers, high-performance computing, edge computing, and artificial intelligence, significantly improving customers' business acceleration capabilities with low cost and excellent performance.

NADDOD provides optical modules, AOCs, DACs, and other interconnect solutions that support both InfiniBand and Ethernet, with options ranging from 100G, 200G, 400G to 800G, catering to the needs of different data centers. These high-quality interconnect products provide higher speed and more reliable data transmission solutions for data centers. With NADDOD's professional technical team, rich implementation experience in various application scenarios, and services, its products and solutions have gained the trust and favor of many customers, enabling the construction of data center networks that meet future technological requirements, providing efficient services while reducing operational costs and energy consumption. NADDOD's products will be the ideal choice forbuilding advanced and efficient data center networks.

In summary, the advancements in data center networking focus on simplifying network planning, deployment, and operations. The use of technologies like Unnumbered BGP eliminates the need for complex IP address planning, reducing configuration errors and improving efficiency. Real-time fault localization tools, such as WJH, provide deep network visualization and enable quick identification and resolution of network issues. The trend in data center infrastructure is towards distributed and multi-data center environments, driving the need for high-speed network connections and improved user service experience.

NADDOD, as a leading provider of optical networking solutions, offers innovative and reliable products, solutions, and services for data centers. Their offerings include optical modules, AOCs, DACs, and other interconnect solutions supporting various speeds from 100G to 800G. With their expertise and experience, NADDOD helps customers build efficient and future-ready data center networks that deliver high performance while reducing operational costs and energy consumption.Overall, the advancements in data center networking aim to create scalable, efficient, and reliable networks that meet the evolving needs of modern data centers.