The Datacenter Series Part 3: Networking Systems

"It is impossible to travel faster than the speed of light." - Woody Allen

Note: This post has been modified from its original version and formatted to fit this Substack. For the optimal experience, please see:

Intro: Your Network is Your Net Worth (really)

Networking is the second largest consumer of AI spend only to accelerators (read: chips) themselves. With good reason, networking performance - how fast and how much data can be communicated between chips, racks, and to your user - is a defining constraint of system performance. While for decades networking was an afterthought in compute workloads, increasingly large compute clusters running multi hundred billion parameter models have now pushed networking systems to their limits. As structural constraints begin to strain the size and scale of the AI capex buildout, networks are the next natural place for optimization as compute budgets and compute demand continue to balloon. Where we’re going, we will stop at nothing short of the speed of light!

Sources: Mckinsey, SemiAnalysis, Dell’Oro Group, IEA...

As the size of compute clusters scale, networking becomes an increasingly binding constraint on system performance. xAI recently proved this in spades when the media leaked the 11% utilization rate on their 550k GPU cluster (scaling is macrohard!) but even the most optimized training runs world (e.g. Meta Llama 3 at ~40% MFU) used less than half their theoretical compute due predominantly to networking and memory latency. Massive compute systems command higher Gbps, power efficiency, and more stability of their networks such that networking itself is now a critical bottleneck in AI system performance.

Networking interconnect consumes of nearly 30% of total cluster power, so it is natural that with access to terrestrial power systems more scarce than ever, focus has shifted to reducing network power consumption while optimizing for performance for inference systems. This is both an intense and frenzied swarm of innovation at the form factor level that has also put public markets into a frenzy to bid the producers of any potential new form factor like co-packaged optics, no matter how speculative the claim. Bound by finite power, every hyperscaler is actively developing their own networking systems alongside custom chips to optimize system performance.

In our last long form report we discussed how the evolution of compute demand drives increasing power densities per rack and what this means at the datacenter infrastructure level. New power and cooling systems are needed to meet the demands of upcoming 800 volt direct current datacenters designed to meet forthcoming 1MW+ power draws per rack. Naturally, scaling compute workloads also scales the demands on the system that connects them, and that system too faces the same ultimate issue of the entire datacenter: increasing token output with finite power. In this report we focus on these networking systems.

kW/Rack Requirements Across Nvidia Generations

And there are multiple threads to unpack within the evolution of networking systems: 3arly Ethernet systems displaced by InfiniBand, an Nvidia acquisition for the ages that created system level moats until the industry rallied behind Ethernet. Copper’s increasing obsolescence over time to fiber optics. How much more copper can we practically eliminate as we get closer to the chip? The transition from a system of very few behemoths to a number of possible paths forward in development now seeking to minimize switch power draw while maintaining low latency for inference workloads.

We’ll break down the components of an AI networking system, discuss the history of how the networking landscape has evolved since the 2000s - from copper to optics to the Infiniband vs. Ethernet Wars, and the future paths of networking systems actively in development as the industry prepares to scale inference workloads to meet boundless levels of token demand. As we predicted several years ago when we started Crucible, the largest physical infrastructure build in human history - data centers - are creating immense structural tailwinds for an innovation economy pushing the limits of physics and thermodynamics.

If you’re new here, see Part 1 on powered shell development and Part 2 on 800V DC power and cooling systems.

Information Theory: God Bless Claude (Shannon)

If we’re talking about data moving across networks, it is imperative to step back and ask ourselves why data exists in the shape it does. To do that, we turn to Claude Shannon, the father of the theory of information. Shannon joined Bell Labs in 1941 and was working on cryptography and secure communication systems. Shannon’s landmark 1948 paper, “A Mathematical Theory of Communication,” answered a strange question: what is information and how can it be defined, quantified, measured, and relayed? Shannon introduced the bit, the binary digit, as the basic unit of information. His work showed that any information - from a sonnet to a symphony - can be encoded into sequences of bits and that the content’s meaning is irrelevant to the communication system; only the statistics of symbols and noise matter.

Data packets are small, structured chunks of bits that carry your data plus just enough addressing and error-checking information so that billions of machines can fling them around the internet independently and then reassemble them into a coherent message. By turning “information” into something quantifiable and encoding-agnostic, Shannon showed that any message - text, audio, video - can be represented as bits.

At the lowest level, everything moving over a digital network is just sequences of 0s and 1s (bits) encoded as electrical, optical, or radio signals. A data packet is a formatted unit built from those bits: it bundles a slice of user data with control fields that say where it came from, where it should go, how big it is, how to order it, and how to check it for errors.

What is striking is how little the core abstractions have changed since Shannon’s work in the 1940s. Modern communication via phones, fiber backbones, Wi‑Fi, deep-space probes, and undersea cables all operate by encoding information as bits and transmitting a raw bitstream over a medium - whether wire or waveform - at the limits of what physics allow.

Under the hood, we’ve gone from relays to vacuum tubes to transistors to nanoscale CMOS, and yet the stack still looks the same as it did in 1941: bits at the bottom, packets in the middle, applications at the top. You can swap out materials, fabrication processes, and physical media, but the underlying process remains intact. This is a core pillar of our infrastructure thesis at Crucible: some abstractions are so fundamental they transcend implementation details.

Defining The Network

What’s In a Networking Fabric?

A network is a dense fabric of switches and cables between GPU nodes and between servers of GPU nodes that transmits waves of data between all the various compute devices on a chip and in a rack. The purpose of the fabric is to establish redundant communications paths between compute nodes and to deliver higher total throughput than traditional standalone networks. The fabric allows the isolated FLOP machines talk to each other in such a way that overall production of the system compounds, enabling efficient routing of data flows within the chip architecture at all time. A diagram for an AI networking fabric looks something like this:

Sample Nvidia RA B300 Networking Fabric

In really simple terms a network is a spine switch (for rack to rack comms) connecting to top of rack (or leaf) switches (for intra rack comms) connecting servers connecting to on chip switches called NICs (network interface cards).

Forthcoming generations of NICs seek to add routing intelligence to the NIC itself as a means to eliminate the top of rack switch and allow more GPUs within the rack.

Our report on CPUs as the bottleneck to GPU inference covers some of the foundational concepts regarding how data moves around the physical structures on a chip, and we recommend a quick browse to further understand the relationship between accelerator components. Again, the physics of how everything is laid out on the chip are the gating item defining the routing and latency of how workloads flow between constituent components. Much of networking is about optimizing the physics of information using novel geometries and novel materials.

Some Foundational Definitions

Here is a brief review of each core component in the network with an overview of its role within the networking fabric:

There’s a lot of stuff going on in the networking fabric. We’re going to focus on switches, since they are the key to routing packets. Let us introduce you to a switch -

This is a state of the art top of rack InfiniBand switch: The Nvidia Quantum-X800 Q3400-RA with 800 gigabits per second (Gbps) throughput powered by Quantum-3 ASICs using air cooling. This generation doubled the Gbps of its predecessor, as networking systems do every two to three years, and also features in network computing for operations like reductions and aggregations directly in the switch rather than in the GPU.

This switch’s successor will actually be a CPO (copackaged optic) based switch using liquid cooling to cool onboard silicon photonics to deliver 5x power efficiency vs. pluggable transceivers. We’ll discuss these transitions at the end of the report, the point for now is that this switch is the state of the art switch pushing the envelope to close out the era of pluggable tranceivers, potentially.

Discussing the landscape of who is involved in the switch supply chain begets a longer conversation of the InfiniBand protocol vs Ethernet protocol - the two dominant networking protocols with a rich rich lore and history. We discuss this next.

InfiniBand v Ethernet for Simpletons

The two dominant networking protocols used today are InfiniBand and Ethernet. Here is a brief summary of the key differences between the two. We’ll cover the history of how these protocols evolved in the coming pages.

North South, East West, Scale Up, Scale Out

With switches defined, next we’ll break down the two types of networks in a compute networking system:

North-South is simply the compute cluster talking to the outside world and storage systems. The front door to the building, if you will.

East-West is comprised of the “Scale Up” network - GPUs talking to GPUs inside the same box (via NVLink in an Nvidia GPU cluster) and the “Scale Out” network - servers talking to other servers to compound compute power. Using our analogy above, EW = all of the happenings inside the building.

The vast majority of networking advancements are aimed directly at solving the East-West bottleneck. The East-West compute fabric demands more innovation vs. the humble North-South network given its task of connecting all of the servers and compute nodes in the compute cluster. Marvell stated this point blank in their May 27 earnings call: “Scale-up interconnect represents one of the newest and most strategically important opportunities emerging in AI infrastructure.” Understanding why the East-West bottleneck is so important is obvious when observing the history of compute workloads and how networking demands have grown alongside them. Without further ado, let us dissect the history of computing from the lens of compute networking systems.

A Networking Retrospective: 2000-2026

The history of networking can be viewed through two lenses: Ethernet vs. InfiniBand and The Inevitability of the Speed of Light Squeezing out Copper. Starting with the former...

How did we go from plugging graphics cards into personal computers to now wiring together buildings - campuses even - of machine gods? Interestingly enough, up until circa 2018, networking was simply dragged along by compute workloads, providing whatever Ethernet path existed to get the job done. We’ll find below that come 2018 and the rise of billion+ parameter models, networking became the active constraint in compute advancement, and remains so today, along with memory & power.

2000-2006: Islanded GPUs for Pixels Only

In 2000, GPUs existed purely to push pixels. They sat on PCI or AGP slots and had no meaningful ability to talk to each other or to the network. Modern digital computers have been using von Neumann architecture since their inception. In this architecture, there are CPUs (arithmetic logic unit and control unit), memory (RAM, hard disk), and I/O (input/output) devices. In the early 1990s, in order to support more and more external devices, Intel was the first to introduce the Peripheral Component Interconnect (PCI) bus design into the standard PC architecture. The PCIe bus connects the CPU to everything else - GPU, the NIC, storage drives, and other expansion cards.

Before and After the PCIe Local Bus

Shortly after the consumer Internet emerged, compute systems entered a phase of rapid development. The continuous growth of online businesses and users stretched the bounds of the capacity of IT systems. At that time, with the support of Moore’s Law, components such as CPUs, memory, and hard drives were rapidly advancing. But the PCI bus was upgrading at a slower pace, limiting I/O performance and becoming a bottleneck for the entire system. Ethernet was the prevailing and only networking system of the time, founded in 1973 at Xerox PARC to simply connect computers to printers - it was ubiquitous, cheap, and the “good enough” standard everyone used. But it was designed for bursty internet traffic, not tight and synchronized communication.

The InfiniBand Trade Association (IBTA) was formed through the merger of two developer forums composed of Intel, Microsoft, SUN, IBM, Compaq, and Hewlett-Packard in 1999 specifically to address the gap in CPU performance and interconnect. It was rapidly gaining traction as a purpose built alternative to Ethernet. The IBTA consolidated two competing efforts into a single architecture and roadmap, avoiding a split fabric standard at the server and cluster level.

By 2000, the first InfiniBand 1.0 specification was released, introducing Remote Direct Memory Access (RDMA) for high-performance, low-latency I/O. Over two decades, the IBTA has maintained the InfiniBand roadmap, driving a 100x performance improvement in link speeds while aligning increases with system level performance gains, which kept the fabric relevant for ever‑larger clusters. IBTA membership has more than doubled since 2023, driven by AI workloads and underscoring the IBTA’s role as a focal point for the industry, ensuring uniform standards in networking fabrics used in thousands of data centers.

RDMA was a breakthrough because it allowed data to jump directly from one machine's memory to another's, bypassing the CPU entirely offering lower latency, greater reliability and faster I/O. Also to note in this time period, Nvidia released CUDA (Compute Unified Device Architecture) in 2006, a parallel computing platform and programming system allowing developers coding in C / C++ to use GPUs for general math vs. graphics alone. You know what’s next...

The RDMA Breakthrough Illustrated

2007-2011: Science and Finance Discover GPUs

Before CUDA, using GPUs for non‑graphics work (so‑called GPGPU) meant abusing graphics APIs and expressing computations in terms of textures and shaders. This required deep graphics expertise, made debugging painful, and kept GPU computing limited to a handful of research groups rather than mainstream scientific or financial developers. CUDA’s 2006 launch turned GPUs from awkward, graphics‑only devices into a general‑purpose, C‑programmable parallel compute platform, which scientists and quants quickly exploited.

The first wave of GPU adopters was the scientific and engineering computing community. They had workloads that were:

• Highly parallel (Monte Carlo simulation, fast fourier transforms ie FFTs, and mat mul)

• Already constrained by CPU performance and cluster budgets

CUDA’s enablement of these workloads for problems like molecular simulations, fluid dynamics modeling, seismic imaging, financial modeling and more was the match that sparked the fire for GPU workloads which in turn pushed forward networking. GPU workloads were growing, but at the time HPC networking looked like this: GPU → PCIe bus → CPU → InfiniBand NIC → network → repeat in reverse. Every hop added latency and consumed CPU resources - they called this the "host-staging" problem.

Come 2011 Nvidia introduced GPUDirect to solve this - GPUDirect let the GPU talk directly to the InfiniBand network card (NIC), cutting the CPU out of the loop. Data could move GPU-to-GPU across a network without touching the CPU at all. It was a meaningful step but still limited by the PCIe bandwidth between the GPU and the NIC sitting in the same server. This sounds similar to the problem InfiniBand 1.0 solved in 2000 already discussed, but in this case GPUDirect solved intraserver networking while InfiniBand RDMA solved the CPU bottleneck between separate servers.

2012: AlexNet Evangelizes Parameters

In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton - some people you might recognize - submitted a new model, AlexNet, to the ImageNet Large Scale Visual Recognition Challenge. Containing 60 million parameters and 650,000 neurons, AlexNet but the top-5 error rate from 26.2% to 15.3%, proving that more GPUs and more data rendered better model outcomes and sparking a wave of deep learning innovation. Researchers immediately understood that the future of model development was multi-GPU training, and multi-GPU training immediately ran into the PCIe bandwidth constraint discussed just above.

Computation to Train AI Systems Over Time

Source: EpochAI

AlexNet was a tipping point that kicked off the arms race in accelerators, high‑speed interconnects, and AI‑oriented datacenter networking that underpins today’s infra builds. Inside the datacenter, clusters had to move from a few GPUs per box to large accelerator pods, which demanded high‑bandwidth, low‑latency interconnects. Between datacenters, the explosion of model led services (search, photo, speech, recommendation) drove higher traffic volumes and more latency‑sensitive paths. As always, necessity is the mother of invention.

2014-2019: Enter NVLink, I Got 99 Problems and PCIe Ain’t One

AlexNet established more GPUs = more parameters = better model outcomes. GPUs were available, but connectivity was still limited by PCIe bandwidth constraints. As you might guess, NVIDIA strikes again!

Nvidia pushed out NVLink in 2016 as an alterative to PCIe with higher bandwidth and features designed for GPU systems. NVLink bandwidth grew from 20 GB/s per link in Gen 1 (2016) to 50 GB/s in Gen 2 (2018). Instead of GPUs shouting at each other across a crowded PCIe bus, they now had a private, high-speed highway between them.

This marks the beginning of Nvidia’s generational run, introducing bespoke solutions to allow direct GPU to GPU communications and to enable more developer use cases of these systems. Then, Nvidia acquired Mellanox for $6.9 billion, one of the greatest acquisitions in tech history.

2019: Nvidia’s Mellanox Acquisition

Formed in 1999 by ex Intel and Galileo Technology employees in Israel, Mellanox stood as the de facto firm responsible for InfiniBand’s survival and commercialization after major founders (Intel) moved on to focus on PCIe. Nvidia’s Mellanox acquisition was pivotal because it gave Nvidia control of the high‑performance interconnect fabric that ties thousands of GPUs into a single logical AI supercomputer, making Nvidia a systems company.

Before Mellanox, Nvidia owned the accelerator (GPU) but not the high‑end network. After the acquisition Nvidia could design, manufacture, sell and optimize complete AI and HPC nodes and clusters where the networking was tuned specifically for GPU workloads rather than relying on third‑party vendors whose roadmaps or priorities it could not fully control. Mellanox is why Nvidia’s HGX and later systems could connect hundreds to thousands of GPUs as one coherent training fabric and outperform mix‑and‑match competitor setups.

Nvidia understood networking was essential to future compute workloads and a harder part of the data center to solve than compute itself. This was the largest acquisition in Nvidia’s history at the time by a 10x order of magnitude - Jensen outbid Intel and others and spent ⅓ of his cash on hand at the time. Jensen put his chips on the table, establishing the view that a GPU maker needed to expand in the network to compete in the future of computing. He was right, and just in time…

2020-2023: LLM Detonation Introduces the Scale Out Problem

LLMs are cooking (GPT 1 enters the chat in 2018)... Up to the A100 moment of 2021, murmurs spread of the 2020 GPT-3 175b parameter model among researchers and infra engineers. Mass deployment of Nvidia’s A100 GPU in 2021 was a moment because this was the first GPU engineered for AI training at scale, with NVLink 3.0 at 600 Gbps inside the server and a purpose-built InfiniBand port on the board.

But its real significance was what it revealed about the networking problem. Training runs were now crossing a threshold where no single server could hold the model - GPT-3 required 800GB-1TB+ of memory while an A100 supported 80 GB max - GPT-3 required dozens of A100s working together. Distributed training, where model parameters are split across servers communicating constantly, became mandatory. This created the “scale-out” problem in its modern form: how do you connect hundreds or thousands of servers so tightly that they behave like one giant computer? The answer at the time was InfiniBand HDR at 200 Gbps per port. It worked but with limitations, 200 Gbps is not fast enough for thousands of synchronized GPUs.

2022: The Dawn of the Training Era

OpenAI introduces ChatGPT and lands 100M users in two months, the fastest growing consumer app in history at that point. The infrastructure implications were immediate and violent. Every major tech company simultaneously decided to build or expand GPU clusters. The demand for A100s and then H100s went from “strong” to “we will pay anything, give us chips please.” Lead times on H100s stretched to 12 to 18 months and Cloud GPU rental prices ripped. Training cluster builds go rampant, but inference is going to need an entirely new fabric that solves for accommodating different tasks running async with different latency profiles, rather than simply GPU to GPU syncing every few milliseconds.

Where were you on November 30, 2022?

Unlike video or static web, LLM responses are highly dynamic and user‑specific,creating “spiky but continuous” load profiles rather than the more periodic workloads typical of batch training runs or media streaming. As ChatGPT‑like workloads spread, this altered how networks are used. To serve interactive inference at scale, operators needed dense GPU clusters with very high bisection bandwidth and low‑latency fabrics, increasing reliance on RDMA‑capable networks and careful L2/L3 design inside clusters.

2023: The Ethernet Consortium and Llama3 shows RoCE Competes, InfiniBand’s Peak?

InfiniBand commanded roughly 80% of AI cluster market share by 2023. Nvidia was dominating vendor lock-in across their GPUs, their NVLink (inside the server), their InfiniBand (between servers) and their NCCL software (coordinating all-reduce operations across the cluster). A customer buying into AI training was buying an Nvidia vertical system from silicon to software, which for the training era was largely DGX H100s deployed through 2023.

A coalition of companies - Intel, AMD, Broadcom, Arista, Cisco, Microsoft, Meta, Google - formed the Ultra Ethernet Consortium (UEC) to develop enhanced Ethernet transmission stacks, directly challenging InfiniBand. The goal was to add to standard Ethernet the features that made InfiniBand special: RDMA (direct memory access without CPU involvement), in-network computing, congestion control designed for AI workloads, and deterministic behavior. By 2025 the UEC released its 1.0 specification - a standard that competed with InfiniBand. Critically, Meta published its results of training Llama 3 on parallel clusters based on Ethernet and InfiniBand fabrics to serve as the industry’s proof of concept that InfiniBand’s dominance could be challenged and that Ethernet was a viable alternative.

2024 Onward: Dispersion is Here to Stay

While training dominated workloads, inference has grown from 1/3 of compute in 2023 to 2/3 today. At the same time foundational research has proven that systematic co-design of network, software, model architectures and hardware renders meaningful results. In turn AI compute systems are increasingly fracturing in shape.

Market Share Visualization of High End Interconnects

A Parallel Transition: Copper to Optics (the Speed of Light is Inevitable)

The Early Days of Copper Interconnect

While the industry transitioned from InfiniBand by reinvigorating Ethernet adoption via intentional development and co-design processes, an even more consequential transition was happening at the material level. For decades, precisely since IBM developed barrier layers to allow copper to viably replace aluminum in 1997, copper has served as the standard material for interconnect. Copper (59MS/m) is the second most electrically conductive metal only to silver (63 MS/m) but it’s more widely used than silver given its affordability. We Cu copper.

Copper’s superior conductivity to aluminum grew more important as transistors shrank over time. Smaller transistors -> more transistors -> more and smaller wires to connect them. The RC Delay = Resistance (thinner wires, higher resistance) x Capacitance. As wires shrink, R and C declined together and by the mid 1990s RC delay in interconnects was a problem. Wires were slower than the actual switches. IBM developed a copper based solution to capture its 40% improvement in conductivity - again, important as wires on chips got smaller. Copper’s lower resistance also mattered as the layers of interconnect scaled over time from 1-2 layers to 5-7 layers stacked by the mid 90s - increasing wire length and resistance. IBM’s first copper chips in 1997 showed a roughly 15-20% speed improvement or equivalent power reduction compared to aluminum at the same node.

Copper was Enough, Until...

In 2011 at the International Supercomputing Conference, an interconnect running at about 56 Gbps was announced and demonstrated by Mellanox - a jump from 40 Gbps prior, to the point where copper’s usefulness over distance became problematic. At this bandwidth, Passive Direct Attach Copper (DAC) cables - essentially just a copper cable with connectors on the ends and no active electronics - could only reliably carry FDR signals up to about 3-5 meters. Beyond that, signal quality degraded.

Active Optical Cables (AOCs) now started to become a mainstream product rather than a niche luxury. An AOC looks like a cable with connectors on each end just like a DAC, but inside it converts the electrical signal to light at one end, sends it over fiber, and converts back to electrical at the other end. Importantly, AOC was easily adopted because it didn’t require any changes to networking chips given the construction of the cabling - it still presented an electrical interface to the switch and server, fiber was abstracted away inside the cable assembly, so copper cables could simply be swapped out for AOCs.

Optics Over Time

InfiniBand continued pushing copper DAC for short distances and AOCs for longer from 2011 onwards, however as link speeds continued to increase, copper’s utility continued to compress. Fiber optics were technically optional until the 2014 generation of InfiniBand fabric, called “EDR”, delivering 100Gbps, limiting copper DAC cables utility to only 1-2 meters - enough to connect a server to the top of rack switch and nothing more. At the time, a typical GPU cluster had:

• DAC copper from GPU server to the top-of-rack switch (1-3 meters)

• AOC fiber from top-of-rack switches to spine/leaf switches (tens of meters)

• Long-reach optical transceivers for any inter-building or long-haul connections

Notice that the era of the EDR product that eroded the use of DAC copper cables from 2014-2019 directly led into Nvidia’s 2019 acquisition of Mellanox previously discussed. With networking speeds doubling every 2-3 years, this pattern of copper’s continued deterioration in utility continued to accelerate.

Networking Speed Has Doubled Every 2 to 3 Years

History Over, Where This Leaves Us Now

With 800+Gbps the new baseline for frontier workloads, copper now only works in the last 1-2 meters of interconnect and everything beyond that is optical fiber.

The Practical Breakdown Today Across a Typical NVIDIA GPU Cluster

One could summarize the progression of materials in networking systems as solving the physics problem governing datacenters more broadly: maximizing compute output with limited power resources. Copper consumes power in a way that becomes brutally more inefficient as you push more data through it.

As we discussed, at high frequencies, copper causes significant signal loss as the electrical signal degrades over distance and speed. To compensate, you have to amplify and re-drive the signal repeatedly, which burns more power. In short the faster you push data through copper the more power you need to maintain signal integrity. At 800Gbps+ the power consumption of the copper becomes a problem.

Light (optics) doesn’t suffer the same signal degradation over distance. A photon traveling down a fiber doesn’t need to be amplified every few centimeters the way an electrical signal does. So while there is a power cost to convert electrical signals to light and back within a transceiver, once you’re in the optical domain you can travel meters or even kilometers for essentially the same power cost.

While copper has been eliminated over time due to its physical constraints, there are a few steps in the networking remaining in which to eliminate copper altogether.

Form Factor Advancements

As we think about the future of where networking systems go we must remember that AI inference involves a different problem than training. Training cares most about high bandwidth and scale to synchronize a ton of GPUs. Inference cares most about latency consistency (you need predictable response times) and power efficiency at scale (inference clusters run 24/7 serving live traffic).

Co-Packaged Optics

CPO promises power optimization + latency reduction that are simply too far of a reach in our view. Traditionally the optical transceiver (the component that converts electrical signals to light between copper on chip and fiber optic cables) sits in a pluggable module on the edge of a switch connected to the switch chip by a copper trace on the PCB. That copper trace between the chip and the transceiver is short, maybe a few centimeters, but at 800Gbps+ it still burns significant power and loses signal.

CPO moves the optical engine physically onto the same package as the switch chip, essentially eliminating that copper stretch entirely which then allows the signal to transform from electrical to optical almost immediately. CPO drives 30-40% (or more) lower power consumption for the optical interconnect, lower latency, and the ability to scale bandwidth further than pluggables allow.

Said differently CPO addresses the issues of power efficiency and latency consistency. This sounds like a no brainer, so the question becomes why CPO adoption is not more advanced today after having been initially deployed by Tencent in 2021, when they deployed a Broadcom CPO marking the first hyperscale CPO deployment globally. Outside of Tencent, Meta has done the most work on testing Broadcom CPOs and Amazon plans to deploy CPO within Trainium 4 - but other domestic hyperscalers are pursuing other optical solutions thus far (Google <> Optical Circuit Switching, Microsoft <> XPOs).

Comparing Switch Architectures: Traditional v Co-Packaged Optics

CPOs are a nightmare to operate and there is not a well oiled manufacturing supply chain. Barriers to scaling CPO adoption include and are not limited to...

• Optical transceivers can be replaced in minutes but a CPO failure = an entire switch replacement which takes hours. Broadcom directly responded to this with detachable sub assemblies in its Bailly generation of 2024. Meta and Broadcom’s results in this study answered the serviceability question of CPOs, but the quantified stability and power draw benefits that Broadcom have cited are still hypothetical until proven at scale.

• The vendor ecosystem is not established yet unlike that of pluggable optics. Manufacturing scale up is costly, prototypes are still 2-3x cost of traditional switches.

• Standards for thermal specifications, mechanical interfaces, etc. are still evolving. The Optical Internetworking Forum is establishing baseline standards in real time.

• Thermal management is challenging because the laser components of a CPO are sensitive to heat. New designs use external lasers but this adds further complexity to assembly and supply chain.

Jensen is walking a fine line in Nvidia’s push into CPOs. He first signaled Nvidia’s stance on CPOs publicly at March 2025’s GTC. “The challenge with scaling out GPUs to many hundreds of thousands is the connection of the scale-out. When the data centers are now the size of a stadium, we need something [new] — and that’s where silicon photonics comes in.” At 2026’s GTC Jensen showcased what he described as the world’s first mass-produced co-packaged optical switch, Spectrum X, saying Nvidia needs “more copper cable production capacity, more optical chip production capacity, and more CPO production capacity.” Hence Nvidia’s March 2026 partnership and investment announcement into Coherent. This is all in line with our findings above.

That said, despite lots of market enthusiasm and chatter on X, Jensen has yet to embrace CPO on an Nvidia earnings call. In fact he didn’t even mention it once on Q1 2026 earnings. We believe a few factors are at play here:

• Nvidia’s networking revenue including optical transceivers are a growing share of Nvidia’s topline, representing 18% of revenue as of Q1 2026, beating estimates for the segment while compute did not. Nvidia is planting seeds to expand CPOs while milking the growing cash cow in pluggable transceivers and switches.

• While Nvidia has made a small investment in Coherent to expand the supply chain, the cost to scale up manufacturing at even the most advanced foundries, especially when accounting for better thermal management that fractures manufacturing further, is high.

Linear Pluggable Optics and eXtended Pluggable Optics

Emerged to combat the adoption barriers to CPOs. Rather than embed optics at the chip level, LPO uses a familiar pluggable format but uses linear transimpedance amplifiers (rather than DSPs on transceivers) to drive optical signals. LPOs prove that you can get 70-80% of CPO’s power benefits while maintaining the familiarity of pluggability (with modules that can be hot swapped), serviceability, and multi-vendor interoperability from day 1. For much of the industry between 2022 and 2025 the promise of LPOs vs the burdens of CPO was a compelling enough argument to wait for LPOs increase bandwidth capabilities. Arista Networks has been a long time LPO advocate and producer - though their adoption to date is limited because LPOs at 100-200G per lane still lag the 800G capabilities of legacy DSP infrastructure (bandwidth > all else for now). A deeper write up on LPO vs CPO from Celestial AI and Cisco Systems can be found here.

Arista one upped the concept of LPOs with the announcement of XPO, or eXtended Pluggable Optics at March 2026’s OFC Conference. XPOs, to go into production in 2027, are a pluggable form factor designed specifically for high-speed optics that makes the optical module larger in order to replace 8 legacy optics modules, allowing networks to use 25% of previously required switch racks. Thermal constraints are directly addressed with built in cold plates to enable this density. XPOs are a promising solution - familiar pluggability = integrates with existing servicing and vendor workflows, address thermal issues head on which in turn enables more bandwidth per rack unit. Since the March announcement, Arista’s XPO has since been endorsed by more than 100 vendors. Microsoft called XPO “an important milestone” and said they believe it can help establish a broadly adopted form factor that enables a diverse optical ecosystem.

Comparing Switch Architectures: Traditional v eXtended Pluggable Optics

We are practical people and think that XPO leveraging of existing pluggable workflows and supply chains will enable this solution to scale faster than CPOs. XPOs are Arista’s most ambitious endeavor to date, attempting to define the optics standard for the next decade of AI infrastructure the same way their open EOS software defined the software standard for the last decade of network operation systems.

Optical Circuit Switching (for Power Reduction and Network Reconfiguration)

Traditional network switches are electrical and packet-based where every packet gets routed individually through silicon. Optical Circuit Switching establishes a direct light path between two points, like a dedicated lane that stays open for the duration of a communication. There are many trade offs here - most critically is that the OCS can’t split traffic across multiple paths which eliminates flexibility to optimize for underused bandwidth. Switching speed is a limitation here, reconfiguring paths takes a millisecond now and electronic packet switches operate at the microsecond level. Google is the only hyperscaler that has deployed OCS at scale to date in their Jupiter data center fabric and outsources their OCS requirements to Lumentum and Coherent. Other start ups approaching OCS include iPronics, Omnitron Sensors and Salience Labs. OCS is proven and has commercial traction but is still relatively nascent - eyes will remain on Google’s results before further adoption likely.

Comparing Switch Architectures: Traditional v Optical Circuit Switches

CPO, XPO and OCS together cover the forward looking networking advancements of Nvidia, Meta, Broadcom, Microsoft, Amazon and Google. A broader group including AMD, Broadcom, Meta, Microsoft, NVIDIA and OpenAI recently established the Optical Scale Up Consortium in March 2026, which seeks to coordinate solutions in a multi-vendor ecosystem. With respect to form factor, the Consortium, as of now, pledges to offer support for pluggable, on-board, and co-packaged optics (CPO). Other solutions simultaneously in development at various stages not discussed here include:

• Disaggregated Memory and CXL (Micron, Samsung, and startups like MemVerge and Pond are actively building around this),

• In-Network Computing (InfiniBand’s SHARP is one version of this), Near-Memory and Near-Network Processing (Samsung and SK Hynix),

• All-to-All Optical Fabrics Within a Rack (Ayar Labs and Celestial AI),

• Silicon Photonic Integrated Circuits (PICs) at Wafer Scale,

• Wireless Optical i.e. Free Space Optics (don’t sleep on laser beams for data, especially with the development of orbital datacenters).

With dispersion and diversification increasingly obvious in the chip market itself, networking form factors look to be on a similar trajectory, that we as investors with opinions by pick our horse in races, and we’ve explained why we think XPOs are better positioned to scale faster than CPOs.

That’s a Lot of Hardware, Gonna Need Some Software

Hardware heterogeneity at the networking and chip levels necessitates different types of software to optimize systems. We need to up those MFU numbers, team. This is a field in nascency still, but we map this space as follows:

Software for Collective Communication Libraries

A collective communication library (CCL) orchestrates how data moves across the networking fabric across 1000’s of GPUs. This is currently a duopoloy between AMD RCCL and Nvidia’s NCCL that solve this and allows multi GPU workloads but still struggles on heterogeneous clusters - particularly Mixture-of-Experts models where token routing is dynamic and unpredictable. Meta forked NCCL into NCCLX, but these types of systems are enormous lifts. The field is open the Ultra Ethernet Consortium to drop a CCL, hardware agnostic transport layer.

Kernel efficiency is a precursor to CCL orchestration worth mentioning. Kernel optimization is new frontier of software that optimizes what happens on the GPU before and after the movement over NCCL, i.e. ordering the matrix multiplications, attention operations, and memory access patterns that determine how efficiently each GPU uses its compute cycles.

Names to Watch: Spectral Compute (we’re proud investors), Standard Kernel, KernelAgent by PyTorch and KernelBench by Stanford.

Software for Congestion Management

Congestion remains an unsolved problem, especially in RoCE clusters where rigid RDMA ordering leads to 30% performance loss in Ethernet systems. The Ultra Ethernet Consortium released a specification inclusive of software to to address this in June 2025, but there remains a need for better Programmable Congestion Management and Congestion Signaling algorithms.

Names to Watch: Liqid is a Series C company developing composable infrastructure software that dynamically allocates network bandwidth across workloads.

Software for In Network Computing

Certain computations are now being pushed inside the network fabric or switch as discussed. Nvidia SHARP executes compute in-network but is not Ethernet compatible. Nvidia Dynamo is designed to optimize how inference requests are routed and allocated between GPUs and switches, again not Ethernet compatible.

Names to Watch: Cornelis Networks.

Software for Network Observability

Telemetry data rocks. SONiC is an OS Linux based network OS that lets operators program switch behavior, monitor traffic and implement custom routing. However, ask a network engineer and they will tell you this is a slog to deploy and maintain. Meta, Microsoft, and Amazon all run SONiC or SONiC derived stacks in production but its use is scarce beyond hyperscalers. Arista’s EOS and CloudVision platform provides real-time telemetry and network-wide automation but are only compatible with Arista hardware. Hardware agnostic, AI native network observability tooling remains an acute painpoint.

Names to Watch: Supertrace is building a vendor agnostic AI Network Engineer.

Conclusion

The evolutionary processes of compute interconnect discussed here are a reflection of two facts today:

• Compute demand is insatiable

• Compute supply is constrained by physical realities

The combination of these realities have driven the acceleration of compute system advancements to continuously increase tokens/watt over time. Per Nvidia, inference energy efficiency in tokens/watt terms has increased 1,000,000x over six generations of Nvidia architectures.

Despite the relentless pace of innovation, it’s clear that compute supply remains constrained and advancements at the hardware level are one force to only partially mitigate this.

As such, we are closely following and interested in advancements in alternatives to autoregressive model architectures that more efficiently use compute resources. We welcome conversations with anyone researching and building at the frontiers of hardware, software, and models discussed here. Our contact info is below :D

And if you enjoyed this: