The datacentre fabric

A GPU training job that should finish in 6 hours is taking 18. The GPUs are 90% idle, waiting. Nothing is wrong with the model or the code — the bottleneck is the datacentre fabric, the physical mesh of switches and fibre that moves gradients between machines.

Clos topology: spine and leaf

A classic two-tier or three-tier Clos fabric replaces a single large switch — which would be a single point of failure and a hardware engineering nightmare — with a mesh of commodity switches. In a spine-leaf design every top-of-rack (ToR) leaf switch has one uplink to every spine switch. No two leaves are connected directly; all traffic flows through the spines.

This gives you three things at once. Non-blocking bisection bandwidth: if you have N leaves and M spines, and each leaf has one M-speed uplink per spine, then cutting the fabric in half exposes N/2 × M uplinks on each side — any server can saturate its NIC to any other server simultaneously, because every leaf has equal uplink capacity. No spanning-tree loops: ECMP (Equal-Cost Multi-Path) hashes each flow’s 5-tuple across the available spine uplinks, so all paths are active simultaneously — no ports blocked by STP. Horizontal scaling: add more spines to add bisection bandwidth; add more leaves to add server ports. You scale without forklift upgrades.

Modern hyperscale fabrics run 400G Ethernet between leaf and spine, sometimes already 800G (IEEE 802.3df, finalised February 2024). The uplinks are Direct Attach Copper (DAC) cables inside a rack and pre-terminated MTP/MPO multi-fibre trunk cables between racks and rows — structured cabling at a scale that makes individual patch cables impractical.

RoCE and lossless Ethernet

GPU training requires collective communication — every node needs to exchange gradient updates with every other node in operations like AllReduce. With normal TCP, the kernel copies data from GPU → CPU → NIC → network → NIC → CPU → GPU. That double copy burns CPU cycles and adds latency that serialises the collective, idle-ing the GPUs.

RDMA (Remote Direct Memory Access) lets a NIC write directly into a remote machine’s memory with zero CPU involvement and zero copy. Over InfiniBand this was standard; over standard Ethernet it became RoCE (RDMA over Converged Ethernet). The catch: RDMA has no built-in retransmission. A single dropped packet stalls or aborts the RDMA operation, so the fabric must be lossless.

Two mechanisms make Ethernet lossless:

• PFC (Priority Flow Control) — when a switch buffer fills beyond a threshold, it sends a PAUSE frame upstream on a per-traffic-class basis, applying backpressure all the way to the sender. No packet is dropped; the sending port is held off instead.
• ECN (Explicit Congestion Notification) — as a switch queue builds, the switch marks packets with a congestion bit (instead of dropping). The receiver reflects the mark back to the sender (DCQCN in RoCE v2), which reduces its rate before PFC has to act.

Lossless Ethernet is not automatic — it requires per-port, per-priority flow-control knobs to be tuned precisely. Over-aggressive PFC creates pause storms (backpressure propagates across the fabric, freezing unrelated flows) or deadlock (circular dependency of paused ports). This is why GPU clusters are typically on a physically isolated lossless fabric, separate from the general IP network.

Power and cooling: the real ceiling

A standard 42U rack with 1U servers draws 5–15 kW. A rack of eight H100 GPUs draws 50–100 kW. Air cooling moves heat by blowing air over heatsinks and out the back of the rack; it handles roughly 15 kW per rack — the ASHRAE Class A2 envelope. A 100 kW rack needs liquid cooling.

Two approaches dominate. Direct-to-chip (DtC) runs coolant through cold plates bolted directly to the GPU die and voltage regulators, removing heat where it is generated. Air still handles the remaining ~20% of rack heat (storage, NICs, fans). Full immersion submerges server boards in a dielectric fluid (engineered oil or a fluorocarbon). Immersion handles 100% of the rack heat with no fans, and the thermal mass of the fluid smooths out demand spikes — but it complicates maintenance and cable management.

The hard constraint for new GPU deployments is often not switch ports or fibre — it is power capacity. A datacentre hall sized for 10 kW/rack is already paid for in HVAC and PDUs; retrofitting it for 100 kW/rack racks means replacing power feeds, cooling loops, and possibly the building transformer. That is a multi-year capital project. Stranded power is why hyperscalers now buy or build facilities expressly for GPU density from the ground up.