AI 101 — What is liquid cooling for AI data centers?

Share
AI 101 — What is liquid cooling for AI data centers?

Liquid cooling means the heat from AI chips is carried away by fluid running through metal plates clamped to the processors, instead of by air blown past them. It is the plumbing change behind almost every new AI data center, and the reason a modern AI hall looks less like a server room and more like a boiler room.

For about thirty years a data center was a windy place: rows of servers, fans at the front, cold air pushed up through a raised floor, hot air pulled out the back. That design works up to a point. One Nvidia GB200 NVL72 rack, which holds 72 Blackwell GPUs and 36 Grace CPUs, draws 120 kilowatts — the heat of roughly a hundred electric space heaters packed into a cabinet the size of a refrigerator, according to Nvidia's own description of the rack. No fan array moves that much heat out of that little volume. So the heat leaves in water.

Why it matters right now

Two things changed at once. Chips got hotter — a top-end AI accelerator burns 700 watts and up — and racks got denser, because the fastest way to train a model is to wire hundreds of accelerators into one machine that fits in one cabinet. Air cooling ran out of headroom somewhere in the 20 to 35 kilowatt range per rack, per engineering work published by Schneider Electric. Everything above that needs a coolant loop.

That is why cooling became a supply-chain story with its own headlines: LG signing on to supply cooling and power for Microsoft's next 26 GW of capacity, and LG's 2.6-megawatt coolant distribution unit clearing Nvidia's reference-design qualification. It is also why the buildout, not the model release, is the slow part — we walked through the electricity side in this AI 101 on how much power AI actually uses.

The mental model: three loops

Think of the heat as water flooding a basement, moved out in three stages.

The first loop is inside the rack. A cold plate — a flat block of copper or aluminium with narrow channels machined through it — is bolted onto each GPU and CPU, with thermal paste or a pad in between so heat crosses the metal. Coolant runs through those channels and picks the heat up by conduction. Nothing is sprayed. The fluid never touches the silicon; it touches the lid of the chip package.

The second loop is the coolant distribution unit, or CDU: a heat exchanger, usually a box at the end of the row, that hands the heat from the rack's fluid to the building's water. The two fluids stay separate, so the rack loop can be filtered and treated like a sealed system while the building loop stays the facility team's problem. The third loop rejects the heat outdoors, through dry coolers, cooling towers, or chillers when the weather will not cooperate.

The detail that surprises people is the temperature. Nvidia's GB200 coolant enters the rack at about 25°C and leaves around 45°C, moving two litres a second, Jensen Huang said at the launch, as The Register reported. Warm water is the point, not a compromise: if the fluid leaving the building is already hot, a mechanical chiller can often stay switched off, and that chiller is one of the largest single loads in the building. Nvidia claims the GB200 is 300 times more water-efficient than a traditional air-cooled hall — a company number, not an independent measurement. Its roadmap pushes further: the Rubin-generation rack is specified for 45°C liquid, and the Kyber rack planned for Rubin Ultra is a 600-kilowatt cabinet, per Data Center Dynamics.

The analogy: the pot on the stove

You are boiling a big pot of soup and the kitchen is unbearable. Option one is the extractor fan: cheap to install, and if the pot gets twice as big the fan has to move twice as much air, which eventually means a wind tunnel and a lot of noise. Option two is pressing a coil of cold water pipe against the side of the pot and running the other end to the drain. Almost none of the heat reaches the room, because it is captured at the source and carried out through a pipe a few centimetres wide.

A cold plate is that coil; the CDU is where your coil meets the building's plumbing. And as with the pot, the reason to bother is density: you can put a much bigger burner under a much smaller pot if the heat has somewhere to go.

Common misconceptions

"They run water through the servers." No. Fluid runs through sealed plates and hoses; the electronics stay dry. Leak detection, drip trays and quick-disconnect fittings are standard, because a leak in a 120-kilowatt rack is an expensive service call.

"Liquid cooling saves energy." It saves the wrong energy. The chips draw the same watts either way. What goes away is the fan power inside the rack and the chiller load outside it, plus floor space — you fit far more compute in the same hall. Efficiency improves; consumption does not fall.

"It is new." IBM water-cooled mainframes in the 1960s and Cray supercomputers in the 1980s. What is new is doing it at commodity scale, in racks a cloud provider buys by the thousand.

"It is one thing." Direct-to-chip, where plates sit on the hottest components, is what nearly all AI racks use. Immersion cooling — lowering whole servers into a tank of non-conducting fluid — exists, works, and is rare outside specialised high-performance computing sites.

The honest trade-off is the retrofit. An air-cooled hall has no manifolds, no CDUs, and no staff trained to service a wet rack, which is why new AI capacity is designed liquid-first and old capacity mostly is not. The other trade-off is lock-in: Nvidia now publishes a reference design for the whole thermal chain, and suppliers that qualify against it land on a very short list.

Do you think cooling, rather than chips, is the real limit on how fast AI can grow? Write to us via our contact page.

Related reading: AI 101 — What is an AI data center? · AI 101 — How Much Power Does AI Actually Use? · AI 101 — What is a GPU, and why does AI need one?

Get SiliconNoon in your inbox

One email a day with the AI stories that matter — what happened, the numbers, and why it matters. No spam, unsubscribe in one click.

Read more

Google books Blue Origin's New Glenn for orbital AI compute in 2029

Google books Blue Origin's New Glenn for orbital AI compute in 2029

Google has hired a rocket for its data centers in the sky. Blue Origin's New Glenn will fly Project Suncatcher's first dedicated heavy-lift mission in the first quarter of 2029, carrying several satellites into sun-synchronous orbit — each with computing capacity "roughly at the scale of a data center rack," Google said on Thursday. The launch services agreement is the project's first commitment to a specific vehicle and a specific year.

Nebius Q3: revenue up 571%, but 2027 capex jumps to $40-45 billion

Nebius Q3: revenue up 571%, but 2027 capex jumps to $40-45 billion

Nebius just posted the best quarter of its life as a public company and gave back 10% of its share price for it. The numbers beat. The plan for 2027 is what the market refused to pay for. The quarter was a genuine beat Nebius reported third-quarter revenue of $980.8 million, up 571% from $146.1 million a year earlier, ahead of the $907.5 million LSEG consensus, according to the company's results release and CNBC's report on the print.

Deep Dive — OpenAI's $30B round fell to $20B at a $1.25T valuation

Deep Dive — OpenAI's $30B round fell to $20B at a $1.25T valuation

In September, OpenAI went to investors asking for $30 billion at a $1.4 trillion valuation. On Thursday night the Financial Times reported the target is now about $20 billion at $1.25 trillion, because some investors cut the size of their commitments amid the AI selloff, according to the FT. Hours earlier, the Wall Street Journal had put a number on how far the round has actually got: roughly $11 billion committed, less than half the original target, the Journal reported. That is the single…

Deep Dive — AI neoclouds hit a credit wall as SpaceX borrows $40B

Deep Dive — AI neoclouds hit a credit wall as SpaceX borrows $40B

On Sunday evening, Elon Musk announced on X that SpaceX had closed $40 billion of debt to buy Nvidia GPUs, and used the announcement to deliver a verdict on his competitors: the debt markets have "almost entirely shut off access to money for any company that doesn't have substantial revenue independent of AI data centers," and "the 200+ neoclouds are going to see guppy-level death and consolidation," he wrote. The $40 billion is the number everyone will quote.