AI 101 — What is a TPU?

Share
AI 101 — What is a TPU?

A TPU — Tensor Processing Unit — is a chip Google designs to do one thing: the grid-shaped multiplication that neural networks spend almost all their time on. It is not a graphics card with a different badge. It is an application-specific integrated circuit, built for matrix math and not much else, deployed inside Google's own data centers since 2015 and rentable by the slice through Google Cloud.

Why it matters now

On October 20, Google open-sourced TorchTPU, a native PyTorch backend for its chips, announced at PyTorch Conference. Existing PyTorch code now runs on a TPU with minimal changes instead of going through PyTorch/XLA, the older route that developers complained about for years. That removes the software objection — "the hardware might be cheap, but the tooling will eat my quarter" — from the TPU-versus-GPU decision. The hardware question is now worth understanding on its own terms.

The one-paragraph mental model

A TPU is two things bolted together. The first is a matrix multiply unit: a physical grid of multipliers, 128 by 128, wired so that numbers flow through it in a wave and each value gets reused as it passes. Google's Cloud TPU documentation describes it as a 128x128 systolic array, and notes the consequences: dimensions want to be multiples of 128, and anything that does not fill the grid gets padded with zeros that burn power without producing an answer. The second is a compiler, XLA, which takes your model and cuts it into shapes that fit that grid before a single calculation runs.

That compile step is the whole personality of the chip. XLA compiles the graph once for the first batch, and if later batches have different shapes the model doesn't work — Google's docs are blunt that models with dynamic tensors are a poor fit. A GPU schedules work as it arrives and adapts. A TPU wants the plan up front, and pays you back when you give it one.

The analogy: kitchen versus factory line

A GPU is a large restaurant kitchen. Thousands of cooks, each able to chop, fry or plate, reassigned as orders come in, good at everything, brilliant at nothing in particular.

A TPU is a factory line built to make one dish very fast. Ingredients in at one end, finished plates out the other, and per plate nothing in the kitchen touches it. But the line is built for a portion size. Send half portions, change the recipe mid-shift, or ask it to make something else, and it has to be torn down and rebuilt — which is what a recompile is, and why the padding hurts. Fixed recipe, enormous volume: that is the deal.

Scale is the actual product

A single TPU is not the interesting object. Ironwood, Google's seventh-generation TPU, carries 192 GiB of high-bandwidth memory per chip and moves 7.4 terabytes per second; 64 chips in a rack form a "cube," every chip wired to six others in a three-dimensional torus, and cubes are lashed together by an optical circuit switch into a superpod of up to 9,216 chips. All of that shares 1.77 petabytes of memory and delivers 42.5 exaflops of FP8 compute, according to Google's description of the Ironwood stack.

The optical switch is the part worth noticing: if a cube fails, the fabric reroutes light around it and swaps in a spare without stopping the job. At this size the hard problem is not arithmetic — it is keeping thousands of chips talking.

Three things people get wrong

"A TPU is a cheaper GPU." It is cheaper per unit of dense matrix math in some workloads and worse at nearly everything else. Irregular or sparse computation, models whose tensor shapes shift per batch, small models that cannot fill a 128-wide grid, and anything spending its time on element-wise work rather than matrix multiplication — Google's own guidance steers those to GPUs or CPUs. The TPU is a specialist, and specialists have a bad day outside their specialism.

"TPUs only run TensorFlow." Years out of date. JAX is the primary path, TensorFlow still works, and since October 20 PyTorch is native too, with vLLM and SGLang serving backends alongside it. Google says Anthropic plans to use up to a million TPUs, which is not a bet you make on a software stack you cannot run.

"TPU capacity is easy to rent." Availability is the real constraint, not benchmark scores. Ironwood time is not something a startup rents by the hour from a dozen neoclouds the way it rents H100s, which is why an easier port helps Google's largest customers more than its smallest.

Where the history explains the design

Google built the first TPU because it had done the arithmetic on its own growth: in 2013 it concluded that neural network demand could otherwise force it to double the number of data centers it ran. The chip went from design to deployment in 15 months, and the original used 65,536 8-bit integer multipliers rather than floating-point units, on the reasoning that inference rarely needs 32-bit precision — Google's account of the first TPU puts the gain at 15 to 30 times the performance of contemporary CPUs and GPUs.

That origin still explains the machine: pods, optics and a compiler-first software stack all follow from optimizing that one customer.

Where to learn more

Google's introduction to Cloud TPU is the shortest accurate primer, and it points to the JAX team's "How to think about TPUs" chapter, the best free explanation of the trade-offs anywhere. Both are worth an hour if you are pricing a training run. The explainers below cover the neighbouring pieces.

Related reading:

Would you move a production workload onto a TPU if the port is now one config change? Write to us via our contact page.

Get SiliconNoon in your inbox

One email a day with the AI stories that matter — what happened, the numbers, and why it matters. No spam, unsubscribe in one click.

Read more

Zelos builds a 15,000-GPU cluster to train its self-driving brain

Zelos builds a 15,000-GPU cluster to train its self-driving brain

An autonomous-driving company answers the long-tail problem with compute, while the agent layer gets a gate that rules before the action runs and communications teams get a way to watch what AI says about them. Zelos, the Suzhou RoboVan maker founded by ex-Baidu autonomous-driving engineers, says it has built the industry's first "10,000-card" L4 compute cluster — nearly 15,000 accelerators in total — built to train its APEX multimodal foundation model.

AI 101 — What is a large language model?

AI 101 — What is a large language model?

A large language model (LLM) is a very big neural network trained on enormous amounts of text to do one job well: predict the next piece of a text, given everything that came before it — then tuned so that when you talk to it, it actually helps instead of rambling. ChatGPT, Claude, Gemini, Llama, DeepSeek: every chatbot in the news is an LLM, and most AI features in the software you already use have one quietly bolted inside. That prediction job sounds too simple to matter, and it is the whole…

AI 101 — What is a transformer?

AI 101 — What is a transformer?

A transformer is the neural network design almost every modern AI system is built on — an architecture that lets a model look at every word in its input at once and decide which other words matter for understanding it. That mechanism is called attention, and it is the reason AI went from clunky to conversational. The name comes from a 2017 paper by eight researchers at Google, titled "Attention Is All You Need." Before it, language models read text one word at a time, in order, like someone…

AI 101 — What is an AI benchmark?

AI 101 — What is an AI benchmark?

If you read AI news for a week, you'll see the same strange nouns over and over: SWE-bench, Terminal-Bench, GPQA, FrontierMath. Every model launch ships a table of percentages. Nobody explains what the test actually is. An AI benchmark is a fixed set of tasks with a fixed, automatic way to grade them — a standardized exam for models, where every model gets the same questions and the answers are scored by code rather than by opinion. That last part is the whole trick.

Brain organoids classified Braille at 83% — in a trio, not alone

Brain organoids classified Braille at 83% — in a trio, not alone

Two stories today: human brain tissue on a chip decoded a tactile signal, and the data-centre company behind some of the largest AI contracts in Europe turns out to be six months behind on its flagship site. Human forebrain organoids grown on a microelectrode array classified Braille letters with 61 percent accuracy on their own and 83 percent when the responses of three organoids were combined, in a peer-reviewed study from the University of Bristol.

AI 101 — What are model parameters?

A parameter is a number inside an AI model that training adjusts, and the "70B" in a model's name means 70 billion of them. They are not facts, not rules and not code. They are the settings the model learned, and once training stops they are the model: copy those numbers into a file and you have shipped the whole thing. Why the number sits in the name Model names carry parameter counts because the count answers the question buyers actually ask: will this fit, and what will it cost to run?