AI 101 — What is a TPU?

A TPU — Tensor Processing Unit — is a chip Google designs to do one thing: the grid-shaped multiplication that neural networks spend almost all their time on. It is not a graphics card with a different badge. It is an application-specific integrated circuit, built for matrix math and not much else, deployed inside Google's own data centers since 2015 and rentable by the slice through Google Cloud.
Why it matters now
On October 20, Google open-sourced TorchTPU, a native PyTorch backend for its chips, announced at PyTorch Conference. Existing PyTorch code now runs on a TPU with minimal changes instead of going through PyTorch/XLA, the older route that developers complained about for years. That removes the software objection — "the hardware might be cheap, but the tooling will eat my quarter" — from the TPU-versus-GPU decision. The hardware question is now worth understanding on its own terms.
The one-paragraph mental model
A TPU is two things bolted together. The first is a matrix multiply unit: a physical grid of multipliers, 128 by 128, wired so that numbers flow through it in a wave and each value gets reused as it passes. Google's Cloud TPU documentation describes it as a 128x128 systolic array, and notes the consequences: dimensions want to be multiples of 128, and anything that does not fill the grid gets padded with zeros that burn power without producing an answer. The second is a compiler, XLA, which takes your model and cuts it into shapes that fit that grid before a single calculation runs.
That compile step is the whole personality of the chip. XLA compiles the graph once for the first batch, and if later batches have different shapes the model doesn't work — Google's docs are blunt that models with dynamic tensors are a poor fit. A GPU schedules work as it arrives and adapts. A TPU wants the plan up front, and pays you back when you give it one.
The analogy: kitchen versus factory line
A GPU is a large restaurant kitchen. Thousands of cooks, each able to chop, fry or plate, reassigned as orders come in, good at everything, brilliant at nothing in particular.
A TPU is a factory line built to make one dish very fast. Ingredients in at one end, finished plates out the other, and per plate nothing in the kitchen touches it. But the line is built for a portion size. Send half portions, change the recipe mid-shift, or ask it to make something else, and it has to be torn down and rebuilt — which is what a recompile is, and why the padding hurts. Fixed recipe, enormous volume: that is the deal.
Scale is the actual product
A single TPU is not the interesting object. Ironwood, Google's seventh-generation TPU, carries 192 GiB of high-bandwidth memory per chip and moves 7.4 terabytes per second; 64 chips in a rack form a "cube," every chip wired to six others in a three-dimensional torus, and cubes are lashed together by an optical circuit switch into a superpod of up to 9,216 chips. All of that shares 1.77 petabytes of memory and delivers 42.5 exaflops of FP8 compute, according to Google's description of the Ironwood stack.
The optical switch is the part worth noticing: if a cube fails, the fabric reroutes light around it and swaps in a spare without stopping the job. At this size the hard problem is not arithmetic — it is keeping thousands of chips talking.
Three things people get wrong
"A TPU is a cheaper GPU." It is cheaper per unit of dense matrix math in some workloads and worse at nearly everything else. Irregular or sparse computation, models whose tensor shapes shift per batch, small models that cannot fill a 128-wide grid, and anything spending its time on element-wise work rather than matrix multiplication — Google's own guidance steers those to GPUs or CPUs. The TPU is a specialist, and specialists have a bad day outside their specialism.
"TPUs only run TensorFlow." Years out of date. JAX is the primary path, TensorFlow still works, and since October 20 PyTorch is native too, with vLLM and SGLang serving backends alongside it. Google says Anthropic plans to use up to a million TPUs, which is not a bet you make on a software stack you cannot run.
"TPU capacity is easy to rent." Availability is the real constraint, not benchmark scores. Ironwood time is not something a startup rents by the hour from a dozen neoclouds the way it rents H100s, which is why an easier port helps Google's largest customers more than its smallest.
Where the history explains the design
Google built the first TPU because it had done the arithmetic on its own growth: in 2013 it concluded that neural network demand could otherwise force it to double the number of data centers it ran. The chip went from design to deployment in 15 months, and the original used 65,536 8-bit integer multipliers rather than floating-point units, on the reasoning that inference rarely needs 32-bit precision — Google's account of the first TPU puts the gain at 15 to 30 times the performance of contemporary CPUs and GPUs.
That origin still explains the machine: pods, optics and a compiler-first software stack all follow from optimizing that one customer.
Where to learn more
Google's introduction to Cloud TPU is the shortest accurate primer, and it points to the JAX team's "How to think about TPUs" chapter, the best free explanation of the trade-offs anywhere. Both are worth an hour if you are pricing a training run. The explainers below cover the neighbouring pieces.
Related reading:
- AI 101 — What is CUDA? — the software layer Nvidia spent a decade building, and the real reason GPUs are hard to displace.
- AI 101 — What is an AI data center? — the buildings these chips sit in, and why power is the binding constraint.
- AI 101 — What is HBM (high bandwidth memory)? — the memory that decides what fits on one chip.
Would you move a production workload onto a TPU if the port is now one config change? Write to us via our contact page.






