AI 101 — What is a GPU, and why does AI need one?

A GPU is a computer chip built to do thousands of small arithmetic operations at the same time — which turns out to be the exact shape of the math inside a modern AI model. The name stands for graphics processing unit, because the chip was designed to draw video games frames. AI is the job it turned out to be extraordinary at.

If you follow AI news, the GPU is the noun that never leaves the sentence: who is buying them, who cannot get enough of them, why one chip costs more than a car, why data centers are sited next to power stations. Every model you have used was trained on tens of thousands of them and answers your questions on a handful.

The mental model

A processor's job in AI is multiplication and addition, repeated at absurd scale. A language model is a stack of matrices — grids of numbers — and each layer multiplies your input by those grids, adds the results, and passes them along. The transformer architecture behind ChatGPT-style models is, mechanically, a long chain of these matrix multiplications, as the original transformer paper lays out.

Here is the fork in the road. A CPU — the general-purpose chip running your operating system — has a few dozen very capable cores. Each can do almost anything, one instruction after another, extremely fast. A GPU has thousands of much simpler cores that all execute the same instruction on different data at once. For work that is one long chain of dependent steps (running a spreadsheet, following an if-then), the CPU wins. For work that is a million independent multiply-adds summed at the end — which is what a matrix multiplication is — the GPU wins by a wide margin, simply because it can do more of them in the same second.

Take NVIDIA's H100, the chip that defined the last AI buildout: 16,896 of those simple cores, 80 GB of memory attached, and a pipe that moves 3.36 terabytes per second between that memory and the chip, according to the H100 specifications.

The analogy: counting a stadium

Imagine counting 80,000 votes by hand. Put one brilliant clerk on it — that is a CPU core. Fast, accurate, able to handle any weird ballot, but the job finishes when it finishes. Now hand each section of the stadium to its own clerk, let them all count simultaneously, then add 80,000 small totals — that is a GPU. Nobody in the second room is smarter than the first clerk. There are just thousands of them working in parallel, and the final summing is cheap.

That last detail is why the GPU needed a new way to program computers before it was useful for anything but graphics: splitting a problem into thousands of identical pieces and combining them is a different craft from writing one clever sequence. NVIDIA's answer was CUDA, the software layer that made its chips generally programmable, and it is the main reason the company's lead survived several generations of better-funded rivals.

Why it matters right now

Answering a question uses the chip differently from training a model. Each word a model generates requires streaming its weights — tens or hundreds of gigabytes — out of memory and doing a modest amount of arithmetic with them. The arithmetic is quick; the reading is not. Databricks' engineering team measured this on production serving stacks and found that the H100's 2.15x memory bandwidth advantage over the A100 translated into 36% lower latency at batch size one, per Databricks' inference benchmarks. That is why "memory bandwidth" now appears in every accelerator announcement: for chat, the pipe matters as much as the math.

It also explains the rule you hit the moment you try to run a model yourself: capacity first. A 70-billion-parameter model at 16-bit precision needs about 140 GB just to hold its weights, so it does not fit on a single 80 GB H100 — it needs at least two, wired together with fast links. What AI inference is covers why splitting a model across chips costs more than the sum of its parts.

Common misconceptions

"GPUs are just faster CPUs." No. Per operation, a single CPU core is often quicker. The GPU wins on throughput — thousands of slower lanes beating a few quick ones on work that parallelizes. Hand a GPU a task with one long chain of dependencies and it sits idle.

"GPUs are for graphics; AI is a side business." The reverse is now true. NVIDIA's data-center revenue is several times its gaming revenue, and its flagship AI chips ship with no display output at all.

"Any GPU will run any model." Capacity is a hard wall, not a slowdown. If the weights do not fit in memory, the model does not run, or runs so slowly it may as well not — which is why quantization, the trick of storing each number in fewer bits, is the other half of the story.

"A bigger GPU means a smarter model." Hardware decides how large a model you can run and how fast it answers, not how good it is. A well-served small model can beat a large one on the task it was trained for.

"GPUs are about to be replaced." The alternatives are real: Google's TPUs and Amazon's Trainium are custom chips built for the same arithmetic, and most large labs are now designing silicon. But the switching cost is software, not hardware, and CUDA's head start is deep. What a TPU is covers the challenger side.

Where to learn more

NVIDIA's H100 product page is the marketing version, though the specifications are accurate — including the vendor claim that its Transformer Engine speeds up large language models by up to 30x over the previous generation, which is a best-case number and should be read as one. For serving realities, the Databricks post above is the clearest public write-up of what actually limits a deployed model. And for the software side, PyTorch's documentation is where most people meet their first GPU tensor.

Would you rent, buy, or wait out the GPU shortage for your own AI work? Write to us via our contact page.

AI 101 — What is a chiplet?

A chiplet is a piece of a processor that is manufactured as its own small chip, then packaged alongside other small chips so that the finished part behaves like one big chip. Instead of printing one enormous processor, you print several modest ones and wire them together inside a single package. Every flagship AI accelerator shipping today is built this way. It matters right now because the constraint on AI hardware has moved.

AI 101 — What is an AI data center?

An AI data center is a building full of specialized computers — mostly GPUs — designed from the power feed to the cooling pipes to train and run AI models at massive scale. From the outside it looks like any windowless warehouse. Inside, the difference is density: racks of chips packed so tight, and drawing so much power, that ordinary air conditioning and ordinary grid connections no longer do the job. Why it matters right now. These buildings have become the physical front line of AI.

AI 101 — What is CUDA?

CUDA is the software layer that lets ordinary programs run on NVIDIA's graphics chips — and it is the foundation nearly all modern AI is built on. Short for Compute Unified Device Architecture, it first shipped on February 16, 2007, and it turned graphics cards — machines designed to draw video games — into general-purpose computers that any developer could program. Before CUDA, using a graphics chip for serious math meant writing code in the chip's own graphics language, in fragments, with…

AI 101 — What is an NPU (neural processing unit)?

An NPU is a chip built for one job: running the math that neural networks do, at a fraction of the power a general-purpose CPU or a graphics GPU would burn. The name covers anything from the block inside your phone's processor to a USB stick in a robot to a rack card in a datacenter. Vendors also call it an AI accelerator or a neural engine, and it is the hardware reason "on-device AI" stopped being a marketing phrase. Why this matters right now The NPU is the quiet spec war of 2026.

AI 101 — What is HBM (high-bandwidth memory)?

HBM is the stacked memory package that an AI accelerator's processor can read from and write to many times faster than normal RAM — and right now it is the single most expensive, scarcest part of the AI buildout. If you have read that the memory shortage is raising chip prices, throttling roadmaps, or forcing startups to rethink fabs, HBM is the "memory" in that sentence.

AI 101 — What is speculative decoding?

If you've ever wondered why an LLM feels slow to type out long answers — or why Nvidia's engineers spent a September 2026 blog post telling model designers how many attention groups to use — the answer is a technique called speculative decoding. Speculative decoding is an inference speedup where a small, fast "draft" model guesses several upcoming tokens, and the big target model verifies them all in a single pass instead of generating one token at a time.

Houmo moves compute inside memory — and A20 enters production

Beijing chipmaker Houmo used its partner conference this week to lay out two compute-in-memory routes at once — and to say the first chip built on the newer one is already in production. Houmo's A20, its first chip built on a 3D compute-in-memory architecture, has entered production. The part supports up to 16 chips fully interconnected, claims a 50% improvement in compute utilization over the previous generation, and covers models from 3B to 30B parameters, according to co-founder and…

AI 101 — What is prompt engineering?

Prompt engineering is the practice of writing instructions for an AI model so it gives you what you actually wanted — from a one-line question in a chat box to the multi-paragraph specifications companies embed in their products. "Prompt" is just the text you send the model; "engineering" is the unglamorous work of making that text specific enough to work. Why it matters right now.