AI 101 — What is a KV cache?

A KV cache is the working memory a language model keeps of the text it has already read, so it can write the next word without redoing the work for every word that came before.

That is the short answer. The reason it deserves an explainer is that this cache — not the model's weights — is usually what decides how many people one GPU can serve at once, and therefore what long conversations and agent loops actually cost.

Why it matters right now

Every agent that runs for twenty minutes and every chat you keep open all afternoon is, from the serving side, a cache-management problem. The model's weights are a fixed cost, loaded once. The cache is the variable cost, it grows with every token in the conversation, and it is why "how big is the context window" turned into the most consequential spec number in AI.

The consequence is visible in how serving software is built. vLLM's PagedAttention work — the change that made their engine a default for open-model hosting — is almost entirely about stopping wasted KV cache memory from capping batch size; they measured 2–4× more throughput at the same latency, per the paper. Nothing about the model changed. They just managed the cache better.

What it actually is

A transformer layer, when it reads a token, produces three vectors from it: a query, a key, and a value. To decide how much attention to pay to earlier words, the model compares the current query against the keys of all previous tokens, then averages their values, weighted by that match.

Without a cache, generating token 500 means recomputing keys and values for tokens 1 through 499 — inside every layer — just to throw them away. Do that for the whole reply and the work grows with the square of the length.

A KV cache does the obvious fix: compute each token's key and value once, store them, and append as generation proceeds. The model still attends to the full history; it just reads stored vectors instead of recomputing them. Hugging Face's cache documentation describes it as storing the attention calculations "so they can be reused without recomputing them."

The mental model: it is a memory bill, not a speed trick

The cache's size is arithmetic, not guesswork. Per token, per layer, you store one key and one value vector for each key/value head. Multiply by layers, by tokens, by concurrent users.

Take Llama 3.1 8B, a perfectly ordinary open model: 32 layers, 8 key/value heads, a head dimension of 128, stored in 16-bit precision. That is 2 × 32 × 8 × 128 × 2 bytes = 128 KiB per token. An 8,192-token conversation holds 1 GiB of cache. The same model at its full 128K context holds 16 GiB — as much memory as the weights, for one user.

The 8 is the interesting number. Llama 3 has 32 attention heads but only 8 key/value heads, because several query heads share one key/value pair. That trick, grouped-query attention, cuts the cache to a quarter of what full multi-head attention needs — the GQA paper showed an existing checkpoint can be converted for about 5% of the original training compute with quality close to the original.

The kitchen analogy

Think of a restaurant kitchen during service. Chopping onions for every dish from scratch would be absurd, so you prep once and keep the chopped onions on the counter. Cooking gets much faster.

But the counter is finite. One cook working through a long order fills it up; two cooks working at once need two sets of boards, and a banquet needs more counter than the kitchen has. That is the KV cache exactly: the work you save is real, the counter space is what limits you, and "serving more customers" means "fit more prep on the same counter" — which is why servers compress the cache, spill older parts to slower memory, or share one prepared base between dishes that start the same way.

Common misconceptions

"It is the model's memory." No. It is per-conversation working state, discarded when the request ends. What people mean by AI memory — saved facts, retrieved documents — lives outside the model, in vector databases and retrieval. The cache remembers the tokens, not the meaning of your preferences.

"The KV cache and prompt caching are the same thing." Related, not identical. The KV cache is the per-request scratchpad. Prompt caching is a product feature that keeps that scratchpad alive between requests so a shared prefix — a long system prompt, a document you ask twelve questions about — is not reprocessed and re-billed each time. We explained the billing side in What is prompt caching?.

"A big context window is a software setting." It is a memory commitment. Doubling context doubles the cache; running four users at once multiplies it again. That is why long-context prices are not linear in the way people expect and why "1M tokens" is a capability claim, not a cheap default.

"Quantizing the model shrinks the cache." Weight quantization and cache precision are separate dials. Serving engines let you store the cache in 8-bit or lower independently, which halves or quarters the memory — at some cost in accuracy, and at some cost in speed when the context is short enough that memory was never the problem.

"It makes the model smarter, or faster at thinking." It changes cost and latency, not quality. The outputs are the same tokens the model would have produced; you just stopped paying to recompute them.

Where to learn more

Hugging Face's cache documentation is the practical tour, including offloading the cache to CPU memory and quantizing it. The PagedAttention paper is readable if you skim the proofs and is the clearest account of why memory management, not model quality, sets serving throughput. If you want the layer-by-layer mechanics, our explainer on what AI inference actually does picks up where this one stops.

Related reading: our analysis of why a plain LRU policy beat three smarter KV-cache policies on 68,000 real agent requests is the practical follow-up — the regime turned out to be capacity-bound, not timer-bound.

Have you hit an out-of-memory error that turned out to be the cache rather than the model? Write to us via our contact page.

AI 101 — What is tool calling?

Tool calling — also called function calling — is a language model's ability to say "I need this specific external action run, with these exact arguments" instead of answering from memory. The model does not run anything. It writes a structured request in a format your code agreed to in advance, and your code does the doing.

AI 101 — What is an AI agent?

An AI agent is a system that uses a language model to do a job for you — not just answer a question, but pick the next step, use tools, and keep going until the work is finished. A chatbot waits for you. An agent takes the goal and acts. That is the whole difference, and it is why the word is on every product page this year. Why it matters right now Software used to automate work by following a script a human wrote in advance. Agents flip that: the model writes the next step as it goes.

AI 101 — What is a GPU, and why does AI need one?

A GPU is a computer chip built to do thousands of small arithmetic operations at the same time — which turns out to be the exact shape of the math inside a modern AI model. The name stands for graphics processing unit, because the chip was designed to draw video games frames. AI is the job it turned out to be extraordinary at. If you follow AI news, the GPU is the noun that never leaves the sentence: who is buying them, who cannot get enough of them, why one chip costs more than a car, why…

AI 101 — What is context engineering?

Context engineering is the work of deciding what text a model sees before it answers: the instructions, the retrieved documents, the tool descriptions, the conversation so far, the notes from last week — and, just as importantly, what to leave out. Prompt engineering is one slice of it. Context engineering is the whole tray. Every time a model writes a reply, it sees exactly one flat block of text and nothing else.

AI 101 — What is an AI browser?

An AI browser is a web browser with an AI model wired into the browser itself, not bolted on as a separate chat tab. The model can see the page in front of you, the tabs you have open, and sometimes your history and logins — and, if you let it, it can act: click, type, fill forms, compare prices, book things. That definition hides the split that actually matters. Reading is one capability. Acting is another.

AI 101 — What is a chiplet?

A chiplet is a piece of a processor that is manufactured as its own small chip, then packaged alongside other small chips so that the finished part behaves like one big chip. Instead of printing one enormous processor, you print several modest ones and wire them together inside a single package. Every flagship AI accelerator shipping today is built this way. It matters right now because the constraint on AI hardware has moved.

AI 101 — What is an AI data center?

An AI data center is a building full of specialized computers — mostly GPUs — designed from the power feed to the cooling pipes to train and run AI models at massive scale. From the outside it looks like any windowless warehouse. Inside, the difference is density: racks of chips packed so tight, and drawing so much power, that ordinary air conditioning and ordinary grid connections no longer do the job. Why it matters right now. These buildings have become the physical front line of AI.

AI 101 — What is prompt engineering?

Prompt engineering is the practice of writing instructions for an AI model so it gives you what you actually wanted — from a one-line question in a chat box to the multi-paragraph specifications companies embed in their products. "Prompt" is just the text you send the model; "engineering" is the unglamorous work of making that text specific enough to work. Why it matters right now.