AI 101 — What is tokens per second (tok/s)?

Tokens per second is how fast a language model writes: the number of word-fragments it produces each second once it has started answering.

It is the spec that has quietly moved to the top of every model release. Context windows used to be the number vendors fought over; now the second line of the announcement is a speed claim, usually with a hardware name attached and a batch size buried in the fine print. Here is what the figure measures, what it hides, and how to tell whether a number is good.

Why it matters right now

DiffuSpace's Clover 35B — a diffusion language model that our deep dive on Clover's Qwen parity at 680 tok/s covered this week — claims 680 tokens per second on a single GB300 at BF16, batch size one, per the launch blog. Inception's Mercury 3 quotes 1,068, as we noted when it launched. Those numbers exist because the thing buyers now feel is not intelligence but waiting.

The reason is agents. A single chat reply at 30 tokens per second feels fine — it types faster than you read. An agent that takes twelve steps, each producing 800 tokens of reasoning and tool calls, generates around 9,600 tokens before it finishes: roughly five minutes of pure generation at 30 tokens per second, about 30 seconds at 300. Same model, same intelligence, different product. If you have watched a coding agent sit on "thinking" for four minutes, you already know which column of the spec sheet that came from.

What the number actually measures

Serving a prompt happens in two phases, and tokens per second describes only the second one.

Prefill is the model reading your prompt. All those tokens go through the network at once, so this phase is limited by raw compute; its duration is what providers call time to first token. Decode is the model writing the answer, one token at a time, each token requiring a fresh pass through the whole network. Tokens per second is the rate of that second phase, which is why prefill-heavy workloads (long documents, short answers) and decode-heavy ones (short prompts, long answers) can feel completely different at the same advertised speed.

There is also a distinction vendors rarely spell out. A single user's rate is latency. The total tokens per second a server produces across many users is throughput. Batching lets a GPU serve several conversations per pass, which raises throughput substantially while barely changing any one user's rate — that is the economics of an inference provider, and why the same model can be "200 tokens per second" in a demo and noticeably slower on a busy API.

The mental model: it is a pipe, not a pump

Every new token requires reading essentially all of the model's weights out of memory, once. Not computing with them — reading them. So the ceiling on decoding speed is close to arithmetic:

tokens per second ≈ memory bandwidth ÷ bytes of weights read per token.

Databricks' engineering writeup on LLM inference performance puts it plainly: at small batch sizes, "the speed is dependent on how quickly we can load model parameters from GPU memory to local caches/registers, rather than how quickly we can compute on loaded data." They even define a metric for it, Model Bandwidth Utilization, with a worked example — a 7B model at 16-bit taking 14 milliseconds per output token is moving 14 GB in 14 ms, which works out to 1 TB/s of memory traffic.

Run the same arithmetic on a familiar model. Llama 3.1 8B in 16-bit is about 16 GB of weights. On a machine that reads memory at 1 TB/s, the ceiling is roughly 1,000 ÷ 16, or about 62 tokens per second — and real systems land under that, because memory never runs at its peak. Serve the same model in 8-bit and the weights halve to 8 GB, so the pipe can fill twice as often: the ceiling roughly doubles. That single relationship explains most of the speed claims you read.

Think of a reservoir behind a pipe. Each token is one full ladle out of the reservoir, and the ladle must pass through the pipe every time. You cannot ladle faster by buying a bigger pump — the pump was never the limit. You ladle faster by widening the pipe (more memory bandwidth) or by shrinking the reservoir (fewer bytes per weight, which is what quantization does, or a mixture-of-experts model, which reads only the experts a token actually uses).

What a number is worth depends on four dials

The same weights report wildly different speeds depending on hardware (Clover's 680 tokens per second becomes 515 on an H200, per the same DiffuSpace post — a card from the previous generation), precision (16-bit, 8-bit, 4-bit: each step halves or quarters the bytes read per token), batch size (batch one is the best case for a single user and the worst case for a server's economics) and the prompts (vendor benchmarks use their own workloads, at their own answer lengths).

One more trap: tokens are not words, and tokenizers differ. Databricks cites Anyscale's finding that Llama 2's tokenizer produces about 19% more tokens for the same text than ChatGPT's — so identical tokens-per-second figures do not mean identical reading speed across models.

How fast is fast?

A useful yardstick is your own eyes. The best meta-analysis of reading speed — 190 studies, 18,573 participants — puts average adult silent reading of non-fiction at 238 words per minute, per Brysbaert in the Journal of Memory and Language.

A token is roughly three-quarters of a word, so 10 tokens per second is about 450 words per minute — nearly twice as fast as you read. In practice: under about 5 tokens per second, text arrives slower than thought and feels broken; 20 to 40 is comfortable chat; several hundred is faster than a human can follow, which is the point — the consumer is another program.

Common misconceptions

"Higher tokens per second means a better model." It means a faster one. A 35-billion-parameter model holds more than a 7-billion one, and the small one will usually stream faster.

"Tokens per second measures how fast it thinks." For reasoning models, the token stream includes thinking you may never see. A fast model that spends 3,000 tokens reasoning before a one-line answer can finish after a slower model that answers directly.

"Thousand-token-per-second models make humans obsolete readers." Nobody reads at 1,000 tokens per second. What changes is that software — agents, evaluators, batch pipelines — can consume output at that rate, which is why the speed race is really about machine-to-machine work.

Where to learn more

Databricks' inference engineering post is the clearest public explanation of why decoding is memory-bound, and it is readable without a hardware background. For the phase-by-phase mechanics, our explainer on what AI inference actually does covers prefill and decode in more detail, and what a KV cache is explains the other half of the memory bill — the part that grows with conversation length rather than model size.

Related reading: Deep Dive — Clover 35B: a diffusion LLM at Qwen parity, 680 tok/s is a worked example of how to read a speed claim sceptically — including which baseline the flattering number was measured against.

When you compare two models, do you trust the tokens-per-second figure or test it yourself? Write to us via our contact page.

AI 101 — What is AI inference?

Every answer an AI gives you is inference: running a trained model on new input to produce an output. Training is how a model learns; inference is how it works. If training is teaching, inference is doing. Why it matters right now. Training makes the headlines — a new model, a bigger cluster, a record run — but training happens once per model, while inference happens every time anyone asks a question.

AI 101 — What is temperature in an LLM?

Temperature is the setting that controls how randomly an AI model picks its next word: near 0 it always takes the most likely word, higher values give unlikely words a real chance, and 1.0 is the default on most APIs. It is the first knob most people turn, and the one most people misunderstand. Why it matters right now. Temperature has been showing up in the news all week as a variable, not a feature.

AI 101 — What is LoRA (low-rank adaptation)?

LoRA is a way of customizing an AI model by training a tiny add-on instead of rewriting the whole thing — the base model's weights stay frozen, and a small pair of "adapter" matrices carries the change. If you have read that someone "dropped a LoRA" to teach an image model a new style, or that a community fine-tune of an open-weight model was trained on a single GPU, LoRA is the technique underneath.

AI 101 — What is a diffusion language model?

A diffusion language model writes text the way an image generator paints a picture: it starts with a block of noise and refines the whole thing over several passes, instead of typing one word at a time left to right. Almost every AI you have used does the typing. This is the main rival design, and in the last week it went from research curiosity to something running real phone calls and coding agents. The reason it is in the news is Inception's Mercury 2.5 release, which the company calls the…

AI 101 — What is an embedding?

An embedding is a list of numbers that represents the meaning of something — a word, a sentence, a photo, a song — so a computer can compare meanings using arithmetic. The rule is simple and surprisingly powerful: things that mean similar things get similar lists of numbers. That's the whole idea, and it's the foundation of semantic search, recommendation feeds, and most of what people call "AI memory." Why it matters right now Embeddings are the invisible layer under nearly every AI feature…

AI 101 — What is a token in AI?

A token is the chunk of text an AI model actually reads and writes — not a word, not a character, but a piece of a word: "un", "believ", "able". Every language model, from the smallest phone-side model to the biggest frontier system, does its work by predicting one token after another. If you understand tokens, you understand why AI bills look the way they do, why some languages cost more than others, and why a model can count the letters in "strawberry" and still get it wrong. It is also the…

AI 101 — What is RLHF?

Reinforcement learning from human feedback (RLHF) is the training technique that teaches an AI model what people actually prefer — by using human ratings, not just text, as the signal for what counts as a good answer. It is the reason a chatbot says "I'm not sure, let me check that" instead of confidently inventing an answer, and the reason modern assistants are polite, follow instructions, and refuse harmful requests.

AI 101 — What is knowledge distillation?

Knowledge distillation is a way of training a small "student" AI model to imitate a big "teacher" model — so the student picks up much of the teacher's skill while being far smaller, faster, and cheaper to run. If you have read tech news lately, you have seen this word doing heavy lifting without ever getting explained. A ByteDance founder reportedly told staff to stop using distillation even though rivals rely on it.