AI 101 — What is quantization?

Quantization is storing an AI model's numbers at lower precision — fewer bits each — so the same model fits in less memory and usually runs faster, at the cost of a small, often invisible loss in accuracy.

That is the whole idea. Everything else is about how much accuracy you lose, and how cleverly you can spend the bits you keep.

Why it matters right now

The arithmetic is what forced it. A 70-billion-parameter model is a list of 70 billion decimal numbers, and in the standard 16-bit format each one takes two bytes: about 140 GB. That does not fit on the 80 GB GPU that costs more than a car, and it certainly does not fit on your laptop. Store the same weights at 4 bits and the file is roughly a quarter the size — call it 35 to 40 GB — which is one card, or a Mac with enough unified memory.

Every "you can run this at home" story of the past two years rests on that shrink. OpenAI shipped its open-weight gpt-oss-120b with its mixture-of-experts weights already quantized to MXFP4 at 4.25 bits per parameter, which is precisely what lets a 117-billion-parameter model run on a single 80 GB GPU and its 20-billion sibling run in 16 GB of memory, according to the gpt-oss model card. Our own radar this week covered colibri, which runs GLM-5.2 at 744 billion parameters in int4 with a 9.9 GB resident footprint by reading the rest off disk — the November 3 Open Source Radar has the numbers.

The mental model

Picture a single weight: 0.017341… In a 16-bit format you get something like five significant digits to describe it. In a 4-bit format you get 16 possible values. There is no way to represent 0.017341 exactly with 16 choices, so the compressor does two things: it picks a small set of representative values, and it stores a scale factor for a block of weights so the set can shift up or down with the local range. The weight becomes "the third value in this block's table," which is a much shorter thing to write down.

Different methods spend the error differently. Round-to-nearest schemes (most GGUF files you download, and AWQ) just snap each weight to the closest representable value, protecting the weights that matter most for the output. GPTQ is smarter: it quantizes one column at a time and nudges the weights it has not touched yet to cancel the error it just introduced, the way you would adjust a budget line to make up for an overspend.

The number that matters is bits per weight. Sixteen is the reference, eight is a clean halving, four is where most local models live, and two — or the 1.58-bit ternary weights in Microsoft's BitNet line — is the experimental edge where the wheels start coming off for some tasks.

The analogy

Quantization is JPEG for model weights. A photo saved as JPEG throws away detail your eye will not miss, and the file shrinks by an order of magnitude; push the quality slider to the bottom and you get visible banding. A quantized model works the same way, with one nasty difference: a bad JPEG looks bad immediately, while a bad quantization can produce a model that chats fluently and then quietly fails the one task you built the product around. That is why the honest way to evaluate quantization is not "does it still talk" but "does it still pass my evals."

What it actually costs

Hugging Face's own survey of quantization schemes notes that the GPTQ method can go as low as 2 bits, but that this "might come with severe quality degradation," and that 4 bits is the recommended trade-off for that family, per the Transformers quantization overview. The gains are real at the other end too: Intel's work on 2-bit kernels reports up to 7× faster inference than a 16-bit pipeline on CPUs and up to 6.7× on its GPUs, the authors write — though that is a speed result on hardware built to run those kernels, not a promise for every machine.

Two things get conflated, so keep them apart. Memory is the obvious win: fewer bits per weight, smaller checkpoint, more of the model in cache. Speed is a second-order effect that depends on what the chip is waiting for. Generating text one token at a time is limited by how fast you can move weights from memory, so a smaller model file often decodes faster. Processing a long prompt is limited by arithmetic, so there the gain is smaller and can even reverse if the hardware has no fast kernel for your bit-width.

Where you meet it

You have already used it. The "Q4_K_M" in a GGUF filename is a 4-bit recipe with a particular mix of precisions per layer. FP8 is the default low-precision format on current datacenter accelerators, and the newest chips add FP4. QLoRA fine-tuning keeps the frozen base model in 4 bits so a single card can train an adapter on top. Serving stacks separately quantize the KV cache — the model's memory of the conversation — because at long context that cache, not the weights, is what fills the GPU.

Common misconceptions

"4-bit means the model knows a quarter as much." It does not lose knowledge, it loses resolution. Weight-only 4-bit typically costs a little accuracy on benchmarks. What bites is compounding: quantize, fine-tune on top, quantize again, and the small errors stack. Well-built releases keep the sensitive parts — attention projections, routers, embeddings — at higher precision and compress only the bulk.

"A 4-bit 70B is a smaller model." It is the same 70 billion parameters, written with fewer bits each. Parameter count is the model's size; bit-width is how tightly you packed it.

"Lower is always faster." Only where a kernel exists. Below the bit-widths the hardware and runtime support well, you fall back to a slow path, or to dequantizing on the fly, and the speedup evaporates.

"It is free compression." It is lossy compression with a quality setting, and the setting interacts with your task. Code and multi-step math are less forgiving than casual chat.

Where to learn more

Start with the quantization section of any model card — it names the format and the bits per weight, which is the only figure that predicts memory. Then read the runtime's documentation for the formats it supports on your hardware, and benchmark your own task rather than trusting a leaderboard average. If you want the theory, the GPTQ and AWQ papers are short and readable, and the Hugging Face overview above maps the families against each other.

Running something locally this month — what bit-width did you settle on, and did it hold up on your own tasks? Write to us via our contact page.

AI 101 — What is tokens per second (tok/s)?

Tokens per second is how fast a language model writes: the number of word-fragments it produces each second once it has started answering. It is the spec that has quietly moved to the top of every model release. Context windows used to be the number vendors fought over; now the second line of the announcement is a speed claim, usually with a hardware name attached and a batch size buried in the fine print.

Deep Dive — OpenAI's 12-hour prompt cache costs 2.5x to write

If you are searching "is OpenAI's 12-hour prompt caching worth it", the honest answer is: not the way most people will use it. OpenAI's October 20 changelog added a longer cache lifetime to the Responses API and priced the write at double the default, which turns a feature that looked like a discount into a prepaid bet on how often you reuse the same prompt prefix. That bet is worth making for a support bot with a pinned knowledge preamble, a coding agent that works in one repo all day, or an…

Google open-sources TorchTPU: native PyTorch on TPUs, no rewrite

For a decade Google sold TPUs on price per FLOP and treated the software gap as somebody else's problem. On Tuesday it stopped: TorchTPU, the native PyTorch backend for its chips, is open source — and DeepSeek spent the same day giving away its serving stack. Google released TorchTPU as an open-source project at PyTorch Conference 2026, and the pitch is blunt — existing PyTorch code runs on a TPU without a rewrite.

AI 101 — What is AI inference?

Every answer an AI gives you is inference: running a trained model on new input to produce an output. Training is how a model learns; inference is how it works. If training is teaching, inference is doing. Why it matters right now. Training makes the headlines — a new model, a bigger cluster, a record run — but training happens once per model, while inference happens every time anyone asks a question.

AI 101 — What is temperature in an LLM?

Temperature is the setting that controls how randomly an AI model picks its next word: near 0 it always takes the most likely word, higher values give unlikely words a real chance, and 1.0 is the default on most APIs. It is the first knob most people turn, and the one most people misunderstand. Why it matters right now. Temperature has been showing up in the news all week as a variable, not a feature.

AI 101 — What is LoRA (low-rank adaptation)?

LoRA is a way of customizing an AI model by training a tiny add-on instead of rewriting the whole thing — the base model's weights stay frozen, and a small pair of "adapter" matrices carries the change. If you have read that someone "dropped a LoRA" to teach an image model a new style, or that a community fine-tune of an open-weight model was trained on a single GPU, LoRA is the technique underneath.

AI 101 — What is a diffusion language model?

A diffusion language model writes text the way an image generator paints a picture: it starts with a block of noise and refines the whole thing over several passes, instead of typing one word at a time left to right. Almost every AI you have used does the typing. This is the main rival design, and in the last week it went from research curiosity to something running real phone calls and coding agents. The reason it is in the news is Inception's Mercury 2.5 release, which the company calls the…

AI 101 — What is an embedding?

An embedding is a list of numbers that represents the meaning of something — a word, a sentence, a photo, a song — so a computer can compare meanings using arithmetic. The rule is simple and surprisingly powerful: things that mean similar things get similar lists of numbers. That's the whole idea, and it's the foundation of semantic search, recommendation feeds, and most of what people call "AI memory." Why it matters right now Embeddings are the invisible layer under nearly every AI feature…