AI 101 — What is gradient descent?

Gradient descent is the loop almost every AI model runs to get better: measure how wrong its answers are, work out which direction would make the error smaller, and nudge every one of its numbers a little that way. Then do it again — millions of times.

That is the whole idea. It is older than the computer, it fits on a napkin, and it is still what happens inside a training run that costs tens of millions of dollars and burns through a power station's worth of electricity.

Why it matters right now

Every headline number about AI training is really a gradient-descent story. When DeepSeek reported that its V3 model took 2.788 million H800 GPU hours to train, that was the price of running this loop over 14.8 trillion tokens of text — and the team also noted, as a point of pride, that the run had no irrecoverable loss spikes, per the DeepSeek-V3 technical report. A "loss spike" is exactly what it sounds like: the number the loop is trying to push down suddenly jumps up, and the run is in trouble.

It also matters on your laptop. When you fine-tune a small model, the settings people argue about — learning rate, batch size, warmup, epochs — are all gradient-descent settings. Get the learning rate wrong and the model either learns nothing or turns to mush. Overcook it and the model memorises its training examples: that is overfitting, and it starts with a step size.

And it is the reason scaling laws can be predicted at all. Training curves are smooth in a way that lets labs forecast a run's final quality from its first few percent — the basis of the scaling laws that decide how big the next model gets.

The mental model: a hill you can only feel

Picture the model as a point standing on a landscape. Every position on that landscape is one complete setting of the model's weights; the height at that position is the loss — how badly the model is doing on the training data. Training means getting to low ground.

The trouble is that the landscape has millions, billions, sometimes trillions of dimensions. You cannot see it. All you can do is feel the slope under your feet. That slope is the gradient: a list of numbers, one per weight, saying "in this direction the loss goes up, in that direction it goes down, and here is how steeply."

So you take a step downhill. Then you re-measure the slope at the new spot and take another. The size of the step is the learning rate, and it is the single most consequential number in the whole process.

Where does the slope come from? That is the job of backpropagation, which we explain in what backpropagation is: it walks backwards through the network computing, for each weight, how much that weight contributed to the error. Gradient descent is the part that actually moves the weights. People use the two names interchangeably; they are a measurement and a decision, not the same thing.

The foggy mountain

You are hiking down a mountain in thick fog. You cannot see the valley. You can only feel which way the ground tilts under your boots, and you can see about one step ahead.

Take small steps and you will get down — eventually. Take enormous strides and you will overshoot the valley floor, climb the far wall, overshoot again, and end up bouncing around uselessly or striding straight off a cliff. That is a learning rate set too high. Take microscopic shuffles and you will still be on the mountain at nightfall: correct direction, no progress.

Now add the refinements that make real training work. Instead of judging the slope from a single glance, you average the tilt over a batch of examples, which smooths out the noise — that is stochastic gradient descent, and it is why training uses batches at all. You give the hiker momentum so a consistent downhill direction builds speed and small bumps do not stop them. And instead of one step size for every direction, you let each weight keep its own adaptive step size, shrinking for weights that have been lurching around and growing for ones that have been crawling. That last trick is essentially what the Adam optimizer does, introduced by Diederik Kingma and Jimmy Ba in 2014 and still the default choice for training large language models a decade later.

Common misconceptions

"Gradient descent finds the best possible model." It finds a low point, not the lowest point. The landscape is not a clean bowl; it is full of valleys, ridges and plateaus. Which valley you land in depends on where you started — which is why the random seed used to initialise a model can change what it learns.

"It's how the model stores facts." It does not store anything directly. Gradient descent only turns dials; the knowledge is an emergent side effect of turning millions of dials in directions that reduce prediction error on text.

"A bigger learning rate trains faster." Up to a point. Past it, the loss goes up instead of down, and no amount of extra compute rescues you. Most large runs warm the rate up slowly at the start for exactly this reason, then decay it near the end so the model settles rather than keeps jittering.

"Reinforcement learning replaced it." No. RLHF and GRPO change what the model is rewarded for, not how the weights move; the update at the bottom is still a gradient step. Same for fine-tuning, same for distillation. The objective changes, the machinery does not.

"Loss going down means the model is getting better." Loss is measured on data the model sees. It can fall steadily while the model gets worse at everything else — the gap between the training curve and the real world is the whole subject of generalisation, and it is why labs watch held-out benchmarks rather than the number they are optimising.

Where to learn more

The clearest free walkthrough of the mechanics, with sliders and worked examples, is Google's Machine Learning Crash Course section on gradient descent. For the optimiser most large models actually use, read the Adam paper; for what training looks like at frontier scale, the DeepSeek-V3 report is unusually candid about stability. And for the layer underneath — how the slope is computed at all — start with what backpropagation is.

Have you watched a fine-tune fall apart and traced it back to a step size? Write to us via our contact page.

AI 101 — What is a reasoning model?

A reasoning model is a language model trained — usually with reinforcement learning — to write a long chain of thought before it answers, so it can spend more computation on a hard problem instead of firing back the most likely next sentence. The label arrived with OpenAI's o1 in September 2024. Before it, a chatbot answered in one pass: read the question, predict the reply token by token. o1 added a visible delay and a trail of working behind the answer.

AI 101 — What is GRPO (group relative policy optimization)?

GRPO is the reinforcement-learning method behind most open reasoning models: instead of training a second neural network to judge how good each answer is, it has the model answer the same question many times and rewards whichever answers beat the group's own average. The name is an initialism nobody says out loud — group relative policy optimization — but the idea fits in a sentence, and once you have it a lot of recent AI news stops sounding like alchemy.

AI 101 — What is model collapse?

Model collapse is what happens to a generative AI model that learns mostly from other AI models: it slowly stops representing the full range of real data, and the loss is not recoverable by training longer. That is the plain version. The precise version, from the 2024 study that named the effect, is that "indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear" — published in…

Claude proved 123 open-source libraries can't crash on RISC-V

Two days after two PHP bugs put a slice of the internet into an unscheduled maintenance window, Anthropic published the closest thing to an antidote anyone has shipped: Claude wrote formal proofs that 123 open-source libraries cannot crash, at the assembly level, for any input up to 64 MB. The qualifier at the end of that sentence is where the real story is. What they actually proved Anthropic worked with Theorem, a verification startup, and pointed Claude at Google's OSS-Fuzz corpus — the…

AI 101 — What are model parameters?

A parameter is a number inside an AI model that training adjusts, and the "70B" in a model's name means 70 billion of them. They are not facts, not rules and not code. They are the settings the model learned, and once training stops they are the model: copy those numbers into a file and you have shipped the whole thing. Why the number sits in the name Model names carry parameter counts because the count answers the question buyers actually ask: will this fit, and what will it cost to run?

AI 101 — What does GPT stand for?

GPT stands for Generative Pre-trained Transformer — the three-word recipe behind OpenAI's language models, from GPT-2 to today's GPT-6. It is also the most-abbreviated phrase in AI: most people who type "GPT" into a search box are really asking what the letters mean and why they keep appearing in the news. Why it matters right now Every model name in the headlines is a small riddle.

AI 101 — What is AGI?

AGI — short for artificial general intelligence — is a hypothetical AI system that could do any intellectual job a person can do, instead of being good at one narrow task. Every AI you can actually use today is narrow: it writes, translates, spots patterns in scans, plays games — each system built for its lane. AGI names the destination where one system covers all the lanes.

AI 101 — What is reinforcement learning?

Reinforcement learning is how a machine learns by trial and error: it tries things, notices which attempts earned a reward, and drifts toward the behaviour that earns more of it. There is no answer key involved. In the supervised learning most people picture, someone hands the model millions of labelled examples and it learns to copy them. In reinforcement learning, nobody says what the right move is — the machine acts, gets a number back telling it how well that went, and adjusts.