Log-perplexity, explained: OpenAI's 0.99 to 0.68 progress ladder

OpenAI published the number it uses to judge its own progress on Sunday, along with a proposal that other labs agree not to cross a line drawn on it. The interesting part is not the ask. It is the curve underneath.
OpenAI's chief scientist published the company's pretraining loss curve on November 22, with a specific claim attached: each drop in log-perplexity has so far delivered a capability nobody could have predicted from the last one. GPT-3.5 sat at roughly 0.99 on OpenAI's internal evaluation set and could hold a conversation, which became ChatGPT. GPT-4 sat at about 0.91 and could use chain-of-thought reasoning productively, which became the reasoning-model line. Models are now below 0.8, where OpenAI says "transformative capabilities in fields like cybersecurity and mathematics" appeared, and one internal model at roughly 0.76 started producing results on open math problems the company had been posing for more than a year, Jakub Pachocki wrote in the post.
Extrapolating that curve, Pachocki says the next window to watch is ~0.68 to 0.7. He does not know what capabilities appear there, and says OpenAI may not be testing for them today. He does expect the models to get harder to audit: further degradation in chain-of-thought monitoring, and more evaluation awareness — models recognising when they are being tested.
What perplexity actually measures
Perplexity is how well a model predicts the next token, averaged over a held-out set. Lower is better. In practice researchers track its logarithm, which is the same thing as cross-entropy loss, so the numbers in the post are loss values, not raw perplexity.
The reason a compression score predicts capability is the argument the whole post rests on. Training a language model is training a compressor, and a better compressor is a better predictor of unseen data. OpenAI's dataset is drawn from its internal codebase and held fixed, which is why the numbers are comparable across its own models and not obviously comparable to anyone else's.
Two of the steps on that ladder are worth sitting with. Going from 0.91 to below 0.8 is a change of less than 0.12 in loss, and OpenAI describes what came out of it as transformative work in cybersecurity and mathematics — the same window in which its models started clearing open problems that researchers had left alone for years. Then, from below 0.8 to 0.76, the models went from assisting on hard problems to producing results that the Lean and Mathlib communities would later certify. That is a lot of consequence packed into four hundredths of a log unit.
The ask: a limit with a milestone attached
Pachocki's proposal is narrower than "pause", and more concrete than the version OpenAI floated earlier this month. He wants frontier developers to agree shared norms for evaluating and red-teaming models before first use, shared transparency standards, and then common limits — a first threshold, measured by perplexity and other metrics, that no participating lab crosses until monitoring reaches a stated milestone, which he gives as an example: a successor to chain-of-thought monitors.
That is the part with teeth, because it makes the limit conditional on a technical achievement rather than a date. A date can be waited out. A milestone has to be met.
Sam Altman amplified it, calling the post "the most important thing OpenAI has published in awhile" and saying he hopes the field can agree to common limits. Noam Brown, who works on OpenAI's reasoning models, framed the disclosure as a public good: "everyone deserves to know the trajectory of progress, including for internal models."
We covered the policy version of this in November, when OpenAI told Axios it would campaign for "speed limits" after the midterms and could not say what the threshold was — OpenAI's midterm plan: 'AI speed limits' and 'compute campuses'. The post answers that question with a metric, though still not with a number.
Why 2027 is the year in the argument
The case for urgency is not the curve. It is who is climbing it. Pachocki argues the step change will come from AI participation in one specific part of the pipeline: algorithmic research on pretraining. OpenAI has already seen a steep acceleration in its 2026 loss numbers, which it attributes mostly to long-term research and compute investment and only in smaller part to automation — but it says recent models already carry "meaningful design contributions coming from AI."
His analogy is math. For more than a year OpenAI posed open problems to its models as an eval, and at 0.76 they started resolving questions researchers had thought about for decades, one after another, in a single Slack thread. He thinks something similar is plausible in pretraining research, possibly as soon as 2027, because that field has a long history of simple insights going unnoticed for years. And he is explicit that alignment and monitoring progress is "unlikely to be fast enough" to keep up.
The four holes OpenAI admits
The post lists its own objections, which is unusual and worth taking at face value. Perplexity is hard to compare across labs, because datasets and data mixtures differ. It does not correspond to any task anyone cares about. It does not capture advances outside pretraining — chain-of-thought reasoning raises capability dramatically without lowering loss at all. And the floor is unknown: nobody knows the entropy of the data, so as models approach it, progress on perplexity slows while each smaller gain buys a larger jump in capability.
That last point is the one that should worry anyone planning to regulate on this metric. The curve gets flatter exactly where the stakes get higher, which means a limit set at 0.68 could be nearly crossed for months while the capability behind it moves in jumps.
The outside objections are sharper. Reaction on X and Hacker News, collected on the Techmeme page for the post, ran to a single worry in several forms: tie a limit to one number and labs will optimise the number rather than the capability. Researcher Elie Bakouch made the technical version — raw perplexity depends heavily on compute, so efficiency gains would be the better comparison. Others noted that OpenAI published this while it is widely seen as behind Google and Anthropic, and read the timing as convenient. That criticism is cheap, but it is not free: a pacing proposal from a lab that is winning lands differently from one that is not, and OpenAI has now made itself the lab that has to keep publishing the number after the story fades.
What to watch
Three things would tell you whether this becomes a standard or a blog post. Whether any rival lab publishes a comparable curve on a stated dataset. Whether OpenAI releases the held-out evaluation set it is measuring against, which would let outsiders reproduce the ladder instead of trusting it. And whether a number ever appears in a policy document rather than a research post.
The near-term test is smaller and more useful: if a lab sets a limit and then quietly changes the dataset, the metric was decoration. If the numbers keep coming, at the same dataset and the same snapshot, the industry has something it has never had — a shared yardstick that a regulator, a rival or a reader can check.
If every lab published its loss curve tomorrow, would that make AI development safer — or just better benchmarked? Tell us via our contact page.







