METR: Anthropic's progress doubled, its own alarm may have rung in July

METR spent eleven weeks inside Anthropic's research pipeline and came back with the number the company has been circling for a year: its models are improving roughly twice as fast as they were before October 2025. The auditors also say something sharper than the headline — Anthropic's own early-warning threshold for automated R&D may already have been crossed on July 15, and the public risk report that said it hadn't understated the evidence.

Neither finding is a claim that a machine woke up. Both are about a slope, and about who gets to see the data behind it.

What METR was allowed to see

The engagement ran August 12 to October 29 with 2.5 full-time staff: benchmark scores for unreleased models, training compute by domain, reinforcement-learning metrics, internal forecasts, lines of code, agent inference spend, and roughly 15 interviews. METR took no payment and worked on Anthropic devices for security reasons. The full report runs about 100 pages; the public version is a fraction of it, per METR's assessment.

One clarification: this is not the "employee-like access" project Anthropic announced on October 15 — METR calls it separate and narrower. The access did exceed expectations. Anthropic shared more than the contract required, usually within days, and lent its two highest-context researchers on R&D measurement to the auditors. The criticism that follows is not that Anthropic hid the filing cabinet. It is that Anthropic kept the key to the pages.

Two years of progress in one

The measurement uses AECI, Anthropic's version of Epoch's Capabilities Index, a composite that bundles many benchmarks into one capability score. Before October 2025, the models gained about 15 points a year. Since then the trend is 29 — a 1.9x acceleration, 95 percent confidence interval 1.5 to 2.8. The past twelve months bought as much benchmark progress as the two years from Claude 2.1 to Sonnet 4.5.

Anthropic's Haiku 5.5 system card put the figure at 1.7x. METR's number is higher mainly because it could see three unreleased models that finished training between July and October. Robustness checks land between 1.1x and 2.9x.

It is not a compute story: METR says it received convincing evidence the acceleration was not driven by faster growth in training compute. It is mostly an AI story — best guess, a majority of the current acceleration, perhaps 80 percent, from Anthropic's own use of AI in research, with substantial uncertainty and an important split. The acceleration that began in late 2025 probably had significant non-AI causes; AI's share grew after. One researcher interviewed put AI's contribution at 15 to 20 percent and said they did not believe their colleagues' self-reports.

The self-reports are striking and nearly worthless as evidence, which METR says itself: informal Slack polls put personal speedup at about 1.2x with Claude 3.7 Sonnet and 12x with internal Model 4 in October, which METR calls "very loose evidence" — low-effort answers, unrepresentative, likely overstated. It weighs the interviews more heavily. Experiments that took a day or two to set up now feel close to free; one researcher's summary was that the team is bottlenecked on ideas and compute, not code.

The cross-lab check is the quiet corrective. Across public models from OpenAI, Google and xAI the trend rises from 17 to 24 points a year — 1.4x, not 1.9x. METR offers two readings and picks neither: Anthropic may have been sub-frontier in 2025, so its baseline was understated and 1.4x is closer to true; or its models were less benchmark-optimized, which would understate the 1.4x. Either way, the doubling is a claim about one company, not the field.

The sentence that matters

Anthropic's public August risk report told readers its "leading indicators point to a picture of meaningful acceleration starting in early-to-mid 2025, though by less than a factor of 2," and concluded the automated-R&D risk was low, with less confidence than in prior reports.

METR had the unredacted version. Its review finds the private report's central estimate of the current rate of progress implies an acceleration of 1.97 — a figure METR calculated by combining two numbers in the private document, since 1.97 does not appear in it. A reader takes "less than a factor of 2" to mean meaningfully below two; "around 2" is the honest description, METR's review concludes. The private report also concedes it "seems plausible we do exceed this factor of 2 overall." METR's verdict: researchers on both sides lean toward the threshold not having been crossed that day, but it is plausible that it was, and the public document understates both the acceleration and the uncertainty around it.

Keep the threshold straight, because it is a warning light rather than a verdict. Under Anthropic's scaling policy, automated R&D is triggered either by models that could substitute for its entire research staff at competitive cost, or by a doubling in the overall pace of progress plausibly caused by automating research. METR agrees the first condition was not met, and that acute catastrophic risk from July 15 models was low. The disagreement is over the second condition, and over how confidently the company said no.

An audit the audited can redact

Anthropic held unilateral redaction rights over all non-public information and used them on a majority of the evidence METR rated most informative: forecasts for future models, breakdowns by training step, the bottleneck analysis, METR's more detailed attribution estimates. One chart in the public report carries the words "FAKE DATA" in large letters, standing in for a redacted figure. METR's summary is blunt for a document its subject had to clear: the public has to rely on its judgment "much more than would be ideal," and the report does not achieve transparency about the evidence.

Then there is the part that was removed rather than redacted. The contract originally included verifying how Anthropic splits compute between capabilities and safety research. Anthropic cut that scope midway, before METR reached any conclusion, saying it wanted to focus on risk assessment first and would revisit compute verification later. Daniel Kokotajlo, the researcher behind the AI 2027 scenario, made the obvious objection: if you cannot verify the compute split, you cannot verify a promise to slow down, as The Decoder reported.

So the most consequential outside audit a frontier lab has ever permitted is also one where the lab chose which evidence the public sees, then deleted the section about where the money goes. METR got unprecedented access to the answer and almost no ability to show the work.

What it changes

If the 80 percent estimate holds, the constraint on frontier progress has moved from engineers to ideas and compute — and the company said as much under securities law three days before the auditors published, in an amended IPO filing whose risk factors forecast that future models may cross this exact threshold: Anthropic says recursive self-improvement may be near. Anthropic had also published its own tally, putting Claude at 26 percent of research tasks "led" in August, judged by a Claude. METR is the first outside read on the same question from different data, and it lands in the same place: direction real, magnitude arguable — Claude now leads 26% of the research at the lab that built it.

The commercial read is less comfortable. Anthropic's balance sheet assumes revenue re-accelerates enough to service infrastructure commitments that reached $895.4 billion by September 30, and capability acceleration is the engine under that assumption. An auditor confirming the engine runs is good for the pitch and awkward for a disclosure that said the gauge was reading under two.

What to watch

The October 15 employee-like access project is the next test, and it will be judged on whether it ships with the same unilateral redaction terms. So will the compute-allocation verification Anthropic deferred: a safety split nobody outside can check is a number, not a commitment. And the next system card is the cheap tell — Haiku 5.5 reported 1.7x while METR, with three more models in view, measured 1.9x. If the gap widens, the pattern is that the public number lags the internal one, which is this finding in miniature.

If an audit can be redacted by the company being audited, what would make it credible? Write to us via our contact page.