METR's time horizon plot: tasks double every four months, judgment lags

Share
METR's time horizon plot: tasks double every four months, judgment lags

Two things landed Friday that pull in opposite directions: a chart saying AI capability is still compounding on schedule, and a White House that wants to tax the companies taking their data centers elsewhere.

METR's refreshed time-horizon report finds that the length of tasks frontier models can complete unaided is still doubling roughly every four months — and that the same models fall short on work where nobody can check the answer automatically.

The metric is easy to state and easy to misread. METR, a nonprofit that evaluates frontier systems, scores each task by how long it took a human to finish it, then measures the longest tasks a model completes at 50% reliability. The resulting plot became the most-shared and most-quoted chart in AI, and, as MIT Technology Review notes in its write-up of the update, the most widely misinterpreted. Its headline finding held: the task length a frontier model could handle doubled about every four months.

The new version expands the suite in the direction that matters for the industry's biggest bet. METR added tasks estimated to take humans far longer than anything in the original set, plus open-ended tasks where success is a judgment call rather than a pass or fail. That matters because of how these systems are trained: reinforcement learning can hammer a model into shape on any task where a grader can declare the answer right or wrong. Coding and math have graders. Choosing a research direction does not.

The results split cleanly along that line. In one new task, models were asked to design AI evaluations that METR could use in its own research, with METR employees scoring the output. Not one AI-designed evaluation was good enough for the staff to consider useful. In another, models had to score philosophical arguments — a proxy for judgment — and they trailed human performance there too.

There is genuine movement at the margin. The newest models, Opus 5.5 among them, now land in what METR calls "borderline above the bar" territory on the evaluation-design task: not yet usable, but close. A year ago, every model was in the "clearly below the bar" bucket. That is the interesting signal in the report — not a plateau, but a frontier that is arriving more slowly in the fuzzy domains than in the checkable ones.

Why it matters comes down to one number the industry has staked a decade of capital on: how fast AI can speed up AI research. Earlier this month, a heavily redacted METR report claimed AI might be accelerating progress at Anthropic by a factor of two, which is precisely the mechanism behind recursive self-improvement — systems that design their own successors. That claim landed in the middle of the largest IPO in history. The new report is the sober counterweight: if research taste is the bottleneck, the exponential curve describes a narrower slice of the work than the timeline debates assume. We made the adjacent argument this morning — that compute caps cannot stop the jump once the capability exists — in AGI vs ASI: why capping compute can't stop the jump.

The counter-case deserves its airing, because it is not fringe. Plenty of researchers argue that training on verifiable tasks generalizes: a model that gets good enough at the checkable work eventually handles the open-ended version, and the judgment gap closes as a byproduct rather than as a separate problem. If that is right, today's shortfall on evaluation design looks like the coding scores of 2023 — embarrassing until it isn't.

The methodological caveats cut the other way, and both sides should feel them. Most completion times in the new suite are estimates rather than measured, because paying humans to sit through tasks that take hundreds of hours is prohibitively expensive. The philosophical-argument task was largely scored by a single rater, so the models are being graded on agreement with one person's taste. And the judgment-heavy tasks are few, which is why METR's own technical staffer Matt Putz told MIT Technology Review that hard-to-verify work is "much harder to study, and especially to quantify," and that clean trends may never be establishable the way they are for easy-to-specify tasks. The honest read is that this report measures a boundary, not a forecast.

The most telling corroboration is that the labs agree. In a Wednesday blog post, OpenAI reported that its models do worse on the real-world tasks its own employees assign than on the easier, verifiable items in the original METR suite. A company selling capability is conceding the same gap its evaluators found — from a different direction, in the same week. UC Berkeley's Sayash Kapoor, who has published on models failing to reproduce human computer science research unaided, told MIT Technology Review that "AI evaluations need to get more creative." That is where this goes next: better judgment benchmarks, not another doubling announcement.


Trump said Friday that the United States will not allow its companies to build data centers abroad and will put a "tremendous tax" on any that do — a threat he floated in public for the first time, before his own staff had heard it.

The setting was a White House press conference a day after a trilateral summit with Xi Jinping and Vladimir Putin in China, where Trump said Xi had congratulated him on US progress in AI, according to CNBC. "I haven't even said that to my people," he told the room. "But I'm telling them now." Alphabet, Amazon, Microsoft, Meta and Oracle already run data centers outside the US, largely to serve nearby clients and satisfy data-residency rules, so the threat lands on existing capacity, not just future projects.

The bind is the story. CNBC's October survey found 64% of registered voters oppose an AI data center within 10 miles of home, and only 13% would welcome one, which is why states and cities keep imposing moratoriums. The administration is now threatening the exit route that local opposition creates — with no tax instrument, no bill and no agency named, and a Democratic-majority Congress being sworn in within weeks, as The Verge notes.

Should a company that builds its data centers overseas to escape a local moratorium pay a federal penalty for it? Write to us via our contact page.

Get SiliconNoon in your inbox

One email a day with the AI stories that matter — what happened, the numbers, and why it matters. No spam, unsubscribe in one click.

Read more

Peking University's 591-run RL study finds uniform sampling wins

Peking University's 591-run RL study finds uniform sampling wins

A Peking University group built the comparison platform, ran the experiments, and reported the boring answer: with everything else held constant, the data policy is not what moves the score. The sharper finding is that the benchmark summary you choose can flip which method looks best. Peking University's Data-Centric AI group ran 591 reinforcement-learning experiments to test whether cleverer data policies beat plain uniform sampling, and could not find one that reliably does.

AGI vs ASI: why capping compute can't stop the jump

AGI vs ASI: why capping compute can't stop the jump

Google DeepMind's research arm has published its map of what comes after human-level AI. The most useful part is not the forecast — it is the warning about the policy everyone reaches for first. The DeepMind Institute's essay "What comes after AGI?" landed on November 19 with a definition of superintelligence that shifts the measuring stick in a useful direction: ASI is a system that "consistently outperforms large groups of human experts on tasks that could take them years to complete, using…

Microsoft's MAI-DxO scores 83% on NEJM cases; doctors 36%

Microsoft's MAI-DxO scores 83% on NEJM cases; doctors 36%

Microsoft's diagnostic system cleared peer review today. The interesting part is not the multiple in the headline — it is which doctors you compare it to, and how much of the win belongs to the model underneath rather than to Microsoft's orchestration. Microsoft's MAI-DxO answered 83% of 304 complex case records from the New England Journal of Medicine correctly, against 36% for primary care doctors who had textbooks and the internet, according to results published in Science on Thursday.

Agent teams cost up to 5x more, barely score higher

Agent teams cost up to 5x more, barely score higher

The multi-agent hype train hit a benchmark this weekend — and the grid and the trucking regulators had quiet weeks of their own. Vals AI put agent teams head-to-head with single agents on its Vibe Code Bench, and the teams cost between 1.8 and 5.1 times more for almost no extra quality. The evals company ran GPT-6 Sol and Claude Opus 5.5 solo and in teams across 50 apps at two reasoning efforts; out of four comparisons, only one was statistically significant — Sol at medium effort, where the…

A 243-cell 'periodic table' puts every AI at zero on control

A 243-cell 'periodic table' puts every AI at zero on control

The most useful thing to come out of a paper this week is a map, not a result. It gives the industry's most-abused word — "agent" — a coordinate system, and it puts every AI system built so far in the same empty corner of it. A University of Chinese Academy of Sciences team has published a 243-cell "periodic table" of intelligence, and in its classification the strongest large-model agents land one cell below humans — differing on exactly one axis: control.