METR's time horizon plot: tasks double every four months, judgment lags

Two things landed Friday that pull in opposite directions: a chart saying AI capability is still compounding on schedule, and a White House that wants to tax the companies taking their data centers elsewhere.
METR's refreshed time-horizon report finds that the length of tasks frontier models can complete unaided is still doubling roughly every four months — and that the same models fall short on work where nobody can check the answer automatically.
The metric is easy to state and easy to misread. METR, a nonprofit that evaluates frontier systems, scores each task by how long it took a human to finish it, then measures the longest tasks a model completes at 50% reliability. The resulting plot became the most-shared and most-quoted chart in AI, and, as MIT Technology Review notes in its write-up of the update, the most widely misinterpreted. Its headline finding held: the task length a frontier model could handle doubled about every four months.
The new version expands the suite in the direction that matters for the industry's biggest bet. METR added tasks estimated to take humans far longer than anything in the original set, plus open-ended tasks where success is a judgment call rather than a pass or fail. That matters because of how these systems are trained: reinforcement learning can hammer a model into shape on any task where a grader can declare the answer right or wrong. Coding and math have graders. Choosing a research direction does not.
The results split cleanly along that line. In one new task, models were asked to design AI evaluations that METR could use in its own research, with METR employees scoring the output. Not one AI-designed evaluation was good enough for the staff to consider useful. In another, models had to score philosophical arguments — a proxy for judgment — and they trailed human performance there too.
There is genuine movement at the margin. The newest models, Opus 5.5 among them, now land in what METR calls "borderline above the bar" territory on the evaluation-design task: not yet usable, but close. A year ago, every model was in the "clearly below the bar" bucket. That is the interesting signal in the report — not a plateau, but a frontier that is arriving more slowly in the fuzzy domains than in the checkable ones.
Why it matters comes down to one number the industry has staked a decade of capital on: how fast AI can speed up AI research. Earlier this month, a heavily redacted METR report claimed AI might be accelerating progress at Anthropic by a factor of two, which is precisely the mechanism behind recursive self-improvement — systems that design their own successors. That claim landed in the middle of the largest IPO in history. The new report is the sober counterweight: if research taste is the bottleneck, the exponential curve describes a narrower slice of the work than the timeline debates assume. We made the adjacent argument this morning — that compute caps cannot stop the jump once the capability exists — in AGI vs ASI: why capping compute can't stop the jump.
The counter-case deserves its airing, because it is not fringe. Plenty of researchers argue that training on verifiable tasks generalizes: a model that gets good enough at the checkable work eventually handles the open-ended version, and the judgment gap closes as a byproduct rather than as a separate problem. If that is right, today's shortfall on evaluation design looks like the coding scores of 2023 — embarrassing until it isn't.
The methodological caveats cut the other way, and both sides should feel them. Most completion times in the new suite are estimates rather than measured, because paying humans to sit through tasks that take hundreds of hours is prohibitively expensive. The philosophical-argument task was largely scored by a single rater, so the models are being graded on agreement with one person's taste. And the judgment-heavy tasks are few, which is why METR's own technical staffer Matt Putz told MIT Technology Review that hard-to-verify work is "much harder to study, and especially to quantify," and that clean trends may never be establishable the way they are for easy-to-specify tasks. The honest read is that this report measures a boundary, not a forecast.
The most telling corroboration is that the labs agree. In a Wednesday blog post, OpenAI reported that its models do worse on the real-world tasks its own employees assign than on the easier, verifiable items in the original METR suite. A company selling capability is conceding the same gap its evaluators found — from a different direction, in the same week. UC Berkeley's Sayash Kapoor, who has published on models failing to reproduce human computer science research unaided, told MIT Technology Review that "AI evaluations need to get more creative." That is where this goes next: better judgment benchmarks, not another doubling announcement.
Trump said Friday that the United States will not allow its companies to build data centers abroad and will put a "tremendous tax" on any that do — a threat he floated in public for the first time, before his own staff had heard it.
The setting was a White House press conference a day after a trilateral summit with Xi Jinping and Vladimir Putin in China, where Trump said Xi had congratulated him on US progress in AI, according to CNBC. "I haven't even said that to my people," he told the room. "But I'm telling them now." Alphabet, Amazon, Microsoft, Meta and Oracle already run data centers outside the US, largely to serve nearby clients and satisfy data-residency rules, so the threat lands on existing capacity, not just future projects.
The bind is the story. CNBC's October survey found 64% of registered voters oppose an AI data center within 10 miles of home, and only 13% would welcome one, which is why states and cities keep imposing moratoriums. The administration is now threatening the exit route that local opposition creates — with no tax instrument, no bill and no agency named, and a Democratic-majority Congress being sworn in within weeks, as The Verge notes.
Should a company that builds its data centers overseas to escape a local moratorium pay a federal penalty for it? Write to us via our contact page.







