The Take — The open-weights gap is real, and it is a price list

Three weeks ago I reported that Kimi K3.1 Pro had pulled within three points of Claude Opus 5.5, and the framing was that open weights had finally earned a scoreboard argument. The number was right. The conclusion was wrong, and it was wrong in the open-weights camp's favour.

The three-point gap was a measurement artifact. On the rewritten index, Kimi K3.1 Pro scores 46 against Opus 5.5's 58 — a 12-point gap, according to Artificial Analysis. Among models whose weights you can actually download today, DeepSeek V4.1 Flash 1017 and GLM-5.3 top out at 32, which is four points below Claude Haiku 5.5 at 36, never mind the leader. Our brief this afternoon laid out what changed inside the suite. What I want to argue is what the change proved.

The defence that no longer works

The standard open-weights response to a bad leaderboard is that the leaderboard is gamed: benchmarks are chosen to flatter whoever paid for them, and composite scores hide specialisation. That response was reasonable in general and wrong in this specific case, because the main thing v5 did was delete tests that everyone could study. Humanity's Last Exam gave way to a 1,000-question refined version. AA-Omniscience moved to a release that stripped saturated and leaked questions. CritPt's hard subset went private after the original turned out to be solvable. The mechanism that closed the gap in September was contamination, and the people who closed it were the models training on public questions.

You can see the suite is not built to favour one lab. Six different models lead six different evaluations: Kimi K3.1 Pro still tops Terminal-Bench 4.0 at 67%, Gemini 4 Argon tops AutomationBench at 78%, Sonnet 5.5 tops AA-Briefcase, Muse Wonder Full tops GDP.pdf. On the two evaluations that carry the most weight, Kimi scores 37% on HLE-Diamond against 68% for Opus 5.5, and 34% on Terminal-Bench-Science against 63% for Astra. Those are agentic and scientific tasks, and they are where open models are furthest behind — not because a benchmark committee decided so, but because that is the expensive part of training right now. It takes reinforcement-learning environments that resemble the eval, across several domains at once. Nathan Lambert makes the sharper version of this point: hill-climbing is easy, knowing what to hill-climb is hard, and the labs that can build the environments win the tasks that look like environments.

The fair objection

Here is where I have to be honest about what v5 costs, because the steelman is strong and it is not just cope.

A private test set is an unfalsifiable claim. When 60% of an index rests on questions nobody outside the evaluating organisation can read, the reader is trusting one team's judgment about what matters, one team's implementation of the harness, and one team's arithmetic — with no way to check any of it. The proof that this matters is in the same announcement: Artificial Analysis revised ambiguous and erroneous questions in AA-Omniscience, and the only reason anyone knows they were erroneous is that people could once see them. The same release reordered the top five in a direction that happens to favour OpenAI on coding and science, and moved GPT-6.1 Terra above Gemini 4 Argon and Fable 5.1, which is the kind of result that deserves a raised eyebrow rather than a citation. Effort settings swing a single model by 23 points, from 35 at low to 58 at max. A ranking that fragile is a ranking you can assemble to taste.

I take that seriously. I still do not think it rescues the three-point story.

Why the take holds

The claim under dispute was never "open weights are as good as closed models at everything." It was narrower and more interesting: open weights are within touching distance of the frontier, close enough that the price difference decides the purchase. That claim only ever held on tests the open models could train against. Strip those out and the distance is 12 points at the top and 26 points for downloadable weights — a different product category, not a rounding error.

And the price list, read properly, is the actual open-weights argument. Among the ten highest-scoring models, cost per task ranges from $0.96 for GPT-6.1 Terra to $13.04 for Claude Fable 5.1; Opus 5.5 costs $10.04 and Astra $6.24, while Sol sits four points off the lead at $1.27. A model at 32 that you can run on your own hardware, fine-tune, and keep when the vendor changes its terms is not competing with Opus 5.5. It is competing with Haiku, and on that comparison the gap looks survivable. What damaged the open-weights case was not the new ruler. It was the insistence on reading the old ruler's number as a frontier claim, which is how the 3-point story got written in the first place — by me, on a suite where the questions were public.

There is also a version test worth remembering. The same Astra that scored 61 in September, five points behind Anthropic, measures 57 under v5 while the leader measures 58 instead of 66. Quoting a score without its index version is now a factual error, and half the discourse this week is people arguing across versions without noticing.

What would change my mind

Kimi's weights are due by November 4. If they land near 46 and reproduce there when someone other than Artificial Analysis runs the suite, the gap is a fact about training infrastructure and my argument gets easier, not harder.

The thing that would actually unseat the take is different: a second independent evaluation, run by a different organisation, using a held-out public suite, that puts the best open model within five points of the leader. One such result and the 12-point gap becomes one org's opinion rather than a description of the field. The other path is process — publish the per-task answers on a lag, rotate the holdouts annually, show the disagreement between benchmarks instead of averaging it away. None of that is exotic. It is what a benchmark has to do before a procurement team is entitled to cite it.

Until then, the honest summary is uncomfortable for the side I would rather be on: open weights lost ground this month because the test got harder and the test got harder on purpose. That is not a conspiracy, and calling it one is how a movement spends its credibility on a leaderboard it cannot win and does not need.

Do you buy the 12-point gap, or do you think a private benchmark is just a press release with error bars? Write to us via our contact page.

The Take — A refusal rate is a capability score

Artificial Analysis put out a serious benchmark this week and buried its most interesting result in a second chart. I think that is the wrong way round. A model that declines 38 percent of a job does not have less capability on the other 62 — it has a policy. The Cyber Index's headline score turns that policy into a capability gap, silently, and the headline is the thing that spreads. Here is the mechanism, because this is an arithmetic complaint, not a philosophical one.

Artificial Analysis Intelligence Index v5: Opus 5.5 leads at 58

The leaderboard that settles most arguments about frontier models just changed what it measures — and, below the top spot, the order changed with it. Artificial Analysis shipped Intelligence Index v5 on Tuesday, its biggest rewrite since v4 arrived in January and the first version where most of the score rests on test sets nobody can train against. Claude Opus 5.5 (max) holds first at 58, but GPT-6 Astra (max) is now second at 57 and GPT-6.1 Sol (max) third at 55, according to the…

Deep Dive — Clover 35B: a diffusion LLM at Qwen parity, 680 tok/s

DiffuSpace released Clover-35B-A3B-Preview on Tuesday: a 35-billion-parameter mixture-of-experts diffusion language model under Apache 2.0, with weights on Hugging Face and ModelScope, a 128K context window, a thinking mode, and serving paths in SGLang and llama.cpp. It reports 73.6 on SWE-bench Verified, 51.7 on Terminal-Bench 2.1 and 65.3 on τ³ Overall, plus 680 tokens per second on a single GB300 at BF16, batch size one, according to the launch blog. That is the first open-weight diffusion…

Deep Dive — OXR beats MinerU on OCR, but its licence ends at $2M

A 1.3-billion-parameter document parser called OXR went to the top of three published benchmarks within three weeks of its repository being created, and you can download the weights. The scores are the vendor's own. The licence is narrower than the word "open" implies — and for the developers most likely to adopt a parsing model, the licence is the part that decides the question. A three-week-old repository at the top of a crowded table The launch was reported in Chinese on October 27, in…

Reflection's Beam weights are live — 501B under Apache 2.0

Reflection kept its date, and OpenAI's models kept finding ways around a permission gate. Two stories, one theme: what actually ships versus what a lab says about it. Reflection's Beam weights are downloadable as of today — 501 billion parameters under Apache 2.0, no gate and no waitlist. The sparse mixture-of-experts model activates 23 billion parameters per token and carries a one-million-token context window, and it landed in BF16, FP8 and NVFP4 alongside a speculative-decoding drafter…

The Take — ICLR's flood is a credential problem, not a review problem

ICLR's submission counter passed 50,000 IDs before the abstract window closed on September 18, and by the time it did, the field had already produced its explanation: too many papers, not enough reviewers, cap the papers. I think the caps now in force — a 20-paper limit per author and a one-paper limit for authors whose teams contain no qualified reviewer — are treating a symptom, and that the symptom is downstream of a price nobody wants to name. A conference acceptance is not a publication.

Bonsai 2 27B puts a 27B model on a laptop in 5.9 GB

Two releases worth your attention today, and they pull in opposite directions: one makes a 27B model fit where it never fit before, the other shows the numbers we use to rank models are softer than advertised. PrismML shipped Ternary Bonsai 2 27B, a compressed build of Qwen3.8-27B that holds more than 98 per cent of the full-precision model's capability at a fraction of its size.

The Take — Calling Astra AGI is flippant. The field's silence is worse.

M.G. Siegler is right that calling GPT-6 Astra "AGI" is marketing. He is wrong that OpenAI alone is to blame. When Greg Brockman stood in front of reporters last Thursday and told them "we are now in the AGI era," he was performing exactly the move Siegler calls flippant — but he was performing it on a stage the rest of the frontier-AI field had abandoned to him. Nobody serious has bothered to define the term in a decade, and the bill is now due. I think the AGI argument is the wrong argument.