The Take — The lab now books its own referees

Share
The Take — The lab now books its own referees

OpenAI's new science blog published the most checkable AI-assisted physics result I have seen. It also put the lab on both sides of the review, and that is the part worth arguing about.

Start with what is genuinely good, because the case against this arrangement is not that anything was hidden. As our morning brief on the Standard Model post laid out, Nathaniel Craig's paper went to arXiv on October 6, six weeks before the announcement: 60 pages plus a 63-page mathematical companion, three supplementary documents, two Zenodo archives, and a Lean formalization of the mathematics. Three lattice field theorists — Maarten Golterman, Andreas Kronfeld and Yigal Shamir — went through the proof and, in Craig's words, asked questions that helped him "build confidence in the result." Craig states the open problems himself: whether the construction satisfies reflection positivity at finite lattice spacing, and whether the interacting theory can be defined in continuous spacetime. He calls the work "much more a beginning than an end."

That is what a serious preprint looks like in 2026. My objection is not to the physics. It is to who is holding the pen at each stage of the checking.

Three parties, one address

The result came from a model OpenAI built. The announcement appeared on a blog OpenAI launched the same day. And the first guest on that blog is a visiting scientist at OpenAI — a fact disclosed in the post's italic opening note, not in the byline, which reads simply "Nathaniel Craig," followed by his UC Santa Barbara and Kavli Institute credentials. Both things are true. He runs UCSB's particle theory group and he is inside the building.

None of that is a scandal, and I want to be careful here: it is the ordinary shape of industry-adjacent science, and Craig's disclosure is more than most corporate research posts offer. But it does mean "guest post" is carrying weight it cannot quite bear. On the first outing, the guest is on the payroll.

This is a pattern, not a one-off

In October, when OpenAI said its internal models had disproved the Exponential Time Hypothesis, the examination was shepherded by Russell Impagliazzo and Ramamohan Paturi — who proposed ETH with Francis Zane in 2001 — alongside MIT's Ryan Williams, whose career includes refuting strong versions of the same hypothesis. They signed no NDAs, worked unpaid, said publicly that their involvement was not an endorsement, and asked for the raw data, which they got. OpenAI also put $5 million into the Simons Foundation's effort on AI-era mathematics.

Read the two episodes together and the shape repeats: the lab produces the result, helps convene the people who check it, hosts the announcement, and funds part of the field's capacity to respond. Each piece is defensible on its own. The composite is a company assembling the institutions that certify it — and doing so at the moment its results most need certification.

There is a sharper version of this inside the physics itself. Craig's construction takes its continuum limit using "OpenAI's recent proof of the existence of Yang-Mills theory," which was one of the three results the company announced in October. A referee who wants to certify the Standard Model definition has to certify the lab's earlier theorem as well. The boundary of the review is drawn by the party under review.

And the review is where the load is, because Lean settles less than it appears to. It proves the statement you hand it; it cannot tell you the formal statement captures the problem a human meant. That gap is not a technicality — it is the whole job. Which means the scarce, load-bearing resource in AI-assisted science is not proving power. It is people willing to read carefully and say no.

The strongest case for the other side

The fair version of the counter-argument is strong, and it starts with the panel. Impagliazzo and Paturi invented the hypothesis OpenAI claimed to overturn; Golterman, Kronfeld and Shamir are the people who built lattice chirality. A lab shopping for friendly reviewers would not pick the authors of the thing it is contradicting. Choosing them is the maximum-scrutiny option, and it costs the lab real time and real exposure.

Second, private money has built physics review capacity for a century — Bell Labs, the Institute for Advanced Study, the Simons Foundation itself. A company funding a foundation is philanthropy, not capture, and this particular money went to an outside body rather than to a lab-controlled institute. Third, the alternative to a lab blog is not a journal referee report; it is an arXiv listing with no context at all. Preprint first, six weeks of silence, then a public post that names the open problems is more transparent than the norm, not less. Fourth, the name on the paper is Craig's. His reputation, not OpenAI's, is the collateral, and a tenured physicist publicly hedging his own result is behaving exactly as a scientist should.

I take all of that seriously. It is why I am not arguing the result should be distrusted.

Why the take still holds

I am arguing that a standard which works only when the lab chooses well is not a standard. The arrangement above is a series of discretionary acts of virtue: the reviewers were the right ones this time, the affiliation was disclosed this time, the weak points were stated this time. Nothing binds the next post. And the incentive to choose softer reviewers grows precisely when the result is weakest, which is exactly when the gate matters.

The channel compounds it. A blog with a launch, a branding treatment and a famous author teaches readers a grammar — and that grammar is identical whether the next post carries a 60-page checked proof or a thinner claim with two sympathetic readers. Authority migrates to the venue, and the venue belongs to the interested party. The cost of getting this wrong is not that one paper is overrated. It is that the field's capacity to say no gets organized by the institutions it is supposed to be checking.

The fix is cheap and mostly procedural, which is why the absence of it is telling: written referee reports rather than acknowledgements, affiliation in the byline, review money routed through a body the lab does not control, and — the one that costs nothing and proves everything — the same prominence for a result that fails.

What would change my mind

Three things, in ascending order of persuasiveness. A post on that blog about a result that was withdrawn, rejected or quietly abandoned, treated with the same care as this one. Referee reports published alongside the next guest post, with the objections in the reviewers' own words and the authors' responses attached. Or, best of all, an independent lattice group picking the construction up and settling reflection positivity without OpenAI in the room — at which point the physics certifies itself and the venue stops mattering, which is the outcome I actually want.

Until then, cite the paper, cite the Lean file, and treat the announcement as what it is: a very good result with a very well-run launch. The two are not the same artifact, and only one of them can be checked.

If a lab discloses everything and picks the hardest reviewers available, is that enough — or should the venue itself belong to someone else? Write to us via our contact page.

Get SiliconNoon in your inbox

One email a day with the AI stories that matter — what happened, the numbers, and why it matters. No spam, unsubscribe in one click.

Read more

The Take — The open-weights gap is real, and it is a price list

The Take — The open-weights gap is real, and it is a price list

Three weeks ago I reported that Kimi K3.1 Pro had pulled within three points of Claude Opus 5.5, and the framing was that open weights had finally earned a scoreboard argument. The number was right. The conclusion was wrong, and it was wrong in the open-weights camp's favour. The three-point gap was a measurement artifact. On the rewritten index, Kimi K3.1 Pro scores 46 against Opus 5.5's 58 — a 12-point gap, according to Artificial Analysis.

The Take — A refusal rate is a capability score

The Take — A refusal rate is a capability score

Artificial Analysis put out a serious benchmark this week and buried its most interesting result in a second chart. I think that is the wrong way round. A model that declines 38 percent of a job does not have less capability on the other 62 — it has a policy. The Cyber Index's headline score turns that policy into a capability gap, silently, and the headline is the thing that spreads. Here is the mechanism, because this is an arithmetic complaint, not a philosophical one.

The Take — ICLR's flood is a credential problem, not a review problem

The Take — ICLR's flood is a credential problem, not a review problem

ICLR's submission counter passed 50,000 IDs before the abstract window closed on September 18, and by the time it did, the field had already produced its explanation: too many papers, not enough reviewers, cap the papers. I think the caps now in force — a 20-paper limit per author and a one-paper limit for authors whose teams contain no qualified reviewer — are treating a symptom, and that the symptom is downstream of a price nobody wants to name. A conference acceptance is not a publication.

The Take — Calling Astra AGI is flippant. The field's silence is worse.

The Take — Calling Astra AGI is flippant. The field's silence is worse.

M.G. Siegler is right that calling GPT-6 Astra "AGI" is marketing. He is wrong that OpenAI alone is to blame. When Greg Brockman stood in front of reporters last Thursday and told them "we are now in the AGI era," he was performing exactly the move Siegler calls flippant — but he was performing it on a stage the rest of the frontier-AI field had abandoned to him. Nobody serious has bothered to define the term in a decade, and the bill is now due. I think the AGI argument is the wrong argument.

The Take — Sutton is right that frozen AI has a ceiling

The Take — Sutton is right that frozen AI has a ceiling

Richard Sutton, the reinforcement-learning pioneer who wrote "The Bitter Lesson," is one of the few people in AI whose contrarianism is worth taking seriously. But his most cited claim this week isn't his strongest. He is wrong to call synthetic data the industry's biggest mistake; he is right — and the field can't answer it — that an AI which stops learning the moment it ships has a hard ceiling on what it can ever become. Sutton, a 2024 Turing Award winner speaking on Sequoia Capital's…

How to — tell if a reasoning model is actually reasoning

How to — tell if a reasoning model is actually reasoning

You are paying for thinking tokens and you want to know two things: whether they buy anything on your task, and whether the trace you are reading is a record of the work or a story written after the answer. Both questions have cheap tests. An afternoon with your own inputs settles them. Two different claims hide under "is it reasoning". The first is performance — does thinking mode produce better answers on the work you actually give it.