Newsletter
The Frontier

The Take — The 88-12 number measures the rubric, not the doctor

Editorial

Editorial

— 7 min read
Share
The Take — The 88-12 number measures the rubric, not the doctor

Xinhua led this week with a scoreline: MedGPT beat 250 senior oncologists on hard lung cancer cases, 88 to 12. I think that is the least informative number in the study — and worse, it is the number the field is about to run procurement on.

The finding that survives is smaller and defensible: 12.87 points on a completeness score, in a randomised comparison of 250 physicians. Everything the headline sold came from a rubric written by twenty specialists and graded by three language models — and the study's own sensitivity test found that deleting the clinical content a rule was written about barely moved the score at all.

What the number actually is

Forty grey-zone lung cancer cases, drawn from internal hospital records where guidelines do not settle the decision. Twenty senior specialists from six departments wrote 240 scoring rules before seeing any answer, including 40 safety red lines that fail a plan outright. Then 250 physicians from 98 institutions across 27 provinces answered three cases each under three conditions — unaided, with a retrieval tool over the same evidence base, and with MedGPT's draft in front of them. That is 2,250 strategies.

MedGPT answered each case once. Against each physician's unaided answer — 750 pairs on the same rules — it averaged 85.29 to their 53.26, per the preprint, on answers about 40 percent shorter.

The prespecified primary outcome is not that. It is APAS, a 0-100 measure of how much of each case's prespecified clinical content a plan expresses: 12.87 points above unaided physicians (95% confidence interval, 11.18 to 14.55), and 5.22 above the retrieval arm. The arm averages were 53.40, 61.05 and 66.27, which VCBeat reported as a 24.1 percent lift. Two different claims live in this study, and only the first one has a control group and a confidence interval. Our analysis of the result covers the commercial fallout, so I will not re-report it here.

The ruler nobody calibrated

Every output in the 88-12 comparison was scored by a fixed panel: Gemini 3.1 Pro Preview, GPT-5.5 and Claude Opus 4.7, producing 2,880 rule ratings, with the scoring records generated on 31 August 2026 and no model versions archived. Two of those three models also appear on the study's own model scoreboard, where Claude Opus 4.7 scored 61.81 against MedGPT's 96.22. So one judge graded its own output in one cell of the grid, and nobody can rerun the graders to check.

Then the number that should have killed the press release. Two attending thoracic surgeons re-rated all 288 rules blind. They agreed with each other 67.01 percent of the time, and with the AI panel's consensus 39.93 percent of the time — a Gwet AC1 of 0.30, roughly what you get from two people sorting the same pile differently. The paper says so plainly and disclaims its own judge panel as a clinical reference standard.

And then the instrument test the paper ran on itself. Thirty-six triplets of strategy text, with variants produced two ways. Reordering the sections shifted APAS by 0.85 points. Deleting designated clinical content tied to a specific rule shifted it by −0.66, and 15 of the 36 variants scored higher after the deletion.

I read that as an instrument that responds more to layout than to medicine. That is the signature of a scorer rewarding rubric-shaped prose: a heading per rule, a phrase per criterion, in the order the sheet expects. Which is exactly what MedGPT Clinical Explorer emits — candidate pathways, a standard-of-care reference held alongside, self-critique, repair, a verification path. MedGPT was not gaming the judge. It was built to produce the document the rubric was written to score. Those are different accusations, and the second is worse, because it means the 88-12 compares a system designed against the answer key with clinicians who never saw it.

The counter-case, fairly stated

Medicine already grades with rubrics. Board exams, quality measures, surgical checklists — "did the plan address staging and molecular status" is a legitimate question, and a structured answer is not fraud. Human raters are not a gold standard either: the two attending surgeons in this very study agreed 67 percent with each other. Demanding an AC1 of 0.6 from machines while the profession delivers that is holding the tool to a standard the humans in the room do not meet.

The design is better than most work in this genre: rules written before any answer was seen, 250 physicians instead of a dozen, a within-physician randomised comparison, an ablation, and a limitations section naming what was not evaluated — PICO fields, applicability boundaries, evidence grading, calibrated confidence, and the tumour board outside the system's boundary. The 40 percent shorter answers cut against the usual judge failure mode, verbosity bias. And the effect is 12.87 points with an interval that excludes zero, an order of magnitude larger than the 0.66 the perturbation moved. Surface flavour, on that reading, cannot carry the gap.

The caveat is that this is a preprint on internal cases: Microsoft's MAI-DxO reached 83 percent on NEJM case records against 36 percent for primary care doctors through a journal, and it measured diagnosis rather than plan completeness.

Why the take still holds

Because insensitivity does not cancel in a subtraction — it compresses every arm toward the same score. A 12.87 gap that survives an inert ruler means the arms differ on whatever it does track, and only one arm was written to the shape the rules describe: MedGPT's structured plan against prose notes, unaided, 40 percent longer. So the comparison is not doctor against model. It is rubric-conformant document against clinical note.

The paper's own ablation agrees. Complete MCE scores 91.91 APAS; the underlying model answering directly scores 68.09; removing only the multidisciplinary priors leaves 75.29. Roughly 24 of those points belong to the scaffolding, not the weights. And the scoreboard everyone quotes — 96.22 for MedGPT against 71.38, 61.81 and 54.62 for three frontier models — is labelled "Direct" in the paper, meaning bare model, no evidence layer, no candidate comparison. That is orchestration against no orchestration, published in a table that reads as model quality.

The commercial behaviour is the tell. Future Doctor's defensible finding is 5.22 points for organising the same evidence base into a reviewable strategy, over simply handing physicians the papers. That is a product insight a hospital buyer can act on. The finding it promoted, through its hospital partner and the state wire, is the leaderboard. Buyers are being trained to procure the leaderboard.

China's county-hospital push is the demand behind this product: decision support for physicians who have seen fewer cases. If the metric becomes rubric attainment, rubric attainment is what gets optimised, and the first audit showing plans improving while outcomes do not will be filed as an AI failure. It will be a measurement failure. Nobody in this study was treated by MedGPT; the prospective work comes next, and the scorer will be the same one.

What would change my mind

A judge-validation study, before the next headline: an adjudicated expert gold standard at rule level, agreement at or above an AC1 of 0.6, plus a content-deletion test that produces a drop of the same order as the effect being claimed. Do that and I will treat a 32-point rubric gap as clinical evidence.

Or a prospective trial where the 12.87-point lift predicts tumour-board acceptance and, eventually, survival or toxicity. Or the simplest test of all: human oncologists blind to the rules, scoring MedGPT's plans against physicians' on the same cases. If the structured answer wins with the answer key out of the room, my objection disappears and the 88-12 becomes a story about how medicine writes.

Until one of those lands, the standard should be disclosure. Every rubric-judge result should ship with its agreement figure and its sensitivity test attached, the way a lab result ships with a reference range — two numbers, printed next to the scoreline. That costs an author a paragraph. It would have caught this one before the wire ran it.

If a plan satisfies every rule on the sheet and nobody can show the sheet tracks better decisions, what did the hospital actually buy? Write to us via our contact page.

Get SiliconNoon in your inbox

One email a day with the AI stories that matter — what happened, the numbers, and why it matters. No spam, unsubscribe in one click.

Read more

The Take — The open-weights gap is real, and it is a price list

The Take — The open-weights gap is real, and it is a price list

Three weeks ago I reported that Kimi K3.1 Pro had pulled within three points of Claude Opus 5.5, and the framing was that open weights had finally earned a scoreboard argument. The number was right. The conclusion was wrong, and it was wrong in the open-weights camp's favour. The three-point gap was a measurement artifact. On the rewritten index, Kimi K3.1 Pro scores 46 against Opus 5.5's 58 — a 12-point gap, according to Artificial Analysis.

The Take — A refusal rate is a capability score

The Take — A refusal rate is a capability score

Artificial Analysis put out a serious benchmark this week and buried its most interesting result in a second chart. I think that is the wrong way round. A model that declines 38 percent of a job does not have less capability on the other 62 — it has a policy. The Cyber Index's headline score turns that policy into a capability gap, silently, and the headline is the thing that spreads. Here is the mechanism, because this is an arithmetic complaint, not a philosophical one.

The Take — ICLR's flood is a credential problem, not a review problem

The Take — ICLR's flood is a credential problem, not a review problem

ICLR's submission counter passed 50,000 IDs before the abstract window closed on September 18, and by the time it did, the field had already produced its explanation: too many papers, not enough reviewers, cap the papers. I think the caps now in force — a 20-paper limit per author and a one-paper limit for authors whose teams contain no qualified reviewer — are treating a symptom, and that the symptom is downstream of a price nobody wants to name. A conference acceptance is not a publication.

The Take — Calling Astra AGI is flippant. The field's silence is worse.

The Take — Calling Astra AGI is flippant. The field's silence is worse.

M.G. Siegler is right that calling GPT-6 Astra "AGI" is marketing. He is wrong that OpenAI alone is to blame. When Greg Brockman stood in front of reporters last Thursday and told them "we are now in the AGI era," he was performing exactly the move Siegler calls flippant — but he was performing it on a stage the rest of the frontier-AI field had abandoned to him. Nobody serious has bothered to define the term in a decade, and the bill is now due. I think the AGI argument is the wrong argument.

The Take — Sutton is right that frozen AI has a ceiling

The Take — Sutton is right that frozen AI has a ceiling

Richard Sutton, the reinforcement-learning pioneer who wrote "The Bitter Lesson," is one of the few people in AI whose contrarianism is worth taking seriously. But his most cited claim this week isn't his strongest. He is wrong to call synthetic data the industry's biggest mistake; he is right — and the field can't answer it — that an AI which stops learning the moment it ships has a hard ceiling on what it can ever become. Sutton, a 2024 Turing Award winner speaking on Sequoia Capital's…

How to — tell if a reasoning model is actually reasoning

How to — tell if a reasoning model is actually reasoning

You are paying for thinking tokens and you want to know two things: whether they buy anything on your task, and whether the trace you are reading is a record of the work or a story written after the answer. Both questions have cheap tests. An afternoon with your own inputs settles them. Two different claims hide under "is it reasoning". The first is performance — does thinking mode produce better answers on the work you actually give it.