MedGPT beat 250 oncologists 88-12, scored by three AI judges

Future Doctor's MedGPT won 660 of 750 head-to-head comparisons against 250 senior Chinese physicians on 40 hard lung cancer cases, the company and its hospital partner announced on November 12, with the Xinhua write-up leading on the 88-to-12 score. The number that should decide how you read the rest of it is smaller and less quotable: 12.87 points on the study's own rubric, and the graders were three language models.

Both numbers are real, and both come from the same experiment. The difference between them is the difference between what the system did and what the headline claims.

What was actually run

The study, posted as a preprint in September and revised in October, works from a hand-picked corpus of 100 complex lung cancer cases. Forty became the reference set: internal hospital records rather than published cases, all grey-zone cases where guidelines do not settle the decision. Twenty senior specialists from six departments wrote 240 scoring rules before seeing any answer — six per case, including 40 safety red lines that fail a plan outright — according to the paper.

Then 250 physicians from 98 institutions across 27 provinces each answered three cases under three conditions: unaided, with a retrieval tool over the same evidence base, and with MedGPT's output in front of them. That is 2,250 strategies. MedGPT answered all 40.

The primary outcome is not a win rate. It is APAS, a 0-to-100 score for how much of each case's prespecified clinical content a plan expresses. MedGPT-assisted physicians scored 12.87 points above unaided ones (95% CI, 11.18 to 14.55), and 5.22 above the retrieval group (3.52 to 6.93), per the preprint. The unaided average was 53.40, retrieval 61.05, assisted 66.27 — the same figures VCBeat reported as a 24.1% lift.

The 88-to-12 comes from a separate, unadjusted comparison: MedGPT's single answer per case against each physician's unaided answer, 750 pairs, scored on the same rules. It averaged 85.29 against the physicians' 53.26. Its answers were about 40% shorter, which the paper treats as a point in its favour rather than a formatting quirk.

The rubric is the product

MedGPT's clinical mode is called MCE, for MedGPT Clinical Explorer, and the paper describes it plainly enough to be useful. For each case it builds candidate pathways, keeps a separate set of frontier or experimental options, and holds a standard-of-care reference alongside them. It then runs scenario simulation, critiques and repairs its own candidates, and emits a verification path — the tests and judgements still needed before anyone acts — before it decides to stop.

One archived run of a single case, dated 27 July 2026, retained 42 source-linked clinical evidence units. The company's own materials put the lung cancer library at more than 13,000 papers turned into 200,000-plus structured evidence units and 430,000 relations between them, 98.3% published since 2024, as 智东西 reported.

The ablation is the most interesting table in the paper. Complete MCE scores 91.91 on APAS. Strip the scaffolding and let the underlying MedGPT answer directly: 68.09. Remove only the multidisciplinary priors: 75.29. So roughly 24 of those points belong to the process wrapped around the model, not to the model. Future Doctor's own write-up makes the same argument, and it is the strongest thing in the package — a general model plugged into this pipeline would inherit the pipeline, and a rival general model without one is being compared unfairly.

That last point explains the model scoreboard everyone is quoting. MedGPT 96.22, GPT-5.4 71.38, Claude Opus 4.7 61.81, Gemini 3.1 Pro 54.62 — all "Direct", meaning bare model, no evidence layer, no candidate comparison. Read the paper and you find the 96.22 is simply the mean of three judges' scores for MCE (96.23, 94.71, 97.72). The comparison is orchestration versus no orchestration, and it is reported as if it were a model-quality comparison. It says something true and narrow: for this task shape, the wrapper matters more than the weights.

Why 88-12 is the weakest number in the room

Three things.

First, the judges. Every output was scored by a fixed panel — Gemini 3.1 Pro Preview, GPT-5.5 and Claude Opus 4.7 — producing 2,880 rule ratings for the main comparison, with scoring records generated on 31 August 2026 and no model versions archived. Two of those three judges are the same family as systems in the comparison table, and Claude Opus 4.7 scored its own output. Each judge graded all four answers per case, so this is a self-assessment in one cell of the grid.

Second, the instrument's sensitivity. The authors ran a controlled-text perturbation: they took 36 triplets of strategy text and produced variants. Reordering the sections moved APAS by 0.85 points (−0.44 to 2.31). Deleting designated clinical content tied to a specific rule moved it by −0.66 (−1.69 to 0.36), with 15 of 36 variants scoring higher after the deletion. A scorer that does not reliably register missing clinical content is a weak instrument for a 32-point gap.

Third, the graders and the humans do not agree. Two attending thoracic surgeons re-rated all 288 rules blind: 67.01% exact agreement with each other, but 39.93% between each surgeon and the AI panel's consensus, with a Gwet AC1 of 0.30. The paper says so directly — the AI consensus "was not treated as a clinical reference standard," and there was no third-rater adjudication and no human gold standard at rule level. So the 88-to-12 rests on machine judgements that match expert humans on about four rules in ten.

None of that makes the study worthless. It makes it a preprint. It has not been peer reviewed; Microsoft's comparable result — 83% of 304 New England Journal of Medicine case records against 36% for primary care doctors — went through Science, as we covered. Those are different currencies: diagnosis accuracy on published cases, versus completeness of a treatment plan on unpublished ones.

Who wins

Future Doctor, commercially, if the finding holds. The result it should sell is not 88-12 but the retrieval-versus-structure gap. Both the middle arm and the MedGPT arm had the same evidence base; organising it into a reviewable strategy added 5.22 points over simply handing physicians the papers. That is a defensible claim about product design, and it is the one a hospital buyer can act on.

The losers are the general-model vendors whose medical scores get quoted in procurement decks, and to a lesser degree radiology-first vendors, since this is a decision-support claim rather than a detection claim. China's push to put specialist-level judgement into county hospitals is the demand behind it: the pitch is that a physician who has seen fewer cases gets a comparison framework, not a diagnosis.

The honest loser is the patient. The study scores plans, not outcomes. Nobody in it was treated by MedGPT, and the paper says prospective work on workflow and patient outcomes is the next step. The 24.1% lift is a lift in rubric attainment.

What to watch

Three dates and one replication. Future Doctor says a multi-centre trial and a prospective real-world study are next, with regulatory registration completed before clinical deployment, per VCBeat. The preprint's own limits section names what was not evaluated: complete PICO fields, formal applicability boundaries, evidence grading, calibrated confidence — and the multidisciplinary tumour board itself sits outside the system's boundary, which is where the real decisions get made.

Watch whether a journal takes it, and whether the AI-judge instrument survives contact with reviewers who ask about that 0.30 agreement. And watch whether anyone outside the company reruns the ablation, because that table, not the 88, is the finding. Our earlier look at Alibaba's RADAR matching 23 of 26 radiologists ran into the same wall: a strong number, a hospital partner, and no independent rerun.

Is a plan that satisfies an expert-written rubric the same thing as a good clinical decision? Write to us via our contact page.