AI 101 — What is an AI doctor?

An AI doctor is software that does part of a physician's job — listening to a visit, reading a scan, narrowing a diagnosis — while a licensed human stays responsible for the decision.

That definition is deliberately unglamorous, because the interesting question was never whether a model can score well on a medical exam. It is which part of the job the software is doing, and who signs the note afterwards.

Why it matters right now

Three things landed within a month of each other, and together they explain the whole argument.

In October, Microsoft's researchers published a study in Science showing their diagnostic system, MAI-DxO, reached the right answer in 83.2% of 304 complex cases drawn from the New England Journal of Medicine, against 36.1% for generalist physicians using reference tools — and 22.3% for the same physicians working without them. Those doctors worked alone, on cases selected because they were hard, and the system simulated a panel that asked an average of 25.5 questions before ordering its first test, against 5.6 for the humans. That is a peer-reviewed result, and it is not a small one.

Four days ago, a Chinese company called Future Doctor said its MedGPT beat 250 senior physicians 88 to 12 on 40 hard lung cancer cases. As our analysis of the preprint found, the 88-12 figure came from three language models grading the answers, and nobody in the study was actually treated.

And underneath both: the US Food and Drug Administration has now authorized more than 1,600 AI-enabled medical devices for marketing, about three-quarters of them in radiology. AI medicine is not a forecast. It is a product category with a regulatory filing history.

The mental model: four jobs, four risk levels

"AI doctor" is an umbrella. The thing under it is usually one of four jobs, and they are nowhere near equivalent.

The scribe listens to the visit and writes the note. It is the least exciting job and the most widely deployed, because the failure mode is a typo a clinician catches while reading.

The detector looks at an image — a chest X-ray, a retinal scan, a pathology slide — and flags what a human might miss. Narrow, measurable, and the reason radiology dominates the FDA list. The hard part is not the model; it is what happens to a clinic's error rate when a radiologist starts trusting the flag.

The triager asks the patient questions and decides how urgent this is. It touches the patient directly, so the bar jumps.

The diagnostician builds a differential, orders tests, and argues with itself about what to do next. That is MAI-DxO, MedGPT, and Google's AMIE — the most impressive demos and the thinnest evidence about what happens to patients, because a correct diagnosis on a case record is not a health outcome.

Risk rises as you read down that list. So does the distance between the demo and the proof.

The analogy: the resident who read everything

Think of the best medical resident you have ever met on their first week. They have read every guideline and every paper, they never get tired at 4 a.m., they will happily recite the differential for a rare vasculitis — and they have never touched a patient, cannot feel a lump, and have no idea when they are confidently wrong.

That resident is genuinely useful, which is why hospitals employ them. But no hospital lets a first-week resident sign a discharge summary alone. The attending reads it, agrees or overrules, and carries the liability. An AI doctor sits in exactly that chair: the value is real and the signature is somebody else's.

This is why the most convincing real-world result so far is also the most modest. Google Research ran AMIE on 100 patients at Beth Israel Deaconess Medical Center before their primary care appointments — a text chat, with a physician watching live and able to stop the session. Google reports that no safety stops were needed, that the eventual diagnosis appeared somewhere in AMIE's list in 90% of cases and was its top pick 56% of the time, and that its plans were rated about as good as the doctors' — except that the doctors' plans were more practical and cheaper, which is what happens when you can see the patient and read their chart.

Three things people get wrong

"The AI beat the doctors, so it is better at medicine." What got scored in most of these studies is a written plan or a diagnosis on a case record, often judged by other models — a measurement problem worth understanding before you quote the number, and LLM-as-a-judge explains why. The comparator matters just as much: Microsoft's 36% is generalists working alone on deliberately brutal cases, a fair test of a tool and an unfair description of your doctor.

"FDA clearance means it has been shown to help patients." Most AI devices go through the 510(k) pathway, which asks whether a device is substantially equivalent to something already on the market, not whether it improves outcomes. A PLOS Digital Health census of all 1,357 cleared AI devices through December 2025 found that 34 were linked to a registered prospective trial and only three had been evaluated on patient-centered outcomes like mortality or readmissions. That gap is the single most important number in this field, and it is the one nobody puts in the press release.

"A chatbot will just diagnose me." The systems that have passed real-world tests were supervised history-taking, not autonomous diagnosis. Licensing, liability, and the physical exam still sit with a clinician. What will change first is the paperwork and the second opinion, not the person in the room.

Where to learn more

The FDA's device list is public and searchable — the fastest way to see what has actually been authorized. And keep an eye on which claims arrive with a journal and which arrive with a press release; that distinction has predicted durability better than any benchmark.

Related reading: AI 101 — What is an AI benchmark? for why a case-record score is not a clinical trial, and AI 101 — What is LLM-as-a-judge? for the grading problem underneath most of these headlines.

Would you trust an AI to take your history before the appointment, if a doctor reviewed it? Write to us via our contact page.