Do AI models outperform physicians on health tasks?

HealthBench, an open-source benchmark built from 5,000 multi-turn health conversations, used physician-written rubrics to grade both AI models and physicians on the same conversations. Developed with 262 physicians who have practiced in 60 countries and across 26 specialties, the benchmark scores responses against 48,562 unique rubric criteria worth between −10 and 10 points each, graded by a model-based grader the authors validated against physician judgment. Measured across roughly two years of model releases, HealthBench scores show "steady initial progress (compare GPT-3.5 Turbo's 16% to GPT-4o's 32%) and more rapid recent improvements (o3 scores 60%)." The authors also report that "GPT-4.1 nano outperforms GPT-4o and is 25 times cheaper." Physicians were asked to produce responses both with and without model assistance, and the paper's human baseline finds "recent models produce higher quality responses than physicians unless physicians are assisted by the same models."

The comparison works because physicians' responses were graded on exactly the same rubric criteria as model responses: criteria are written per-conversation by a physician, cover facts to include, communication behaviors, and context-seeking, and are summed and normalized into a 0–1 score. A subset of 34 "consensus criteria," assigned to a conversation only when two or more reviewing physicians agree it applies, lets the authors isolate narrow behaviors such as prompt escalation of emergencies. That shared rubric is what makes the physician-versus-model and physician-plus-model comparisons commensurable: unassisted physicians and the frontier models answered under identical scoring rules, and the paper reports physicians closing the gap specifically when assisted by the same model they are being compared against.

This sits in tension with Why do LLMs fail when users interact with them?, where lay users paired with an LLM did no better than unaided controls on diagnostic identification — HealthBench's physicians, by contrast, match or exceed the model's own score once assisted by it, suggesting the benefit of AI assistance on health tasks may hinge on who is doing the assisting, not only on model capability. It also extends Can language models reason better than physicians at diagnosis?: both report models outscoring physicians working alone, but HealthBench specifies the condition under which that gap closes rather than only flagging superhuman performance as awaiting trial. And it runs alongside Can conversational AI safely take patient histories without supervision?, a reminder that rubric-graded response quality and workflow-level practicality are different measures that can diverge.

The excerpt does not give the actual assisted-physician scores or a statistical comparison against model-alone scores, so the size of the "unless assisted" effect is unclear from this passage alone. The authors caution that physician agreement on consensus-criteria grading ranges only from 55% to 75%, that criteria written by individual physicians were not independently validated, and that HealthBench "does not measure health outcomes of specific workflows, which depend not only on the quality of model responses but also, critically, implementation." The implication is that rising benchmark scores and a closing physician-assistance gap are evidence of improving response quality under a rubric — not yet evidence that deploying a model at this score level changes real-world health outcomes.