Harvard study: AI outperforms emergency physicians at diagnosis

A Harvard study found that large language models can diagnose acute conditions in emergency departments more accurately than experienced physicians. The researchers ran large-scale tests of LLMs across a range of medical contexts, including real cases from emergency rooms and archived medical records.

How the AI was tested

The researchers gave large language models real clinical cases from emergency departments, using exactly the data doctors see when a patient arrives: a description of symptoms, prior medical history, and results from initial examinations and lab tests. The models analyzed this information and produced a free-form presumptive diagnosis, much as a physician would in a clinical note.

At least one of the tested models reached the correct diagnosis significantly more often than two emergency physicians working independently, who analyzed exactly the same clinical data without the help of any tools. Many experts found the result surprising: until now, it was unclear whether an LLM could outperform experienced physicians at the difficult task of diagnosing acute conditions.

The testing went beyond emergency medicine to cover other medical contexts and specialties. That gave the researchers a better sense of how broadly LLMs could apply in clinical practice and showed which areas of medicine AI delivers the most promising results in.

• Analysis of real emergency department cases with complete clinical information
• Comparison of AI diagnostic accuracy against independent, experienced physicians
• Testing across various medical contexts and specialties

Potential and limitations

The results look impressive, but the study is only a first step. Serious questions remain: how the model handles rare and atypical diagnoses, whether it can reliably explain its reasoning to a physician, and how to integrate AI into real clinical workflows without doctors following its recommendations mechanically or blindly.

Critically, AI cannot and should not replace a physician. It doesn't see the patient, doesn't hear their voice, can't perform a physical exam, and doesn't know their social circumstances or psychological state. A physician's language, experience, and intuition remain irreplaceable and critical to a good treatment outcome.

What it means

Language models could become a support tool for physicians: an assistant for second opinions, quick diagnosis checks, or analysis of complex and contested cases. If the findings hold up in larger samples and across different geographic regions, they would open up a new class of LLM applications in healthcare and could speed up diagnosis.

But the core principle doesn't change: the physician remains responsible for the clinical decision and for the patient.