A review of 18 medical LLMs: specialist models diagnose more accurately than general-purpose ones

In July 2026, a systematic review of medical LLMs appeared on arXiv. Its authors tested 18 current language models on clinical reasoning tasks. The main finding: narrowly specialized medical models make more accurate diagnoses, while general-purpose language models do better at supporting clinical decisions and holding clinical conversations with patients.

Five levels of medical reasoning

To measure model capabilities systematically, the authors relied on Miller's Pyramid, a classic framework for assessing medical competence that is widely used in professional education. The framework lays out five levels, from simple to complex: from recalling facts to independently managing dynamic clinical cases.

• Level 1: recalling facts — does the model know anatomy, pharmacology, and clinical protocols?
• Level 2: understanding mechanisms — can it explain pathophysiology and cause-and-effect relationships?
• Level 3: applying knowledge to a case — can it apply what it knows to a specific patient?
• Level 4: clinical diagnosis — can it build a differential diagnosis from incomplete data?
• Level 5: case management — can it handle ongoing monitoring and adjust the treatment strategy?

The authors mapped three types of reasoning onto this hierarchy: deductive (from a rule to a specific case), inductive (from observed symptoms to a hypothesis), and abductive (finding the most likely explanation when data is incomplete). This approach ties a model's capabilities to real clinical tasks rather than simply to accuracy on standard test sets.

What did the comparison of 18 models show?

The researchers found a clear split in strengths. Narrowly specialized medical models win decisively on diagnostic tasks, where they need to reproduce clinical standards precisely and build a differential diagnosis. General-purpose language models, by contrast, are better at clinical dialogue and at helping doctors weigh treatment options when the data is ambiguous.

To make the comparison systematic, the researchers built, for the first time, a unified benchmark covering all five competence levels of Miller's Pyramid. That is a fundamental step: most existing medical tests measure factual accuracy, which means they effectively operate at levels 1–2 out of five. The new tool makes it possible to assess how ready a model is for a real clinic, not just for passing a standard medical exam.

Three barriers to the clinic

The review identifies three open challenges that still keep language models out of routine medical use.

Hallucinations remain the core problem. Models confidently describe treatment regimens that don't exist or cite fictitious studies. In a clinical setting, such errors are unacceptable: the cost of a wrong prescription is in a different league from the cost of a typo in an ordinary chatbot.

Data scarcity holds back progress across the entire field. Labeling high-quality medical cases takes the time of experienced clinicians, and annotated datasets covering the full range of clinical situations, from rare diseases to multimorbid patients, are critically scarce.

The grounding problem means models struggle to link their answers to verifiable primary sources, such as clinical guidelines and peer-reviewed research. Without that link, it is practically impossible for a doctor to verify a model's recommendation and take responsibility for acting on it.

What this means

The gap between specialized and general-purpose medical models is not a fluke but a systematic pattern. Bringing AI into real clinical practice will likely require hybrid approaches: narrow models for diagnosis paired with general-purpose ones for communication and clinical decision support. The review sets out a common framework for such work and offers the first benchmark built on real clinical competencies rather than simply on the accuracy of test answers.