Review of 18 medical LLMs: narrow specialists diagnose more accurately than general-purpose models

In July 2026, a systematic review of medical LLMs was published on the arXiv platform, in which the authors tested 18 contemporary language models on clinical reasoning tasks. The main conclusion: narrowly specialized medical models diagnose more accurately, while general-purpose language models perform better at supporting physician decisions and clinical dialogue with patients.

Five Levels of Medical Thinking

To measure model capabilities systematically, the authors relied on Miller's Pyramid — a classical framework for evaluating medical competencies adopted in professional education. The framework establishes five levels from simple to complex: from factual reproduction to independent management of dynamic clinical cases.

• Level 1: factual reproduction — does the model know anatomy, pharmacology, clinical protocols?
• Level 2: understanding mechanisms — does it explain pathophysiology and causal relationships?
• Level 3: application to a case — is it able to apply knowledge to a specific patient?
• Level 4: clinical diagnosis — does it build a differential diagnosis with incomplete data?
• Level 5: case management — does it maintain dynamic observation and adapt treatment strategy?

The authors linked three types of reasoning to this hierarchy: deductive (from rule to specific case), inductive (from observed symptoms to hypothesis), and abductive (finding the most probable explanation with incomplete data). This approach allows correlating model capabilities with real clinical practice tasks, rather than simply with accuracy on standard test sets.

What Did Comparison of 18 Models Show?

Researchers discovered a clear division by strengths. Narrowly specialized medical models clearly outperform in diagnostic tasks: where precise reproduction of clinical standards and building a differential diagnosis is needed. General-purpose language models, conversely, conduct clinical dialogue better and help physicians weigh treatment options in situations with ambiguous data.

To make the comparison systematic, researchers for the first time created a unified benchmark covering all five levels of competency from Miller's Pyramid. This is a fundamental step: most existing medical tests evaluate factual accuracy — essentially working at levels 1–2 out of five. The new tool makes it possible to assess how ready a model is for real clinical practice, not just for passing a standard medical exam.

Three Barriers to Clinical Implementation

The review identifies three open challenges that currently prevent language models from routine application in medicine.

Hallucinations remain a key problem. Models confidently present nonexistent treatment schemes or cite fictitious research. In a clinical context, such errors are unacceptable: the cost of incorrect prescription is incommensurable with the cost of a typo in an ordinary chatbot.

Data scarcity impedes all progress in this direction. Labeling quality medical cases requires the time of experienced clinicians, and annotated datasets covering the full spectrum of clinical situations — from rare diseases to polymorbid patients — are critically scarce.

The grounding problem means that models poorly link their answers to verifiable primary sources — clinical guidelines and peer-reviewed research. Without this, it is practically impossible for a physician to verify a model's recommendation and take responsibility for its application.

What This Means

The gap between specialized and general-purpose medical models is not coincidental, but a systemic pattern. For real AI deployment in clinical practice, hybrid approaches will likely be needed: narrow models for diagnosis coupled with general-purpose models for communication and support of physician decisions. The review establishes a unified framework for such development and offers the first benchmark based on real clinical competencies — not merely accuracy on test answers.