ChatGPT got all five medical cases right on diagnosis but fell behind on action plans

The experiment was designed on the expectation that ChatGPT would make at least one diagnostic error. It ended with a result the authors did not anticipate: the model identified the primary diagnosis correctly in all five medical cases. But winning on the wording of the diagnosis did not turn into an overall win in the clinical comparison. The most visible gap appeared afterward, in the practical action plan that follows the answer: which examinations are needed before starting therapy, which specialists to refer the patient to, which target values to monitor and when to repeat tests. It was at this stage that ChatGPT more often fell behind the specialized service MedAssist.

The comparison covered five cases: metabolic syndrome, subclinical hypothyroidism, perimenopause, MGUS and statin-induced rhabdomyolysis. In every case ChatGPT landed on the main diagnosis, which is notable in itself for a general-purpose LLM. The authors admit that before the run they expected at least one serious error, but it did not happen.

However, in medicine the diagnosis is only part of the task. The next step matters no less: whether it is safe to start treatment, which red flags to check in advance and which follow-up tests are needed so as not to miss contraindications or accompanying risks. This is where the difference between the models became systematic.

In the four planned cases, ChatGPT answered worse the question of what the patient should do over the next two weeks. This was not a matter of polished wording but of applied clinical logic: for example, reminding about PSA before testosterone replacement therapy, about mammography before menopausal hormone therapy, about target levels of indicators and timing of rechecks. In the rhabdomyolysis case, interpreting the AST-to-ALT ratio also proved important, a detail that affects how the causes of changes in test results are understood and what happens next.

But the comparison also had a counterexample. In the MGUS case, monoclonal gammopathy of undetermined significance, it was MedAssist that came out weaker. ChatGPT explicitly calculated the albumin-to-globulin ratio and separately listed the confirmatory tests to run when referring the patient to a hematologist. The authors state outright that their service did neither, and that is exactly why their analysis of this case turned out the most detailed. Such an episode matters not only as a local loss but also as a reminder: a specialized product does not gain an advantage automatically just because it was built for a narrow task.

Separately, the authors note a possible conflict of interest: the text was prepared by the team that makes MedAssist, one of the two participants in the comparison. They do not try to hide this and say they fixed the methodology in advance, published both services' answers verbatim and analyzed their own failed case in detail rather than in passing. This does not remove questions about full neutrality, but it makes the material more useful than a typical marketing demo that shows only convenient examples. For the reader, the most valuable thing here is not the score by case but the transparency about where exactly the models are strong and where they start to err in applied decisions.

The main takeaway from this test is fairly down-to-earth: large language models can already hit the diagnosis consistently even in complex cases, but the quality of a medical answer cannot be judged by its first line alone. If a system correctly names the condition but does not point to the mandatory examinations before therapy, does not outline the route to the right specialist and does not specify monitoring timelines, the risk of error does not go away. For developers of medical AI services, this is a signal to shift focus from impressive diagnoses to the full patient-management scenario.

For users, it is a reminder that the value of such systems is determined not only by recognition accuracy but also by the safety of the next step.