PubHealthBench: 7,929 Questions Test How RAG Keeps Medical AI From Hallucinating

Researchers have extended the PubHealthBench benchmark, a set of 7929 questions based on official UK government public health guidance, to a retrieval-augmented generation (RAG) setting. They systematically tested how different retrieval and generation settings affect the accuracy of AI answers. The paper was published on arXiv on 7 July 2026.

Why LLMs in medicine need RAG

Large language models show promising results on medical question-answering benchmarks, but their use in public health is limited by hallucinations and by the speed at which official guidance changes. RAG reduces these risks by anchoring the model's answers to an explicitly maintained document corpus. Even so, the final quality depends heavily on how retrieval is configured and on how results are evaluated, not only in a multiple-choice format.

What the comparison of retrieval methods showed

The authors compared dense, sparse and hybrid retrieval across several embedding models and corpus variants.

• Hybrid retrieval consistently improves recall and ranking quality
• Chunk length and query topic affect ranking quality
• Adding retrieved context substantially improves multiple-choice accuracy across a wide range of LLMs
• Small open models with RAG match or outperform larger models without RAG
• The accuracy gain is determined primarily by retrieval quality and careful context selection

How free-form answers were evaluated

To assess realistic free-form answers rather than just multiple-choice selection, the authors proposed an LLM judge based on a rubric covering faithfulness, completeness, clarity and factual consistency. They checked its output against double annotation by humans. Agreement between the judge model and humans was strongest on faithfulness and completeness, while factual consistency and clarity are reproduced less reliably. That is a reason to interpret scores on those two criteria with more caution at large scale.

What this means

The results show that retrieval quality is the main lever for reliable AI answers to public health questions, and that even small open models can compete with large ones if the retrieval side of the system is carefully tuned.

Frequently asked questions

What is PubHealthBench?

It is a question-answering benchmark of 7929 questions built from official UK government public health guidance. This work extended it to a RAG setting.

Can a small model replace a large one with the help of RAG?

Yes. According to the authors, providing retrieved context allows small open models to match or outperform larger models that run without RAG.