PubHealthBench: 7929 questions test how RAG keeps medical AI from hallucinating

Researchers have extended PubHealthBench — a set of 7929 questions based on official UK government public health guidance — to a retrieval-augmented generation (RAG) setting and systematically tested how different retrieval and generation configurations affect the accuracy of AI answers. The paper was published on arXiv on July 7, 2026.

Why medical LLMs need RAG

Large language models show promising results on medical question-answering benchmarks, but hallucinations and fast-changing official guidance limit their use in public health. RAG reduces these risks by grounding model answers in an explicitly maintained document corpus. Final quality, however, depends heavily on how retrieval is configured and on how the output is evaluated — and multiple-choice formats alone are not enough.

What the retrieval comparison showed

The authors compared dense, sparse, and hybrid retrieval across several embedding models and corpus variants.

• Hybrid retrieval consistently improves recall and ranking quality
• Chunk length and query topic affect ranking quality
• Adding retrieved context substantially improves multiple-choice accuracy across a wide range of LLMs
• Small open models with RAG match or outperform larger models without RAG
• Accuracy gains are driven primarily by retrieval quality and careful context selection

How free-form answers were evaluated

To assess realistic free-form answers rather than just multiple-choice picks, the authors proposed a rubric-based LLM judge covering faithfulness, completeness, clarity, and factual consistency, and validated its scores against dual human annotation. Agreement between the model judge and humans was strongest on faithfulness and completeness. Factual consistency and clarity were reproduced less reliably, which calls for caution when interpreting scores on those two criteria at scale.

What it means

The findings show that retrieval quality is the main lever for reliable AI answers to public health questions, and that even small open models can compete with larger ones when the retrieval side of the system is carefully tuned.

FAQ

What is PubHealthBench?

It is a question-answering benchmark of 7929 questions built from official UK government public health guidance; in this work, it was extended to a RAG setting.

Can a small model replace a large one with RAG?

Yes — according to the authors, providing retrieved context lets small open models match or outperform larger models that run without RAG.