PubHealthBench: 7,929 Questions Test How RAG Saves AI from Medical Hallucinations

Researchers expanded the PubHealthBench benchmark — a set of 7929 questions based on official recommendations from the UK government on public health — to a retrieval-augmented generation (RAG) scenario and systematically examined how different search and generation settings affect the accuracy of AI responses. The paper was published on arXiv on July 7, 2026.

Why LLMs in Medicine Need RAG

Large language models show promising results on medical question-answering benchmarks, but their application in public health is limited by hallucinations and the rapid evolution of official recommendations. RAG reduces these risks by anchoring the model's responses to an explicitly maintained corpus of documents, but the resulting quality heavily depends on search configuration and how results are evaluated — not only in multiple-choice format.

What the Comparison of Search Methods Showed

The authors compared dense, sparse, and hybrid search across multiple embedding models and corpus variants.

• Hybrid search consistently improves recall and ranking quality
• Fragment length and query topic affect ranking quality
• Adding retrieved context significantly increases answer accuracy across many different LLMs
• Small open models with RAG compare to or outperform larger models without RAG
• Accuracy gains are primarily determined by search quality and careful context selection

How Generated Answers Were Evaluated

To evaluate realistic generated answers rather than just multiple choice, the authors proposed an LLM judge based on a rubric covering faithfulness, completeness, clarity, and factual consistency, and validated the judge against double human annotation. The judge's agreement with humans was strongest on faithfulness and completeness criteria, whereas factual consistency and clarity were reproduced less reliably — suggesting caution when interpreting these two criteria at scale.

What This Means

The study results show that search quality is the main lever for reliable AI answers to public health questions, and even small open models can compete with larger ones if the retrieval component is carefully configured.

Frequently Asked Questions

What is PubHealthBench?

It is a question-answering benchmark consisting of 7929 questions based on official recommendations from the UK government on public health; in this work it was expanded to a RAG scenario.

Can a small model replace a larger one with RAG?

Yes — according to the authors, providing retrieved context allows small open models to match or exceed larger models that operate without RAG.