The "no gold documents" label lost accuracy, from 0.68 to 0.10, on strong retrievers

Conversational RAG benchmarks such as MTRAG and RETECO evaluate the whole system: the model answers from the documents it retrieved itself and must refuse if they do not contain the answer. To credit a refusal, you need to know whether the retrieved context is sufficient. Labeling the contexts of every system by hand is not feasible, so in practice evaluators use a shortcut: if none of the retrieved documents is a gold document from the relevance labels, the context is treated as insufficient to answer.

The author of a reproduction study, from Letovo School in Moscow, tested this rule on RECOR (the second track of RETECO) and showed that it stops working as the retriever, the module that does the first-pass document search, gets better. The rule's precision fell from 0.68 to 0.

10 while nDCG@10 (a metric of ranking quality in the top 10 results) rose from 0.18 to 0.59.

How the rule was tested

RECOR consists of dialogues. Each one expands a question from StackExchange or BRIGHT into 3–5 short, context-dependent turns. Every turn has gold documents and a reference answer generated by GPT-4.1 from verified atomic facts. The domain corpora contain from 16 thousand to 121 thousand documents, including negative examples mined for each question. The author used 858 turns from 211 dialogues in the dev split.

First, he exactly reproduced the official BM25 baselines: the scores matched the published ones to the fourth decimal place in all 11 domains (macro-average nDCG@10 on dev of 0.183 and 0.438). He then built a "ladder" of four retrievers: BM25 on the current turn, BM25 with dialogue history, Fusion (a merge of BM25 runs over three LLM query expansions), and a hybrid with the dense retriever reason-embed-qwen3-4b. Their nDCG@10 on dev is 0.183, 0.438, 0.550 and 0.591, and the share of turns with at least one gold document in the top 5 is 0.28, 0.65, 0.77 and 0.82.

The content of each context was judged by two open LLM judges and a panel of four small models: Qwen3-4B-Instruct-2507, Phi-4-mini-instruct, SmolLM3-3B and Granite-3.3-8B-instruct. In addition, Claude Opus 5.5 labeled 250 contexts blind, seeing neither the retriever nor the label from qrels. Its two runs agreed in 97% of cases, and a third run settled the disputed examples. The answering models were Qwen3-1.7B, Qwen3-4B-Instruct-2507 and Qwen3-8B.

The stronger the retriever, the more often the rule is wrong

Among contexts without gold documents, the share that still contain the answer rose from 28% to 72%. The rule's precision as a detector of insufficient context fell from 0.68 to 0.10, and from 0.70 to 0.21 in the blind audit. The cause is incomplete labeling. Most often, the answer sits in a hard negative mined for the same question, in a gold document of a different turn, or in a neighboring passage of a gold page. In an example about glucose meters, the retrieved document explains that the enzyme glucose oxidase is specific only to D-glucose. But qrels marks it as a negative, and the search did not return the actual gold document containing the same passage.

Independent checks support these findings: counting copies of gold documents without any judges, the blind audit, a negative control, human ratings on MTRAG, and the answers of the model itself. Qwen3-4B answers correctly on 62–67% of these false misses, on 23–30% of contexts the judges rated insufficient, and on 1% of turns with no documents. The four small judges reproduce the drop, but they grade more leniently than the large ones.

What changes for evaluating refusals

The choice of label can flip conclusions. When evaluation relies on the presence of gold documents, the AUROC for answerability detection is understated, falling as low as 0.19, and the best-looking sufficiency detector is the retriever's score, which does not read the documents at all. On MTRAG, where annotators did the labeling and there are no mined negatives, the rule's precision drops from 0.72 to 0.50. So the effect is not limited to RECOR.

There are caveats. The query expansions for Fusion and the hybrid were generated by a closed model, Claude Sonnet 5. When rebuilt with an open model, Query2doc passages with BM25 give 0.453 instead of 0.546. The auditor's 97% agreement between two runs reflects the self-consistency of a single model, not agreement between independent annotators.

The author's practical recommendation: when comparing retrievers of different strength, credit a refusal based on a content label for each context, not on the presence of gold documents. Otherwise a strong retriever looks worse, and a model that refuses to answer even though the documents contain the answer gets credit. The code, runs and labels are published in the reteco-2026 repository.