Rubric4Setwise: Evaluating Whole Document Sets for RAG Instead of Ranking by Relevance

In July 2026, researchers introduced SetwiseEvalKit, a benchmark for evaluating the quality of an entire document set in search for large language models, along with Rubric4Setwise, a training-free method. In a test of 12 rerankers, even the best one covered no more than 45% of the evaluation criteria.

What is wrong with relevance-based evaluation

Existing search evaluation systems rank each document independently and roll the results into an nDCG metric, ignoring relationships between documents. According to the study on arXiv, this approach misses redundancy (documents that duplicate each other), conflicts (data that contradicts itself) and complementarity. As a result, it cannot answer the central question: why one set of documents is better than another.

This matters because language models and AI agents have become the main consumers of search results: the quality of the document set sets the upper bound on the quality of the final generation. A bad set produces a bad answer, no matter how strong the reranker is.

How SetwiseEvalKit works

SetwiseEvalKit is a benchmark with three levels and nine dimensions that evaluates a document set as a whole rather than document by document. It covers both short-form and long-form generation scenarios and contains about 28,000 high-quality evaluation criteria (rubrics).

• Three levels and nine dimensions for evaluating a document set
• About 28,000 evaluation criteria (rubrics)
• Coverage of both short-form and long-form scenarios
• Dedicated dimensions for coordination between documents: redundancy, conflict, complementarity

These set-level dimensions break the usual logic: a document can be relevant on its own yet hurt the set by duplicating a neighbor or contradicting it.

What the test of 12 rerankers showed

None of the 12 rerankers tested handled the task: the best method reached no more than 45% criteria coverage, and the dimensions measuring coordination between documents proved weak across the board, without exception. No method held the lead in both short-form and long-form scenarios at once: a method that was strong in one mode dropped off in the other.

"Even the best method achieves no more than 45% coverage, and the

cross-document coordination dimensions are universally weak," the researchers write in their arXiv paper.

How Rubric4Setwise helps

Rubric4Setwise is a training-free method (it requires no training) that turns criteria from rubric-based evaluation into signals for selecting a document set. According to the study, it produces better final generation with fewer documents and fewer search rounds. It is the only method that held state-of-the-art results in both scenarios, short-form and long-form, closing the loop of evaluation, diagnosis and optimization.

What this means

As LLMs and agents take humans out of the loop of reading search results, what matters is no longer the single most relevant document but the quality of the whole set. SetwiseEvalKit shows that today's rerankers are not ready for this, and Rubric4Setwise hints at how to fix it: without fine-tuning, and with savings on the number of sources.

Frequently asked questions

What is Rubric4Setwise?

Rubric4Setwise is a training-free method for selecting a document set in search for LLMs. It converts rubric-based evaluation criteria into selection signals and achieves better generation with fewer documents and fewer search rounds.

How does evaluating a document set differ from ordinary ranking?

Ordinary ranking with nDCG scores each document separately and cannot see relationships between them. Set evaluation accounts for redundancy, conflicts and complementarity among documents, that is, the quality of the selection as a whole rather than the sum of individual scores.