Rubric4Setwise: оценка наборов документов для RAG вместо ранжирования по релевантности

Researchers in July 2026 presented SetwiseEvalKit — a benchmark for evaluating the quality of an entire set of documents in search for large language models — and a training-free method called Rubric4Setwise. When testing 12 rerankers, even the best of them covered no more than 45% of the evaluation criteria.

What's wrong with relevance-based scoring

Existing search evaluation systems rank each document independently and roll the result up into the nDCG metric, ignoring the relationships between documents. According to the research on arXiv, this approach misses redundancy (when documents duplicate each other), conflicts (when data contradicts), and complementarity — and therefore cannot answer the main question: why is one document set better than another.

This is critical because language models and AI agents have become the primary consumers of search results: the quality of the document set sets the upper bound on the quality of the resulting generation. A bad set means a bad answer, no matter how strong the reranker is.

How SetwiseEvalKit works

SetwiseEvalKit is a benchmark with three levels and nine dimensions that evaluates a document set as a whole, not document by document. It covers both short and long generation scenarios and contains about 28000 quality evaluation criteria (rubrics).

• Three levels and nine dimensions for evaluating document sets
• About 28000 evaluation criteria (rubrics)
• Coverage of both short-form and long-form scenarios
• Separate dimensions for coordination between documents: redundancy, conflict, complementarity

It's precisely these "set-level" dimensions that break the usual logic: a document can be relevant on its own but still hurt the set — by duplicating a neighbor or contradicting it.

What the test of 12 rerankers showed

None of the 12 tested rerankers handled the task: the best method achieved no more than 45% criteria coverage, and the dimensions measuring coordination between documents turned out weak across the board, without exception. No method held the lead simultaneously in both short and long scenarios — one that was strong in one mode sagged in the other.

"Even the best method achieves no more than 45% coverage, and

cross-document coordination dimensions are universally weak," the researchers' paper on arXiv states.

How Rubric4Setwise helps

Rubric4Setwise is a training-free method (requiring no training) that turns rubric-scoring criteria into signals for selecting a document set. According to the research, it delivers better final generation with fewer documents and fewer search rounds. It's the only method that held state-of-the-art results in both scenarios — short and long — closing the loop of "evaluation → diagnosis → optimization."

What this means

As LLMs and agents increasingly replace humans in reading search results, what matters is no longer "the single most relevant document" but the quality of the whole set. SetwiseEvalKit shows that today's rerankers aren't ready for this, and Rubric4Setwise hints at how to fix it — without further training and with savings on the number of sources.

Frequently asked questions

What is Rubric4Setwise?

Rubric4Setwise is a training-free method for selecting document sets for LLM-facing search. It converts rubric-scoring criteria into selection signals and achieves better generation with fewer documents and fewer search rounds.

How does document-set evaluation differ from ordinary ranking?

Ordinary ranking via nDCG scores each document separately and doesn't see the relationships between them. Set evaluation accounts for redundancy, conflicts, and complementarity among documents — that is, the quality of the selection as a whole, rather than the sum of individual scores.