Starts Nov 23Live Cohort 3 of Vibe Coding & AI Agents: 3 weeks, 9 live hands-on sessionsCohort 3 (Nov 23) →
ReleaseAPI / Product · ActionableHabr AI→ original

MWS AI and SberAI propose DRAGOn, a dynamic benchmark for evaluating RAG systems

MWS AI, SberAI and academic partners have introduced DRAGOn, a benchmark for RAG systems built on a regularly updated corpus. Instead of a static question set, it automatically extracts new facts from news, generates QA pairs and filters out overly easy queries that a model can answer from memory. The authors have also launched a public leaderboard for comparing retrievers and generators.

AI-processed from Habr AI; edited by Hamidun News
MWS AI and SberAI propose DRAGOn, a dynamic benchmark for evaluating RAG systems
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Simplea question about a single fact
Seta list of several objects sharing a common relation
Multi-hopa question that goes through an intermediate entity

Researchers from MWS AI, SberAI and several universities have introduced DRAGOn, a dynamic benchmark for evaluating RAG systems that run on a regularly updated corpus. The work was published on arXiv in July 2025, and in March 2026 it was included in the EACL 2026 proceedings as a practical way to test RAG on genuinely new data rather than on a long-frozen set of questions.

Why this is hard

RAG evaluation almost always runs into the same problem: tests age faster than the systems themselves. If a benchmark is built on a fixed corpus, a model can score well not because it retrieves and links documents effectively, but because it already saw some of the facts during training. There is another difficulty: in the final quality score, it is hard to separate the retriever's contribution from the generator's. And preparing question-answer pairs by hand for continuous updates is too expensive, too slow and barely scales for teams that want to compare new versions of their pipelines regularly.

How DRAGOn works

The authors propose building the benchmark as a pipeline. Parsers regularly pull material from news sources, and a separate module then extracts atomic facts from the texts as "subject — relation — object" triplets. The system then matches entities against Wikidata and discards facts that are already known, so that only new knowledge remains in the sample. Questions of varying difficulty are built automatically from the resulting graph, and the benchmark can be re-released regularly without manual rebuilding and with clear version control.

  • Simple: a question about a single fact
  • Set: a list of several objects sharing a common relation
  • Multi-hop: a question that goes through an intermediate entity
  • Conditional: an answer that must satisfy two conditions at once

On top of this, the authors added a public leaderboard and split evaluation into public and private parts. The public part allows open comparison of results, while the private part allows precise checks against the reference and protects against tuning to known answers. This format makes comparing RAG configurations fairer: one team can test a new retriever, another a new generator, and both get comparable results on a fresh corpus rather than on a set the model may have already learned.

How validation works

To keep automatically generated QA pairs from turning into noise, DRAGOn runs them through several filters. First, basic linguistic correctness is checked with RuRoBERTa-large, and then the questions go through an NER check with Natasha. After that, overly easy examples are removed: if small models such as Qwen 2.5 7B or LLaMa 3 8B answer without relying on context, the question is unsuitable for fair RAG evaluation and is excluded from the final version.

Final quality control is handled by POLLUX 7B in LLM-as-a-Judge mode. The model rates grammar, naturalness, correctness and the question's dependence on context, and these ratings are then checked against human annotation. In an experiment with 532 examples, the automatic judge showed high accuracy, although it turned out to be fairly strict.

After filtering, the authors keep 150 high-quality questions per category, then evaluate the systems separately on retrieval and on generation. In the tests, combinations with Qwen 3 Embedding 8B and E5 Mistral 7B Instruct looked strongest. The takeaway is simple: if the retriever finds the right context, the generator has a noticeably easier time giving an accurate answer.

What this means

DRAGOn is an attempt to turn RAG evaluation from a one-off demonstration into a continuously updated process. For teams building search over documents, news or internal knowledge bases, the approach is useful because it reduces the risk of false confidence: a system may answer smoothly on familiar data yet fail on genuinely new facts. A dynamic benchmark helps catch that gap earlier and gives a more honest picture of how ready a RAG system is for real-world use.

Story

Key Questions Answered

Why this is hard?

RAG evaluation almost always runs into the same problem: tests age faster than the systems themselves. If a benchmark is built on a fixed corpus, a model can score well not because it retrieves and links documents effectively, but because it already saw some of the facts during training. There is another difficulty: in the final quality score, it is hard to separate the retriever's contribution from the generator's.

How DRAGOn works?

The authors propose building the benchmark as a pipeline. Parsers regularly pull material from news sources, and a separate module then extracts atomic facts from the texts as "subject — relation — object" triplets. The system then matches entities against Wikidata and discards facts that are already known, so that only new knowledge remains in the sample.

How validation works?

To keep automatically generated QA pairs from turning into noise, DRAGOn runs them through several filters. First, basic linguistic correctness is checked with RuRoBERTa-large, and then the questions go through an NER check with Natasha. After that, overly easy examples are removed: if small models such as Qwen 2.5 7B or LLaMa 3 8B answer without relying on context, the question is unsuitable for fair RAG evaluation and is excluded from the final version.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…