MWS AI and SberAI propose DRAGOn, a dynamic benchmark for evaluating RAG systems
MWS AI, SberAI and academic partners have introduced DRAGOn, a benchmark for RAG systems built on a regularly updated corpus. Instead of a static question set, it automatically extracts new facts from news, generates QA pairs and filters out overly easy queries that a model can answer from memory. The authors have also launched a public leaderboard for comparing retrievers and generators.
AI-processed from Habr AI; edited by Hamidun News
Researchers from MWS AI, SberAI and several universities have introduced DRAGOn, a dynamic benchmark for evaluating RAG systems that run on a regularly updated corpus. The work was published on arXiv in July 2025, and in March 2026 it was included in the EACL 2026 proceedings as a practical way to test RAG on genuinely new data rather than on a long-frozen set of questions.
Why this is hard
RAG evaluation almost always runs into the same problem: tests age faster than the systems themselves. If a benchmark is built on a fixed corpus, a model can score well not because it retrieves and links documents effectively, but because it already saw some of the facts during training. There is another difficulty: in the final quality score, it is hard to separate the retriever's contribution from the generator's. And preparing question-answer pairs by hand for continuous updates is too expensive, too slow and barely scales for teams that want to compare new versions of their pipelines regularly.
How DRAGOn works
The authors propose building the benchmark as a pipeline. Parsers regularly pull material from news sources, and a separate module then extracts atomic facts from the texts as "subject — relation — object" triplets. The system then matches entities against Wikidata and discards facts that are already known, so that only new knowledge remains in the sample. Questions of varying difficulty are built automatically from the resulting graph, and the benchmark can be re-released regularly without manual rebuilding and with clear version control.
- Simple: a question about a single fact
- Set: a list of several objects sharing a common relation
- Multi-hop: a question that goes through an intermediate entity
- Conditional: an answer that must satisfy two conditions at once
On top of this, the authors added a public leaderboard and split evaluation into public and private parts. The public part allows open comparison of results, while the private part allows precise checks against the reference and protects against tuning to known answers. This format makes comparing RAG configurations fairer: one team can test a new retriever, another a new generator, and both get comparable results on a fresh corpus rather than on a set the model may have already learned.
How validation works
To keep automatically generated QA pairs from turning into noise, DRAGOn runs them through several filters. First, basic linguistic correctness is checked with RuRoBERTa-large, and then the questions go through an NER check with Natasha. After that, overly easy examples are removed: if small models such as Qwen 2.5 7B or LLaMa 3 8B answer without relying on context, the question is unsuitable for fair RAG evaluation and is excluded from the final version.
Final quality control is handled by POLLUX 7B in LLM-as-a-Judge mode. The model rates grammar, naturalness, correctness and the question's dependence on context, and these ratings are then checked against human annotation. In an experiment with 532 examples, the automatic judge showed high accuracy, although it turned out to be fairly strict.
After filtering, the authors keep 150 high-quality questions per category, then evaluate the systems separately on retrieval and on generation. In the tests, combinations with Qwen 3 Embedding 8B and E5 Mistral 7B Instruct looked strongest. The takeaway is simple: if the retriever finds the right context, the generator has a noticeably easier time giving an accurate answer.
What this means
DRAGOn is an attempt to turn RAG evaluation from a one-off demonstration into a continuously updated process. For teams building search over documents, news or internal knowledge bases, the approach is useful because it reduces the risk of false confidence: a system may answer smoothly on familiar data yet fail on genuinely new facts. A dynamic benchmark helps catch that gap earlier and gives a more honest picture of how ready a RAG system is for real-world use.
- Nov 12, 2026Eight Countries, OpenAI and Anthropic Promise an AI "Kill Switch" — but Set No Deadlines or Penalties
- Nov 12, 2026ElevenLabs Scribe v3 Live Teaches Voice Agents Not to Interrupt: 25% Fewer Errors
- Nov 12, 2026Russian Prompts in ComfyUI: FLUX.2 and Qwen-Image Lose 4–13 Points, SD-Turbo Loses Almost Everything
- Nov 8, 2026AI visit notes fill 50 fields in 35 seconds in Moscow, while private clinics remain silent
Key Questions Answered
Why this is hard?
RAG evaluation almost always runs into the same problem: tests age faster than the systems themselves. If a benchmark is built on a fixed corpus, a model can score well not because it retrieves and links documents effectively, but because it already saw some of the facts during training. There is another difficulty: in the final quality score, it is hard to separate the retriever's contribution from the generator's.
How DRAGOn works?
The authors propose building the benchmark as a pipeline. Parsers regularly pull material from news sources, and a separate module then extracts atomic facts from the texts as "subject — relation — object" triplets. The system then matches entities against Wikidata and discards facts that are already known, so that only new knowledge remains in the sample.
How validation works?
To keep automatically generated QA pairs from turning into noise, DRAGOn runs them through several filters. First, basic linguistic correctness is checked with RuRoBERTa-large, and then the questions go through an NER check with Natasha. After that, overly easy examples are removed: if small models such as Qwen 2.5 7B or LLaMa 3 8B answer without relying on context, the question is unsuitable for fair RAG evaluation and is excluded from the final version.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.