Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseResearch Paper · For R&DHabr: MTS AI→ original

DRAGOn Benchmark from MWS AI and Sber Tests RAG on Fresh News: 150 Questions per Category

RAG means the model searches your database for data before it answers, and that is hard to test honestly, since the model may have seen old test sets during training. DRAGOn from MWS AI and Sber takes fresh news, extracts facts, and discards any that already exist in Wikidata. The remaining facts are turned into questions of four types, with the best 150 kept per category. That is why RAG is better compared on things the model did not know.

AI-processed from Habr: MTS AI; edited by Hamidun News
DRAGOn Benchmark from MWS AI and Sber Tests RAG on Fresh News: 150 Questions per Category
Source: Habr: MTS AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr: MTS AI ↗
Editor-in-Chief
Key metric #1Qwen3 Embedding 8B and E5 Mistral 7B Instruct showed the best retrieval
The POLLUX 7B judge is strictits precision is high, but it rejects some valid questions
Key metric #3For now, the benchmark covers only Russian-language news

Today you can build a RAG system in an evening: connect a vector database, take an embedding model, and add an LLM. RAG (Retrieval-Augmented Generation) means that before answering, the model first searches the connected database for the relevant documents and adds them to the prompt. It is much harder to tell whether all of this actually works well.

The paper DRAGOn: Designing RAG On Periodically Updated Corpus tackles this problem. Its authors from MWS AI, Sber, and several universities presented the work at EACL 2026. DRAGOn is a benchmark, that is, a set of test questions that is assembled automatically from fresh news.

After all the filters, 150 questions remain per category. The idea is to test the retrieval, not the model's memory.

Why standard benchmarks mislead about RAG

Most RAG tests share a few common weaknesses. The document corpus is fixed and over time stops reflecting the real world. Questions and answers are usually labeled by hand, which is slow and expensive. It is also hard to tell which component of the system affected the result.

But the most unpleasant problem is different: it is unclear what explains a good result. Perhaps retrieval really found the right passage. But the base model may simply have memorized these facts during training and answered from memory. Imagine you are testing a support bot that is supposed to read your knowledge base, but it answers from its head. The test will show "excellent," even though retrieval may not have worked at all.

The authors reduce this to one point: the test set is not independent of the training data. Their direct solution is to build the benchmark on knowledge the model has certainly not seen. This requires three parts: a regularly updated document corpus, automatic generation of question-answer pairs, and a single public evaluation system.

How DRAGOn creates questions from news

The pipeline runs in sequence, and each step can be reproduced.

First, parsers pull texts from external sources. In the paper these are news aggregators, but the approach is not tied to news. Then Llama 3.3 70B Instruct extracts facts from the texts as subject-relation-object triples, for example "company X — acquired — startup Y."

Next comes the operation that matters most for the test's cleanliness. Each fact is checked through the Wikidata API. If it is already there, it is discarded, because the model could well have known it. After that, the same Llama normalizes entity and relation names, and only new facts enter the graph.

From the graph, questions of four types are assembled:

  • single-fact: "Who heads SpaceX?";
  • with a list of answers: "For which films did Hans Zimmer compose music?";
  • chain of facts, where the intermediate entity is hidden in the question: "In which country is the company that sold 2139 cars in 2023 located?";
  • intersection of conditions, where the answer is a shared entity: "Who performed at M-bar and met with Dmitry Dibrov?"

Then comes multi-stage cleaning. One model filters out grammatical errors and unnatural phrasing. An NER library checks named entities. Small models try to answer without context, and if they succeed, the question is removed: it is too easy and does not require retrieval. The main assessment is made by POLLUX 7B acting as an LLM judge, scoring from 0 to 2 for fluency, naturalness, correctness, and dependence on context. The final set keeps 150 questions per category.

The authors checked the judge against human annotators. POLLUX 7B showed high precision but moderate recall: it reliably selects good questions, but sometimes rejects acceptable ones. For a benchmark this is acceptable, since it is better to have fewer questions of verified quality.

What the tests showed and where the limits are

Retrieval and generation were evaluated separately. For retrieval, the authors measured how often the correct document lands in the top-N results and how high it ranks (the MRR metric). The best results came from pairings with Qwen3 Embedding 8B and E5 Mistral 7B Instruct. For generation, they used ROUGE-2 and ROUGE-L, which measure overlap of words and phrases with the reference answer. The main emphasis was on POLLUX 7B's evaluation: how correct and complete the answer is, and how well it is grounded in the retrieved context.

The conclusion is down to earth: what context you feed in is what answer you get. Systems with strong retrieval also gave the best answers. In practice, this means that when debugging RAG, you should check the retriever and embeddings first, rather than swapping the generating model.

The authors also launched a public leaderboard. Results are submitted through a dedicated library. Part of the data is hidden for verification, so the comparison stays fair.

The work has limitations. For now the benchmark covers only Russian-language news, and a news stream differs from medical, legal, or scientific texts. The architecture allows the approach to be moved to other domains, but there are no ready-made versions for them. In addition, the LLM judge speeds up the work but does not replace an expert: it still errs on the side of caution.

The practical takeaway for teams building RAG in Russian: if you have your own stream of regularly updated documents (support, knowledge base, industry news), you can repeat this scheme. Extract facts, filter out what the model already knows, and build questions only from new material. This way the test measures how retrieval works, not the model's memory.

Story

Key Questions Answered

Why standard benchmarks mislead about RAG?

Most RAG tests share a few common weaknesses. The document corpus is fixed and over time stops reflecting the real world. Questions and answers are usually labeled by hand, which is slow and expensive. It is also hard to tell which component of the system affected the result. But the most unpleasant problem is different: it is unclear what explains a good result. Perhaps retrieval really found the right passage. But the base model may simply have memorized these facts during training and answered from memory.

How DRAGOn creates questions from news?

The pipeline runs in sequence, and each step can be reproduced. First, parsers pull texts from external sources. In the paper these are news aggregators, but the approach is not tied to news. Then Llama 3.3 70B Instruct extracts facts from the texts as subject-relation-object triples, for example "company X — acquired — startup Y." Next comes the operation that matters most for the test's cleanliness. Each fact is checked through the Wikidata API.

What the tests showed and where the limits are?

Retrieval and generation were evaluated separately. For retrieval, the authors measured how often the correct document lands in the top-N results and how high it ranks (the MRR metric). The best results came from pairings with Qwen3 Embedding 8B and E5 Mistral 7B Instruct. For generation, they used ROUGE-2 and ROUGE-L, which measure overlap of words and phrases with the reference answer.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…