Custom RAG Beats OpenAI and Grok in the Legal Agentic RAG Legal Challenge

The authors of the write-up break down how their team handled the Agentic RAG Legal Challenge, an international competition on answering questions from legal PDFs. The main takeaway is uncomfortably practical: what wins is not a big model name but the ability to tie an answer precisely to the right page of a document.

How the systems were tested

More than 300 teams entered the challenge. The corpus consisted of real court rulings, laws and regulatory acts from the DIFC, in English. Participants first received 30 documents and 100 questions for a warm-up, then nearly 300 documents and 900 questions for the final. The questions came in different types: dates, numbers, names, lists, yes/no and short free-form answers. But what mattered most was not only the accuracy of the wording but the Grounding metric: whether the cited pages matched the pages the answer was actually taken from.

"Even a perfect answer scores zero if it cites the wrong page."

For this, the team built its own pipeline: PDFs were converted to Markdown, split into semantic chunks, each chunk was given generated context, dense and sparse embeddings were computed, and everything was stored in Qdrant. Part of the work ran locally on a Mac Studio M3 Ultra. The machine quickly handled parsing of 30 PDFs and local embeddings, but generating context for the chunks proved too slow: because of the long prefill, each chunk took 15–20 seconds, so this stage had to be moved to an external API.

Who failed first

The authors first tried the laziest route: simply uploading the documents to OpenAI's built-in knowledge base. By eye the result looked decent, but the metrics gave a Total of 0.362: the answers were often fine, but page citation ruined everything.

They then tested CAG, where almost the entire corpus is sent to the model at once, with no chunk-level retrieval. An experiment with Qwen 3.5 Flash and a context of up to 1 million tokens showed that CAG is not useless: accuracy was high, but Grounding let it down again. Pure BM25 did even worse and was the weakest attempt.

This run produced several unpleasant but useful conclusions:

• built-in knowledge bases from big players do not guarantee good citation;
• CAG can answer accurately, but without careful page attribution it loses on the final score;
• classic BM25 alone can no longer handle complex legal questions;
• hybrid RAG with proper reranking proved stronger than the built-in OpenAI and Grok solutions.

The team's own system, MORAG, was not impressive at first either: a small local Qwen held accuracy poorly and fell apart especially on multi-document questions. The turning point came after switching to Grok via OpenRouter and selecting chunks more strictly. In the warm-up, the team rose from a Total of 0.362 in early runs to 0.780 in the final attempt, and Grounding climbed from roughly 0.45 to 0.90. This improvement, rather than swapping one fashionable model for another, was the main driver of progress.

What actually helped

The biggest gains came not from abstract "quality improvements" but from a handful of very concrete engineering decisions. The team separated reasoning and non-reasoning modes by question type, added an agentic loop that searches again when data is insufficient, and separately built a gold set to check answers across the 900 questions. This let them avoid shooting blind in the final and quickly find systematic errors, such as misreading the wording about an appeal that was filed but rejected.

• the reasoning model was kept for boolean, name and names questions, where non-reasoning lost 8–16% accuracy;
• the faster non-reasoning mode was used for date, number and free_text with no noticeable drop;
• the first 1–3 pages of documents mentioned in the question were added to the search, because key case details often sit there;
• summaries and sparse vectors were rebuilt for the legal domain;
• chunks were trimmed to fit the limit of the FRIDA embedder, which truncates anything longer than 512 tokens.

In the final phase, MORAG trailed the prepared golden submission on overall Total, 0.603 versus 0.631, but beat it on three of five metrics: accuracy of deterministic answers, quality of free-form answers and speed. The shortfall again came from Grounding. This is an important nuance: the RAG system itself already answered better than the "manual" reference, but the technical linking of the answer to the right page still lagged.

What this means

The story shows that CAG has not killed RAG, that a Mac Studio suits part of a local pipeline, and that ready-made OpenAI and Grok knowledge bases do not replace tuning for a specific corpus. When the data is complex, the winner is not the loudest brand but the team that can measure errors, control chunking and bring Grounding to a working state.