Starts Nov 23Live Cohort 3 of Vibe Coding & AI Agents: 3 weeks, 9 live hands-on sessionsCohort 3 (Nov 23) →
ReleaseInsight · On RadarHabr AI→ original

Agentic Legal RAG Challenge 2026: How Sparks of intelligence Tested the Limits of Agentic RAG

The Sparks of intelligence team has published a write-up of its participation in Agentic Legal RAG Challenge 2026, a hackathon on answering questions over DIFC court documents. The authors built two architectures: a simple one based on hybrid search, and an agentic one with a tool router. The simpler solution turned out to be more predictable. The main bottlenecks were chunking, grounding and a lack of time for testing.

AI-processed from Habr AI; edited by Hamidun News
Agentic Legal RAG Challenge 2026: How Sparks of intelligence Tested the Limits of Agentic RAG
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1The simple version split documents by page and gave clear grounding right away.
Key metric #2Its search combined vectors, metadata and regex filters.
Key metric #3The agentic version used a router and four tools: metadata search, exact match, document comparison and hybrid search.

The Sparks of intelligence team has published a detailed write-up of its participation in Agentic Legal RAG Challenge 2026, an international legal RAG hackathon. This is not a story about a big win, but a useful engineering report on why document search systems more often break at context preparation than at the choice of the LLM itself.

How the hackathon worked

The competition was run by EORA AI Applications and Services. Participants had to build a system that answers questions about the documents of the courts of the Dubai International Financial Centre (DIFC). The hackathon ran in two stages: from 11 to 19 March 2026, participants worked with 30 documents and 100 questions, and in the final, held from 20 to 22 March 2026, the volume grew to 300 documents and 900 questions. The prize pool was $32,000, and more than 300 people took part in total.

The difficulty was not only volume. The organizers built in different answer types in advance: boolean, name, date, number and free text. A single generation model was therefore not enough: the system had to extract a fact precisely, hold context, and avoid spending too much time and too many tokens. Free-text answers were scored by an LLM, and the key criteria included accuracy, speed and processing cost. In effect, participants were tested not on their ability to "plug in a chatbot" but on the maturity of the entire retrieval pipeline.

Two versions of the system

The team built two architectures on the same stack: Qdrant as the vector database, LlamaIndex for indexes and LLM abstractions, and Unstructured for extracting text from PDFs while preserving structure. After that, the paths diverged.

The first version was as practical as possible: page-based chunking with overlap, hybrid search, and filtering by metadata and regular expressions. The second was noticeably more ambitious: hierarchical chunking, preliminary structure analysis via an LLM, and a router agent that picks the right search tool for each question.

  • The simple version split documents by page and gave clear grounding right away.
  • Its search combined vectors, metadata and regex filters.
  • The agentic version used a router and four tools: metadata search, exact match, document comparison and hybrid search.
  • Both setups used a reranker to reshuffle the top-k candidates and improve relevance.

In practice, the simple architecture proved more robust. It could be assembled quickly, its behavior was predictable, and the source of an answer was easier to trace. The agentic design looked stronger on paper but cost more time: two LLM calls, unstable chunking and more points of failure. Even after fixing some of the bugs, the team did not manage to fully run and tune the whole pipeline. For a hackathon with a hard deadline this is critical: extra complexity quickly eats the gains of a "smart" architecture.

Where things broke

Chunking was the main problem. The same splitting template worked differently on different pages, and small fragments with no meaning had to be simply glued onto neighboring pieces. In the simple setup, regexes also got in the way: they sped up pattern search, but easily missed relevant cases or produced false positives.

Grounding came up separately: at first the necessary links and metadata were not loaded properly; the team then fixed this, but accuracy dropped as grounding improved. It is a good illustration of how a retrieval system is rarely optimized on a single metric without side effects.

"Building a system like this in such a short time is practically impossible without coding agents."

The final results only confirmed this. The simple solution reached accuracy of 0.79 with grounding of 0.63 and showed stable, though imperfect, behavior. The more complex agentic version lost on accuracy in the preliminary stage and ran slower, and in the final it was not submitted at all because of API errors before the deadline.

The authors also warn about another trap: coding agents are useful for scaffolding and routine tasks, but on complex problems they may substitute real steps with stubs, "magic numbers" or narrow regex hacks that look like a solution but fail under real testing.

What this means

The write-up gives a clear picture of the real status of agentic RAG in 2026. On legal-document tasks, the winner is not the flashiest design but the one that keeps chunking, grounding, metadata and testing under control. For teams building AI search over internal knowledge bases, the conclusion is simple: first build reliable retrieval and measurability, and only then add routers, agents and complex orchestration.

Story

Key Questions Answered

How the hackathon worked?

The competition was run by EORA AI Applications and Services. Participants had to build a system that answers questions about the documents of the courts of the Dubai International Financial Centre (DIFC). The hackathon ran in two stages: from 11 to 19 March 2026, participants worked with 30 documents and 100 questions, and in the final, held from 20 to 22 March 2026, the volume grew to 300 documents and 900 questions. The prize pool was $32,000, and more than 300 people took part in total.

What are the key details on Two versions of the system?

The team built two architectures on the same stack: Qdrant as the vector database, LlamaIndex for indexes and LLM abstractions, and Unstructured for extracting text from PDFs while preserving structure. After that, the paths diverged. The first version was as practical as possible: page-based chunking with overlap, hybrid search, and filtering by metadata and regular expressions.

Where things broke?

Chunking was the main problem. The same splitting template worked differently on different pages, and small fragments with no meaning had to be simply glued onto neighboring pieces. In the simple setup, regexes also got in the way: they sped up pattern search, but easily missed relevant cases or produced false positives. Grounding came up separately: at first the necessary links and metadata were not loaded properly; the team then fixed this, but accuracy dropped as grounding improved.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…