Legora raised legal citation accuracy from 70% to 99% without access to the data

The Legora team faces an awkward but increasingly familiar problem: its legal agent works with confidential documents, while developers cannot simply open a trace and see where the model took a wrong turn. The data belongs to law firms' clients. That means the usual cycle of "find a bad answer, read the query, fix the prompt" does not work here.

Legora approaches evaluation differently. Engineers and lawyers create test environments resembling real work: rooms containing thousands of files, tasks involving finding clauses in contracts, preparing drafts, and checking citations. These are not perfect copies of client matters, but they provide a sufficiently realistic proving ground for catching recurring errors before release.

How to evaluate what cannot be seen

Imagine an agent assigned to review 10,000 contracts and compile a table containing the term, termination rights, penalties, and a link to the source passage. In a real project, the developer is not allowed to read either the contracts or the agent's answers. But the developer can assemble an artificial dataset with the same structure: long documents, contradictory appendices, missing fields, and several phrasings of the same clause.

The team then defines the correct answers in advance and measures individual stages: whether the right document was found, whether the clause was extracted, whether the conclusion matches the source, and whether the citation points to the correct location. In an interview, Legora CTO Jacob Lauritzen describes a dedicated team of engineers and lawyers that builds such environments. They resemble game "worlds" used to train agents: the system receives a realistic task without touching a client's private matter.

When a failure does occur on real data, Legora tries not to move the original text outside the protected environment. The team uses anonymized or paraphrased examples and reduces problems to 10–20 types. For example, the agent may select the wrong version of a contract, miss an exception to a rule, or provide a citation that looks convincing but does not support the conclusion. That class of failure can then be reproduced on a safe dataset.

Metrics matter more than a polished demo

The citation example is particularly revealing. According to Lauritzen, in one test the model correctly cited sources in about 70% of cases. After another update to the underlying model, the figure rose to 99%, even though Legora had not changed its own system. That is good news for the product, but also a warning: agent quality depends on more than your code. A model provider can suddenly fix a weakness—or break something that worked yesterday.

That is why one overall score is not enough. Search, fact extraction, citation, long-document processing, and table completion require separate evaluations. A new model should not be released merely because it answers a handful of manual questions better. It should be run against a fixed task set and compared with the previous version across every critical scenario.

What ordinary products can adopt

Legora's approach is useful beyond legal work. A banking assistant, medical agent, or enterprise search system faces the same boundary: the product must improve, but raw data cannot be handed to developers. The practical formula has four parts: describe real workflows, build safe analogues, label the expected results, and maintain a catalog of recurring failures.

The limitation is obvious: a simulation is always less rich than reality. It will not expose an unusual phrase, a strange archive structure, or an error that appears only in a particular combination of documents. Synthetic tests therefore do not replace protected user feedback and aggregated product metrics. They provide something different: a reproducible environment in which the model, search system, and interface can be changed without accessing other people's secrets.

The main lesson for developers is simple: when traces are private, observability does not disappear—it must be designed. Start not by logging everything, but by listing the decisions for which the agent is responsible and the measurable signs of a correct result. Privacy then stops being an excuse for missing evals and becomes a constraint around which the engineering system is built.