Legora runs 2,000 parallel runs of its legal agent in Modal sandboxes

Testing a chat assistant is simple: ask a question, get an answer, compare it to a reference. That doesn't work for an agent that spends several hours digging through thousands of documents, editing files and querying databases. The legal platform Legora ran into this in early 2026. It replaced its assistant with a full-fledged agent and rebuilt its evaluation system from scratch.

Why a laptop is no longer enough

The old evals were designed for small tasks: a few documents in the environment, everything running locally. The new agent works inside a realistic project with thousands of documents and gigabytes of data. A typical experiment now looks like this: 100 cases, 4 harness variants (the scaffolding around the model that makes up the agent) and 5 runs for each. That makes 2,000 runs, and all of them have to go at the same time. Legora engineer Eirik Dræge Steen says it plainly: even 10–20 parallel runs would overload a laptop.

The second problem is subtler: environment fidelity. Legora's agent covers the entire product: a headless DOCX editor, dedicated Turbopuffer instances, PDF conversion, Postgres and an authorization database, blob storage and dynamic sandboxes for code execution. To measure the effect of a database migration or a change in the code sandbox, every run needs exactly that version of the whole application. Mocks don't work here.

How it works: one sandbox, one run

Each run gets its own Modal Sandbox with a custom image and file system. The entire application runs inside, pinned to a specific commit. Engineers launch experiments from their laptops, while all the computation happens in the cloud. Legora compared several cloud sandbox providers and chose Modal for its scaling, low startup latency and support for complex environments: both the application and the image are very large.

A few techniques worth remembering:

• World state in Volumes. Large document collections are available in all sandboxes. A snapshot includes documents, the database, permissions and blobs, so repeated runs start from identical data, and the shared part is deduplicated.
• Hourly warm-up. Building the huge image initially slowed runs down. Now the pipeline builds the image and warms up sandboxes once an hour, so resetting to a commit comes down to the delta since the last build rather than a full application build.
• Secrets. A separate job syncs values from the key vault into Modal Secrets once an hour. Frequent rotation, minimal permissions, dependency-age requirements and a domain allowlist are also used.

A separate story is the move from gVisor to VM Sandboxes. Legora's application consists of Docker services, and under gVisor the team had to build workarounds: host-mode networking with its own DNS resolution and running code sandboxes as bare processes because FUSE would not mount. In a virtual machine with a real Linux kernel, Docker behaves as it does on an ordinary host, and the team removed all of these hacks.

What it delivered, and the catch

The platform has become the main tool for evaluating agent quality. It runs the internal BAR benchmark, a set of realistic long legal tasks, and compares foundation models on quality, cost and latency. The choice of model in production and the design of the harness depend on these results.

What is more interesting is that the evals were wrapped in a CLI and a skill, so agents now create cases, launch experiments and analyze results on their own. Most engineers work with evals through agents. If you give an agent a metric, experiment tools and a budget, it will keep tweaking the harness until the metric improves. This is how Legora optimized tool search, PowerPoint generation and the reproduction of scientific papers. One outcome is a 30% reduction in agent cost thanks to smarter context compaction and a custom implementation of tool search.

The catch is that this is a Modal blog post, which means a vendor case study. There are no figures on startup latency, the price of 2,000 runs or the share of flaky runs. For the "30%", it doesn't say on which benchmark or whether quality was preserved. The approach itself isn't tied to one platform, though: it can be replicated elsewhere, provided that platform handles heavy images and offers real Docker.

Takeaways:

• If your agent touches databases, files and external services, test it in a full copy of the product, not on mocks.
• Pin not only the code but also the data: a snapshot of documents, permissions and database state makes runs comparable.
• Warm up images on a schedule so that resetting to a commit doesn't require a full build.
• Run several runs per case: the agent is non-deterministic, and Legora uses 5.
• Wrap evals in a CLI so agents can launch experiments and improve the harness on their own within a budget.