Snorkel runs millions of agent simulations a month and saves 20% on sandboxes

Evaluating agents is not the same as evaluating a chatbot. The model doesn't just answer a question: it opens a terminal, installs packages, launches training runs, and breaks and fixes services. To understand how good it is, you need to give it a real workplace. That's the subject of a case study from Snorkel, a data lab that builds datasets and benchmarks for frontier labs and enterprises. Among other things, it contributes to Terminal-Bench.

Why an agent needs a realistic environment

Anyone who has run agent evals knows the problem. The agent failed a task, but why? Did the model fall short, or did it simply not have the right tool? If the environment is stripped down, the benchmark measures not the model's capabilities but the limits of your infrastructure.

Snorkel's tasks vary: coding, infrastructure, ML engineering. An ML agent needs a GPU to train a model. An infrastructure agent may need an entire Linux kernel and several containers. So the team runs simulations in Modal sandboxes, including VM sandboxes with multi-service systems and GPU workloads. A single provider for GPU and CPU simplifies both the infrastructure and cost accounting. According to Snorkel, Modal was the only place where they got GPU sandboxes at the scale they needed.

The scale: millions of sandboxes a month, with peaks in the tens of thousands per hour. This isn't a whim. When a new model comes out or an existing one is tuned, the customer wants to know within a day whether it fits their use case. That means quickly launching many parallel runs across a mix of public and private benchmarks.

Why agent resources are so hard to estimate

Previously, the team estimated CPU and memory for each sandbox in advance. It went badly in both directions. If you allocate too little, the agent crashes from resource shortages and the run is invalid: this isn't a model error but noise in the metric. If you play it safe, you pay for idle hardware.

Guessing is hard for two reasons. First, the load is bursty. The agent waits for an LLM response and consumption is nearly zero; then it runs, say, grep over a large repository, and there's a spike. Second, all the code in the sandbox is written by the model itself, and there are many benchmarks, so the load profile is unknown in advance.

The solution is bursting in Modal. A sandbox expands memory or CPU on the fly when needed. You pay for either actual consumption or the declared minimum, whichever is greater. This made it possible to request fewer resources without crashes. Snorkel reports savings of around 20% on its overall sandbox bill, and the number of runs that failed due to limits went down.

A caveat: for now, bursting is used by only 10–15% of Snorkel's sandboxes. The 20% figure is the company's own estimate, and it comes from a case study published by a vendor. Treat it as a guideline, not a guarantee for your workload.

Inference for benchmarks

The second half of the problem is the models themselves. For initial checks of new open-weight models, Snorkel uses a ready-made shared endpoint. If a customer needs a model they've fine-tuned, the team deploys it with custom weights on a dedicated endpoint. An important detail: all this is done without involving an infrastructure team. Spin up the model, run the benchmark, tear it down.

The plan is to also evaluate Modal for training, to close the "data → training → evaluation" loop on a single platform.

Takeaways

• If you build agent evals, separate "the agent couldn't" from "the environment didn't allow it." Log failure reasons and count resource-related failures separately.
• Don't allocate resources blindly, whether by the average or by the maximum. Capture peak CPU and memory on a sample of runs (Snorkel built such a chart from 5,000 sandboxes) and consider elastic limits.
• If agents need a GPU inside the sandbox, check that your provider can supply it at the required scale before you design the benchmark.
• If a customer expects results within a day, what matters is not average speed but peak parallelism.

Overall, this isn't a technological breakthrough but a careful engineering breakdown of what applied infrastructure for agent evaluation looks like. The main takeaway: a fair agent metric depends not only on the model but also on how correctly you've provisioned its resources.