OrderBench: почему JSON-схема не спасает LLM-агентов от опасных заказов

In July 2026, researchers introduced OrderBench — a deterministic benchmark for restaurant LLM agents that showed: even when a model outputs JSON with 100% schema compliance, about 20% of the strongest model's orders remain semantically incorrect. The paper's conclusion is simple: structured output eliminates parsing errors, but it does not guarantee that a transaction is safe and accurate.

What OrderBench is

OrderBench is a deterministic test that separately measures six properties of an agent's response: syntactic correctness, schema validity, correct status decisions, accurate item-level semantics, constraint preservation, and unsafe order confirmations. The motivation is that LLMs increasingly function as a "transaction compiler": the user states an intent in ordinary words, the model produces a structured object, and an API executes it. JSON Schema and provider structured-output modes eliminate an entire class of parsing errors — but on their own they do not determine whether the transaction the agent is about to execute is correct and safe.

What the experiment showed

The authors made 2400 calls through the Nebius Token Factory platform to four open models in two modes — a plain prompt and an enforced JSON schema. The key result: a schema-valid response can still carry major semantic errors. For the strongest model, both modes yield 100% schema validity, but semantic success holds at around 80% — meaning every fifth order is formally correct yet substantively wrong. For weaker models, the share of schema-valid but unsafe order confirmations runs into double-digit percentages.

• Benchmark: OrderBench, deterministic, for restaurant orders
• Volume: 2400 calls to four open models
• Platform: Nebius Token Factory
• Modes: prompt-only and enforced JSON schema
• Key figure: 100% schema validity at ~80% semantic accuracy for the best model

Why the schema doesn't save you

Schema validity and order correctness are different things, and this is the paper's central finding. A JSON schema only guarantees the shape of an object: the required fields are present and the types match. It does not verify that the agent picked the right item, preserved constraints — allergies, substitutions, quantity — or that it didn't confirm an order that should have been rejected. Such an unsafe confirmation stays completely invisible to a schema validator: the object passes validation and moves on to the API for execution, even though it is semantically wrong or dangerous.

"Structured output is a necessary interface layer, but not a

substitute for domain verification and fail-closed execution," the study's abstract on arXiv states.

What this means

Structured output solves the parsing problem, but not the trust problem. For developers of agentic systems, the paper's direct advice is not to rely on the schema as a guarantee: a second layer is needed — domain verification and a fail-closed principle, where the agent rejects an order rather than confirming it whenever there is any doubt. Otherwise, a neatly formatted JSON object will keep executing incorrect transactions smoothly and unnoticed.

Frequently asked questions

What is OrderBench?

OrderBench is a deterministic benchmark for restaurant-ordering agents, introduced in July 2026. It separately evaluates six properties of a response, including unsafe order confirmations that a JSON schema validator does not catch.

Does a JSON schema guarantee order correctness?

No. According to the study, for the strongest model both modes yield 100% schema validity, but semantic accuracy is around 80%, while for weaker models a double-digit percentage of orders pass the schema while remaining unsafe.

What does fail-closed mean for an agent?

Fail-closed is a principle under which an agent rejects an action rather than confirming it whenever there is any ambiguity. The authors of OrderBench call it, together with domain verification, a mandatory complement to structured output, since the JSON schema itself does not filter out unsafe orders.