OrderBench: Why a JSON Schema Doesn't Protect LLM Agents from Dangerous Orders

In July 2026, researchers introduced OrderBench, a deterministic benchmark for restaurant-ordering LLM agents. It showed that even when a model returns JSON with 100% schema compliance, about 20% of the strongest model's orders are still semantically wrong. The paper's conclusion is simple: structured output eliminates parsing errors, but it does not guarantee that a transaction is safe and accurate.

What Is OrderBench

OrderBench is a deterministic test that separately measures six properties of an agent's response: syntactic correctness, schema validity, correctness of the status decision, exact item semantics, preservation of constraints, and unsafe order confirmations. The motivation is that LLMs increasingly act as a "transaction compiler": the user states an intent in plain words, the model produces a structured object, and an API executes it. JSON Schema and providers' structured-output modes remove a whole class of parsing errors, but on their own they do not determine whether the transaction the agent is about to execute is correct and safe.

What the Experiment Showed

The authors made 2,400 calls through the Nebius Token Factory platform to four open models in two modes: a plain prompt and an enforced JSON schema. The main result: a schema-valid response can still carry major semantic errors. For the strongest model, both modes yield 100% schema validity, but semantic success holds at about 80%. In other words, one in five orders is formally correct but wrong in substance. For weaker models, the share of schema-valid but unsafe order confirmations is measured in double-digit percentages.

• Benchmark: OrderBench, deterministic, for restaurant orders
• Scale: 2,400 calls to four open models
• Platform: Nebius Token Factory
• Modes: prompt-only and enforced JSON schema
• Key figure: 100% schema validity with ~80% semantic accuracy for the best model

Why a Schema Doesn't Save You

Schema validity and order correctness are different things, and this is the paper's central finding. A JSON schema guarantees only the shape of an object: the required fields are present and the types match. It does not check that the agent picked the right item, preserved constraints such as allergies, substitutions and quantities, or avoided confirming an order that should have been rejected. Such an unsafe confirmation is completely invisible to a schema validator: the object passes the check and goes to the API for execution, even though it is wrong or dangerous in meaning.

"Structured output is a necessary interface layer, but not a

substitute for domain verification and fail-closed execution," the study's abstract on arXiv states.

What This Means

Structured output solves the parsing problem, not the trust problem. The paper explicitly advises developers of agentic systems not to treat a schema as a guarantee. They need a second layer: domain verification and a fail-closed principle, where the agent rejects an order rather than confirming it at any doubt. Otherwise, neatly formatted JSON will keep executing wrong transactions, reliably and unnoticed.

Frequently Asked Questions

What is OrderBench?

OrderBench is a deterministic benchmark for restaurant-ordering agents, introduced in July 2026. It separately evaluates six properties of a response, including unsafe order confirmations that a JSON schema validator does not catch.

Does a JSON schema guarantee that an order is correct?

No. According to the study, for the strongest model both modes yield 100% schema validity, yet semantic accuracy is about 80%, and for weaker models a double-digit percentage of orders passes the schema while remaining unsafe.

What does fail-closed mean for an agent?

Fail-closed is a principle under which an agent rejects an action, rather than confirming it, whenever anything is ambiguous. The OrderBench authors name it, together with domain verification, as a necessary complement to structured output, since a JSON schema by itself does not filter out unsafe orders.