Schema Passed, Invoice Total Off by 10x: How to Validate LLM Structured Output

Here is a four-line invoice: number 2026-0417, payer Northwind Traders, "Total due: EUR 1,250.00", payable within 30 days. You ask the model to extract the number, the currency, the amount in cents, and the line from which it took the amount. The answer comes back: amount 12,500, currency EUR, quote "Total due: EUR 1,250.00".

The JSON is valid, all keys are present, the types are right, EUR is in the allowed list, and the quote really does appear in the invoice. But 12,500 cents is EUR 125, while the invoice says 1,250. That is a tenfold error, and no parser will notice it: 12500 is a perfectly good integer. The only way to catch it is to compare against the source text, and you have to write that comparison yourself.

Machine Learning Mastery has published a walkthrough showing how to build a three-layer defense. Structured output has long been a standard feature: OpenAI's Responses API accepts a JSON schema or a Pydantic model, and vLLM does the same on a local server via xgrammar. Most tutorials end at the moment "the JSON parsed." The author argues that this is exactly where the real work begins.

The contract and cross-checking against the source

The first layer is a contract owned by the application, not by the prompt. In the example it is an Invoice model with four fields. Two configuration parameters do most of the work. strict=True disables Pydantic's implicit coercions: the string "125000" in an integer field raises an int_type error instead of silently turning into a number. The same goes for the boolean True, which would otherwise become 1. The extra="forbid" parameter turns an extra key into an error rather than ignored junk.

One more detail: three fields allow null, but none has a default value. The model can honestly say "the invoice doesn't have this," but it cannot stay silent, and it should not invent a number just to avoid writing null. Money is stored in cents as an int, so that 1,250.00 doesn't turn into 1249.9999.

Next, the accept() function takes four steps. First, parsing and shape validation, the only step that can return shape_error. Then a check that the quote appears in the source verbatim. Then the invoice number and currency are checked against the text. Finally, the amount is compared with the digits from the quote: "1,250.00" without separators gives 125000, the model sent 12500, and the object is rejected. There are four outcomes: accepted, needs_review, rejected, and shape_error. If the invoice states no amount, the result is needs_review, meaning a human decides, not an invented zero.

When to retry and when not to

The second layer is retries with a strict policy. Only shape_error is worth retrying: the model can be asked to fix the shape by being given the list of errors. If the answer is rejected, a retry is pointless, because the contents of the invoice will not change. A provider refusal or a configuration error also stops the loop immediately. The maximum is three attempts: the first one and no more than two regenerations.

Feedback is limited to five reasons so that the prompt doesn't turn into a transcript of every failure. The schema and the source text do not change between attempts. According to the article, the SDK's own hidden transport retries must also be kept under control, otherwise the actual number of requests will exceed what you budgeted for.

Constrained decoding: useful, but not a cure-all

The third layer is constrained decoding. The generator can physically emit only tokens that keep the response within the schema, so a whole class of shape errors never reaches validation. But a schema cannot express the rule "the cents must match the quote." A decoder that follows the schema perfectly will still output that same 12,500. A valid object is a pass to the next check, not the final result.

There are limitations too. The example is a teaching one: two currencies, a simple digit check. Real formats require code that understands your decimal separators. If invoices arrive as scans, check the recognized text first: a wrong digit in the input will pass every check as long as the object is internally consistent.

The practical takeaway is simple. Don't trust a "green" schema validation: keep a quote as evidence in the response and check the number against it. Use retries only where a repeat can help, and set a hard limit. Turn on constrained decoding as savings on shape errors, but not as a replacement for your own checks. Every failure path deserves tests with a mock client, as the author does.