Stop the waste: compact TOON instead of inefficient JSON in LLM pipelines
JSON wastes tokens in LLM pipelines because field names repeat. TOON (Token-Oriented Object Notation) is a compact format that reduces token usage and speeds up
Language models work with tokens, and each token has a cost. If you regularly pass structured data into LLM pipelines, you probably use JSON. But JSON can be inefficient: field names repeat in every record, wasting precious tokens. TOON is an alternative that solves this problem.
Why JSON Wastes Tokens
JSON is a universal format for structured data, but when working with LLM, its drawbacks become critical. Imagine an array of 100 customer records, each containing fields: id, name, email, status. In JSON, every record repeats all these field names, even though they could be specified once. Here's a simple example: JSON requires repeating the field names in every object. TOON solves this differently — it defines the structure once, then transmits only the values. This can give 30–50% token reduction for large datasets.
Where TOON Is Most Useful
TOON is most effective when working with homogeneous data:
- Arrays of records — CRM tables, logs, database query results
- Support tickets — all tickets have the same structure (author, subject, status)
- Product catalogs — SKU, price, description, availability repeat in each item
- Agent memory — interaction histories, where each record follows a single template
Implementation Strategy
You don't need to switch to TOON entirely. The optimal approach: store data in JSON (familiar, scalable), convert to TOON before sending to LLM, get the model's response, save in the original format. This requires minimal changes but delivers significant token savings.
Key tip: before switching to TOON, benchmark your pipeline.
Compare token count, latency, response quality, and final cost. Savings can be substantial or marginal — it all depends on your data.
What This Means
TOON is not a revolution, but a practical tool for optimizing LLM expenses. If you work with large volumes of structured data, switching to a compact format can reduce bills by 20–40%. Start by testing on a single pipeline, measure the real result, then scale the approach. In the era of expensive models, every token saved counts.