Flightscape agent cuts token use by up to 71% and responds up to 80% faster on 23 MB of JSON
When an AI agent calls a tool that returns tens of megabytes of JSON, the entire file lands in the model's context, and every follow-up question gets slower and more expensive. The Flightscape team moved the heavy data into S3 and now gives the model a short digest with a link to the full result. On 23 MB of JSON, token consumption fell by 71%, and the average response time dropped from 15.4 to 3.1 seconds. The same approach can work for any agent with heavy tools.
AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Imagine that every time you ask a colleague a question, the reply comes with a 23-megabyte spreadsheet attached, and this happens ten times in a row. That is roughly how the AI agent at Flightscape, a platform provider for airlines, used to work. The data is JSON (a structured text format) files of 20+ MB, there are more than forty tools, and each tool returned its slice of that data straight into the model's context, which is its "working memory."
Tokens (the units of text you pay for) were burned in huge quantities, and the average response took more than 15 seconds. After a redesign, on a 23 MB test file, token consumption fell by up to 71% and latency (the wait for a response) dropped by up to 80%. The engineers described the process on the AWS blog, and it is useful to anyone building agents on top of large datasets.
What the problem was
The agent checks flight crew training assignments against business rules. It takes three documents as input: a request to the schedule optimizer, the optimizer's response, and a validator report. The agent is built on LangGraph (LangChain's open-source framework) and runs on Amazon Bedrock AgentCore, while session memory is stored in AgentCore Memory.
Functionally, everything worked, but the heavy data sat too close to the conversation. Every tool response went into the context in full, a single response could take up more than 80 000 tokens, and in a multi-turn conversation that piled up. On large datasets, the context ran out before the agent could finish working out an answer.
Three techniques that fixed it
First: session memory holds only references. The request, the response, the validation report, and derived data go to Amazon S3 (cloud file storage). AgentCore Memory stores messages and object keys. At the start of a turn, the hydrate_state() function restores the working state, and at the end, dehydrate_state() offloads the heavy data back to storage. The key is computed as a SHA-256 hash of the content, so identical data is never uploaded twice. A small in-process cache sits alongside, so a single call doesn't hit S3 twice.
Second: long lists are served in pages. If a tool returns a list longer than a set threshold, the full result is saved to S3. The model gets a compact response: the total number of records, a small sample, and a handle, which is a link to the full result. If it needs more, the model calls get_result_page_tool with the handle, an offset, and a limit, and fetches the chunk it needs. It's like asking a librarian for the table of contents and a couple of examples instead of demanding that the entire archive be carried to your desk.
Third: large dictionaries are replaced with a description. Here the structure matters more than the content. For lists, the item count and five examples remain; for nested dictionaries, the number of keys remains, and simple values stay unchanged. The full version again goes to storage with a handle. There is also a safety net: if the serialized response still exceeds the character limit, it is replaced with a description.
The results in numbers
The team ran the same questions against the same dataset on the old and new versions of the agent and compared metrics in AgentCore Observability.
- A 5-question conversation on 23 MB of data: tokens fell from about 289 500 to 82 900 (−71%), and average trace latency from 15.4 to 3.1 seconds (almost −80%).
- 100 test questions: tokens went from 23.6 million to 8.5 million (−64%), and average latency from 7 930 to 2 461 ms (−69%).
- The heaviest query, a list of valid crews across all validation categories: input tokens fell from about 32 000 to 8 000, output tokens from 2 800 to 477, and turn time from 165 to 9.9 seconds (−94%).
The authors also note a side effect: context growth became predictable, and in long conversations the agent no longer slows down with each new question.
Caveats and takeaways
The figures come from one company's own benchmark, on one type of data. Your gains will depend on how your tools' responses are structured. If every response is small, this scheme will only add complexity.
There is a cost, too. You take on a dependency on S3 and extra calls to storage. The model now decides for itself when to load the next page, and a sample of the first few records can mislead it. For questions that require all the records, the number of calls to the pagination tool will grow. In addition, the material gives two values for the heaviest turn: 9.9 seconds in the table and about 15 seconds on the trace timeline, so the absolute numbers should be treated with caution.
The practical takeaway is simple. Don't give the model what it doesn't need to read in full: keep large objects outside, leave references in the conversation, return a table of contents with examples from your tools, and give the model a way to read the rest of what it needs. All of this takes just Python 3.12+, boto3, langchain-core, and langgraph, and the authors have published the code in a separate GitHub repository. If your agent works with large reports, logs, or CRM exports, start by measuring how many tokens go to tool responses in a single turn. Chances are, that is where most of your bill is hiding.
- Nov 18, 2026TokenEconomizer plugin cuts AI coding agent costs by 18.4–30.9% without changing the model
- Nov 18, 2026Five AI Models Tried to Fix the Same Bug: Two Succeeded, Three Broke More Than They Fixed
- Nov 18, 2026Qwen on two DGX Spark units runs 34–59% faster in tokens per second and outpaces GLM
- Nov 18, 2026Developer gives a web AI chat hands: an MCP server with 14 tools and 21,000 lines of code
Key Questions Answered
What the problem was?
The agent checks flight crew training assignments against business rules. It takes three documents as input: a request to the schedule optimizer, the optimizer's response, and a validator report. The agent is built on LangGraph (LangChain's open-source framework) and runs on Amazon Bedrock AgentCore, while session memory is stored in AgentCore Memory. Functionally, everything worked, but the heavy data sat too close to the conversation.
What are the key details on Three techniques that fixed it?
First: session memory holds only references. The request, the response, the validation report, and derived data go to Amazon S3 (cloud file storage). AgentCore Memory stores messages and object keys. At the start of a turn, the hydrate_state() function restores the working state, and at the end, dehydrate_state() offloads the heavy data back to storage. The key is computed as a SHA-256 hash of the content, so identical data is never uploaded twice.
What are the key details on The results in numbers?
The team ran the same questions against the same dataset on the old and new versions of the agent and compared metrics in AgentCore Observability. A 5-question conversation on 23 MB of data: tokens fell from about 289 500 to 82 900 (−71%), and average trace latency from 15.4 to 3.1 seconds (almost −80%). 100 test questions: tokens went from 23.6 million to 8.5 million (−64%), and average latency from 7 930 to 2 461 ms (−69%).
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.