Agent strikes out its own old tool outputs and cuts input tokens by half

An agent that works through tools quickly accumulates history: logs, dumps, command outputs, file contents. The useful part of each such response is often an order of magnitude smaller than the response itself, yet all of it rides along in every subsequent request. The authors of an arXiv preprint propose handing the cleanup to the model itself. In their research case, the final prompt under this approach came to 231 951 tokens versus 912 492 with ordinary history storage. Total input volume fell by 50%, and the estimated API cost dropped to $1.28–1.44 instead of roughly $4.38.

How the forgetting works

The mechanics are simple. The acting model selects previously received tool-call results and replaces each one with a short note at its original position in the conversation. The order of messages and the structure of the history stay intact. The exact original is not deleted. It goes into an archive, and the agent can bring it back with an explicit command if it needs it again.

The approach is implemented as a Python harness that gives the agent two actions: batch archiving and restoring. The model does not need to be fine-tuned for the task. There is also a safeguard: user instructions and assistant messages are exempt from these operations. Only observations can be forgotten, not what the human asked for or what the agent itself said.

What the experiments showed

The main case is a research one: debugging with OpenTelemetry, followed by an unrelated implementation task. It is exactly the kind of noisy trajectory where bulky outputs matter for a couple of steps and then only get in the way.

• Final prompt: 231 951 tokens versus 912 492 with the history kept intact.
• Total input tokens: 50% fewer.
• Estimated API cost: $1.28–1.44 versus roughly $4.38.
• Behavioral test: both variants passed the main two-case oracle.
• Follow-up evaluation: neither variant fully satisfied it.
• Costs: more requests and 17% more execution time.

Saving tokens is not the same as saving across the board. The agent spends steps on the curation process itself, and that shows up in the run time.

Where the method did not work

The authors candidly present counterexamples. On a contrast pair involving application development, there were no savings in either context or cost. An earlier experiment with a continued session showed lower quality by manual evaluation, even though the context shrank. The sample is small, the cases are exploratory, and there are no statistics here. The work does not prove that the technique suits every workload. It shows that the gain depends heavily on the nature of the workload, and the authors name this dependence as the central factor for reversible context management.

What to try in practice

If you have an agent with long debugging sessions, log analysis, or searches over large result sets, the idea is worth an experiment. Start with observations that are easy to restore: test output, traces, dumps. Leave instructions and assistant messages untouched, as in the original design. Measure not only tokens but also the number of requests, run time, and result quality on your own tasks, because the savings may turn out to be illusory. For tasks where every code or configuration fragment is needed again and again, archiving may yield nothing. Reversibility here is insurance, not a guarantee: if the agent forgot the wrong thing, it has to notice that and request it back.