ASTC Compresses Agent Context Without Training, Cutting Attention Pairs by 50.68%

Long agentic workflows run into context growth. Each step adds system instructions, earlier decisions, tool output, code, and environment feedback to the next model call. Processing the full context is expensive, and the facts that matter have to compete with stale clutter. The authors of an arXiv paper propose Agentic Soft-Token Compression (ASTC), a method that compresses this context inside the model itself and does not require training a separate compressor. According to the authors, task quality stays at the level of the uncompressed model or higher, while the number of Attention Pairs (the "query-key" pairs that attention computes) over the whole run drops to 50.68%.

Why ordinary compression breaks agents

The authors distinguish agent context compression from ordinary summarization. An agent's history mixes text that can be shrunk harmlessly with exact values: paths, identifiers, checksums, API arguments. Changing a single character in such a value makes the next action invalid. On top of that, information that is unimportant now may become decisive several steps later, so a one-time importance estimate is unreliable.

Existing methods make the decision before the model processes the context. Hard methods drop or rewrite text and risk losing exact values. Soft methods store continuous representations, but they need separately trained encoders, and their objective may diverge from what the actual model has already absorbed and what it will need later. The authors themselves first tried to train a built-in compressor made of attention and MLP blocks through on-policy distillation. Preparation was expensive and the result was weak, and adding compression layers did not fix the situation.

An analysis of trajectories from real user sessions showed that tool observations take up the most space. The systems named in the paper include Fable 5, GPT-5.6 Sol, and Kimi K3. Tool outputs therefore became the main target of compression.

How ASTC uses GDN state

The method is designed for hybrid models in which Gated Delta Network (a recurrent layer with associative memory) alternates with full attention. A update to such a layer yields a novelty signal: if the input value is already well represented in the state, the correction is small, while new or contradictory information changes the memory more. In effect, the model itself shows what it has already absorbed.

Early layers read the entire new block. ASTC then selects positions that are both predictable from the previous state and still readable from the state at the end of the observation. Recognized exact values and the action boundary are protected by a mask and remain ordinary tokens. The selected positions are split into causal groups, outliers are removed from them, and the remaining hidden states are averaged into a single soft token in the model's space. Single-position groups and groups with excessive distortion are rejected, and those positions stay as the original tokens.

The cache after the compression boundary holds a shorter, mixed sequence. It is reused during decoding and in later agent steps, so the gain is not limited to a single prefill. The compression also has to pay for itself: the attention it saves must exceed the cost of selection and pooling.

What the measurements showed and where the limits are

The reported results: across a range of long tasks, ASTC keeps quality close to the uncompressed model or higher and reduces end-to-end Attention Pairs to 50.68%. If the protection of exact values is removed, accuracy drops sharply. The criterion also works beyond tool observations: when the entire history is compressed, the uncompressed model's accuracy is reached with 24.52% fewer Decode KV accesses. The gain is also shown on long-context tasks and code repair. Measured in Attention Pairs per successful task, the method beats trained compressors by a factor of 15.4–103.7 while also delivering better accuracy.

There are several limitations. The method relies on hybrid GDN models, so it is not directly applicable to pure transformers. Recognition of exact values may miss some fields. The theoretical estimates, including the conditions under which soft tokens distort prediction less than hard tokens of equal length, depend on assumptions that the selection thresholds do not guarantee. The Attention Pairs metric counts attention computations, not time and money on real hardware, so the final savings need to be verified on your own stack.

The practical takeaway for teams building long-running agents on hybrid architectures: compression can be drawn from the model's internal signals, without spending resources on a separate compressor.