Context Compaction Erases a Third of Anti-Cheat Rules in ProgramBench

Long agentic tasks increasingly do not fit into the context window, so the agent harness has to free up space: compress the history, drop the unneeded parts, and keep working from a short summary. This technique is called compaction, and almost every coding agent has it. It has rarely been tested systematically on frontier models and realistic tasks. Earlier studies ran small open models on toy or synthetic trajectories. Jeanne Salle's group, together with four co-authors, filled that gap and got a result with a double bottom. Benchmarks barely notice compaction, but real-world work does.

How the experiment was set up

The authors fixed the OpenCode agent harness and implemented five compaction strategies inside it. They then ran three frontier models through three long-horizon benchmarks. Two of the three models have a 1M-token context; the third is GPT-5.3-Codex, which has the smallest window in the sample. The authors varied the threshold at which compaction triggers.

• Harness: OpenCode, identical for all runs.
• Strategies: five compaction variants. The baseline is self-summarization, where the model retells its own history.
• Models: two with a 1M window, plus GPT-5.3-Codex.
• Benchmarks: PaperBench, ProgramBench, PostTrainBench.
• Additional dataset: real user sessions from SWE-chat.

Where compaction is safe and where it is not

For the models with a 1M window, the compaction threshold has no noticeable effect on results. GPT-5.3-Codex is a different story: on PaperBench, quality drops at low thresholds, while on PostTrainBench the effect is weaker and less clear-cut. The logic is simple: the smaller the window, the more often and more aggressively the harness has to compress, and the more losses accumulate.

The inventive methods did not pay off. None beat plain self-summarization. Approaches that discard more content cost some models up to 20 points and cost others nothing. In other words, sensitivity to aggressive compaction depends on the model, and you cannot guess it in advance.

The most unpleasant finding concerns constraints. On ProgramBench, the very first summary loses on average about a third of the task's anti-cheat rules, and the share of lost rules grows monotonically with each subsequent compaction. An agent that has forgotten it must not peek or bypass the check formally keeps working, but now under different rules. And it does not report this.

On real SWE-chat sessions, the gap is even wider. Instructions about how the agent should work and what it should not do are dropped roughly 6 times more often than specifications of what to build. The model remembers the goal well and remembers the manner of work poorly: style, prohibitions, agreements with the user.

Why benchmarks mislead

The authors call current benchmarks "compaction-friendly." In them, the goal is fixed and written into the task statement, and the agent's progress sits on disk: code, files, logs. So almost everything that compaction removed from the context can be recovered by rereading the repository. Only what is recorded nowhere else cannot be recovered: constraints, things said verbally in the dialogue, the reasons behind past decisions. A live session has plenty of such things; a benchmark has almost none. Hence the conclusion: green numbers on tests do not guarantee that an agent will not lose requirements in a real project.

For those building agents, the takeaways are practical. Do not overcomplicate compaction: default self-summarization remains a strong baseline, and saving tokens through aggressive discarding can cost up to 20 points of quality. Rules and prohibitions should live outside the dialogue history: in the system prompt, in an instructions file in the repository, or in a pinned block that compaction does not touch.

After each compaction, it makes sense to automatically check the summary against a list of constraints. And when choosing a model, the context window matters not in itself: the smaller it is, the more often compaction triggers and the higher the risk of silently losing requirements. Finally, agents should be evaluated on scenarios where part of the conditions exists only in the dialogue; otherwise the benchmark will measure memory of the goal, not working discipline.