GPT agents lose up to 11.3 points of accuracy when the simulator report sits in a file

A correct calculation by an agent does not guarantee a correct answer. A new paper from Hugging Face Daily Papers shows how this happens in a popular context-management scheme: the long tool output is saved to a file, and the agent looks for what it needs using a section index. On the USGS geochemical simulator PHREEQC, GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy in this mode. Claude Opus 4.6 and Claude Sonnet 4.6 show no noticeable loss.

A testbed where every number can be traced

The researchers chose PHREEQC for a reason. It is a widely used simulator, and its reports run up to 846.8 thousand characters. It is deterministic, so any printed value can be traced back to its source. The agents answered 158 multiple-choice questions, and the correctness of each answer was confirmed by execution. This setup separates two possible causes of error: the model ran the simulation incorrectly, or it read the result incorrectly.

The authors re-ran all 11 007 logged simulator calls and checked how often the agent obtains the needed value at all. For both GPT models, this share barely changed. They did not get worse at simulating. What changed is where they take their answer from.

Evidence substitution

The authors named this failure evidence substitution. When the report has to be found through an index, GPT models often skip it. They answer from their own output files, where numbers are written with higher precision, and rounding sends them to the neighboring answer option. The calculation is correct, but the chosen option is wrong.

The effect is not explained by the model calculating worse. With the simulation held fixed, retrieval still costs GPT-5.2 9.4 points. The picture for GPT-5.4 is more interesting: the net change is indistinguishable from zero. But that zero balance is the sum of two opposing effects. More frequent correct simulations add accuracy, while the loss from how the model uses evidence takes away about ten points. The headline metric hides the problem.

Key figures from the study:

• 158 tasks with execution-based verification;
• up to 846.8 thousand characters in a single report;
• 11 007 re-run simulator calls;
• −11.3 points for GPT-5.1 and −7.9 for GPT-5.2 when reading through the index;
• −9.4 points for GPT-5.2 with the simulation held fixed;
• about −10 points of hidden loss for GPT-5.4;
• +6.3 to +7.9 points for three GPT models when the authoritative output file is named.

What helps and what doesn't

The most effective fix is to say directly which output counts as the main one. Naming the authoritative file in the prompt raises accuracy by 6.3–7.9 points over ordinary retrieval for three GPT models. For GPT-5.1 and GPT-5.2, inserting the outputs into context works just as well, if they fit. Asking the model to "round correctly" had no effect: the model knows the rounding rules but does not know which source to trust.

For those building agent harnesses, the practical takeaway is simple: final accuracy alone is not enough to judge the decision to store output outside the context. You need to separate the quality of the tool call from the quality of how its result is used, as the authors did with logs. Otherwise, the seemingly neutral result for GPT-5.4 is easy to mistake for the absence of a problem.

If the report fits in context, it is simpler to insert it in full. If it does not, it makes sense to name the main file explicitly in the instructions and forbid the model from substituting intermediate or more precise copies. Finally, when choosing a model for agents that handle long tool output, it is worth testing exactly this scenario: in this test, Claude Opus 4.6 and Sonnet 4.6 handled file-based retrieval without a noticeable loss, while the GPT models did not.