Study: A Single Random Failure Makes AI Agents Abandon a Working Tool 11.6% of the Time

When an AI agent keeps its own work log and writes itself "notes" for the future, a hidden danger emerges: a single unlucky outcome can permanently poison its opinion of a perfectly good solution. That's exactly what the authors of a paper published on arXiv (2610.33029) tested, in a strictly pre-registered experiment — the protocol, code, and test data were frozen and timestamped (via Zenodo and a Bitcoin blockchain entry) before the actual run began, to rule out results being tailored to a desired conclusion.

How the experiment was designed

The researchers invented 256 fictional "worlds" — recurring work tasks like data exports or invoice mailing — in which an agent had access to two interchangeable tools. One tool was reliable (an 80–93% success rate over the last several hundred runs), the other noticeably worse (50–70%). In each world, a scenario was run in which the agent correctly chose the reliable tool — and the outcome of that choice was artificially assigned: either success, or a random, temporary failure unrelated to the quality of the decision.

After that run, the model — following the ReasoningBank method, widely used in agentic systems for self-learning — wrote a text "lesson" based on the outcome. Then a new, "clean" session of the same model was launched in the role of reader: it saw the full and accurate log of all past runs, knew the result of the most recent run, and was given one of several memory variants — either that specific lesson, or a lesson written after the opposite outcome, or a neutral fact unrelated to the task, or a direct (unsubstantiated) instruction to use the worse tool. In total, the researchers collected 90,084 responses across 11 models.

What the numbers showed

The headline finding: when memory contained a lesson written after a random failure of the reliable tool, the reading model chose the bad tool 11.6% of the time — even though the log plainly showed that tool performed much worse. By comparison, with a neutral prompt the error occurred only 0.4% of the time, and the same rate applied when memory held a lesson written after a success (rather than a failure) of that same reliable tool. In other words, the problem wasn't the failure itself, but how the model textually described and "generalized" it into a lesson.

The effect turned out to be stronger in worlds where the log didn't explicitly explain that failures are independent, random events. When the text clearly stated "failures are random and unrelated to each other," the model drew the wrong conclusion less often. The reverse case — a random lucky success for the unreliable tool — showed a weaker, though not zero, effect: statistically, the authors can only claim the harm is "probably small," not that it's entirely absent.

Of the 11 models, six clearly carried randomness forward into their subsequent decisions: Gemini 3.8 Flash, Grok 4.7, Qwen 3.8 Max, DeepSeek V4 Pro, Mistral Large 4, and Kimi K3. Five models — all three Claude versions (Opus 5.5, Sonnet 5.5, Haiku 5.5), plus GPT-6.1 Sol and GPT-6 Luna — scored below 10%, though even for them the effect didn't vanish entirely.

Can it be fixed?

The authors tried two simple patches to the prompt the model uses to write its lessons. Adding three sentences asking the model to "evaluate the quality of the decision, not just the outcome" lowered the share of wrong choices after a random failure from 11.6% to 7.5%. A single unified instruction covering both outcomes (rather than separate prompts for "success" and "failure"), with an explicit "no change needed" option, had a more noticeable effect — 3.1%. But neither patch brought the rate to zero across all 11 models simultaneously, meaning there's still no fully reliable fix.

For teams building agents with long-term memory and self-learning from their own logs, the takeaway is practical: mechanically turning "lessons" from single outcomes into behavioral rules is a vulnerability. Without an explicit distinction between the quality of a decision and the randomness of its outcome, an agent risks abandoning perfectly functional tools and processes because of one bad break — and will stubbornly cling to that false belief until someone manually rewrites its memory.