Anthropic: Claude with 413 note files earned 1 badge in Pokémon, but with a handful of files it earned 4
An agent that works on a project for several days has to remember the past, but per-step notes quickly go stale and get in the way. In Pokémon Red, the older Claude Sonnet 3.7 created 413 files in 225 folders over 10,000 steps and earned one badge. Claude Fable 5 kept a few files with strategies and verified rules, and earned four. It is like keeping a note for every commit instead of a short README. The takeaway: what an agent chooses to remember matters more than how much it stores.
AI-processed from The Decoder; edited by Hamidun News
An agent that works on a project for several days starts each session from a blank slate, so it gets a folder for notes: it writes there itself and reads the notes later. The problem is that notes go stale quickly. Imagine a repository with a separate note for every commit, half of them long since wrong, and a newcomer forced to read them all.
That is how Claude behaved in an Anthropic test. In Pokémon Red, the version that wrote down everything created 413 files in 225 folders and earned one badge. The version with a few files earned four.
The study's conclusion: the quality of an agent's memory depends not on volume but on the ability to decide what not to save.
An agent that records everything gets lost
The main difficulty is selection. At the moment of writing, the agent does not know what it will need later, so every note is a bet on the future. The older Claude Sonnet 3.7 treated everything as important. Over roughly 10,000 game steps it created 413 files in 225 folders, often starting a separate file for every battle and every item it found. The game state changes constantly, so such an archive quickly becomes outdated. At times the model did not play at all and only tidied its own notes. The result was one badge.
The newer Claude Fable 5 got by with a few files. It did not record what is already visible on screen, and saved only plans, strategies, and rules it had verified itself. The result was four badges. One important caveat: the comparison rests on a single run per model, so it is more an illustration than statistics.
Anthropic's practical advice is to explain to the model why it needs memory and let it decide how to use it. In one benchmark, rigid instructions such as "write down only mistakes" ate up most of the advantage of stronger models. Older models, by contrast, benefited from clear rules, such as a requirement to read the notes before every answer. So the memory prompt should be tuned to the class of model.
Dreaming: checking notes "in sleep"
The second problem is wrong notes that the agent then trusts blindly. In the Pokémon test, Claude wrote down incorrectly which side of a mountain a landmark was on. Relying on that entry, it fell into the same trap three times, eventually wanted to give up, and did not reach its goal. A similar thing happened with a wrong note about the location of a Pokémon Center: because of it, Claude kept falling off a ledge.
For such cases, Anthropic proposes a process called Dreaming, which builds on ideas from researchers at Letta and UC Berkeley. Just as the brain processes experience during sleep, Claude reviews old transcripts in the pause between sessions and checks its notes against them. After such a pass, the agent found the error, chose the correct route, and made it through Mt. Moon to Cerulean City. It worked only with the corrected note.
Dreaming can do more: it can also organize large knowledge bases. Claude built a linked wiki from roughly 350,000 internal documents of Stainless, a software company that Anthropic acquired. The system creates new pages, fills gaps, removes duplicates, and fixes broken links. The best results came from combining the wiki with search: the wiki complements search but does not replace it. In a separate legal test, the wiki helped most where the answer was spread across several documents: 87% correct answers versus 73% for search alone. On simple queries, the wiki and search performed about equally, and in one case the wiki even grew larger than the source documents.
What remains unsolved and what to do
Anthropic itself acknowledges the limitations. Taking notes during a task distracts the agent from the work. Dreaming requires additional compute, and the result arrives with a delay. Wikis and notes need regular maintenance, or errors and hallucinations accumulate. Nor should you automatically load too many memories into every session: they eat up context and pass a bad note on to all future sessions. All tests were run by Anthropic itself.
The company introduced Dreaming in May as a research preview for Claude Managed Agents. The service reads an existing memory store and up to 100 past sessions, removes duplicates and outdated entries, and creates a new store without touching the original. At launch, only Claude Opus 4.7 and Sonnet 4.6 were supported. Others are moving in a similar direction. OpenAI also calls Dreaming the updated ChatGPT memory it released in June, which maintains a user profile in the background and corrects outdated information; according to the company, time-aware accuracy rose from 52.2% to 75.1%. Google has WikiSkill, a wiki for agents with separate layers for raw transcripts, distilled knowledge, and specific instructions.
If you are building an agent with long-term memory, do not make it keep a diary of every step. Store strategies and verified rules, and do not write down what is visible in the current context. Add a regular check of notes against session logs, and keep the original so you can roll back. For large knowledge bases, combine a wiki with search.
- Nov 18, 2026Opus 5.5 Tells 19 Safety Tests From 36 Ordinary Tasks Without a Single Error
- Nov 18, 202612 Design Services Keep AI-Built Sites From Looking Like Slop: Free or From $6 a Month
- Nov 18, 2026Claude Code with Kimi waited 22.6 s on command checks — a faster model cut it to 9.9 s
- Nov 18, 2026Kub Tri gave an AI agent project memory: reviews get easier, but the model costs almost a third more
Key Questions Answered
What are the key details on An agent that records everything gets lost?
The main difficulty is selection. At the moment of writing, the agent does not know what it will need later, so every note is a bet on the future. The older Claude Sonnet 3.7 treated everything as important. Over roughly 10,000 game steps it created 413 files in 225 folders, often starting a separate file for every battle and every item it found. The game state changes constantly, so such an archive quickly becomes outdated. At times the model did not play at all and only tidied its own notes.
What are the key details on Dreaming: checking notes "in sleep"?
The second problem is wrong notes that the agent then trusts blindly. In the Pokémon test, Claude wrote down incorrectly which side of a mountain a landmark was on. Relying on that entry, it fell into the same trap three times, eventually wanted to give up, and did not reach its goal. A similar thing happened with a wrong note about the location of a Pokémon Center: because of it, Claude kept falling off a ledge.
What remains unsolved and what to do?
Anthropic itself acknowledges the limitations. Taking notes during a task distracts the agent from the work. Dreaming requires additional compute, and the result arrives with a delay. Wikis and notes need regular maintenance, or errors and hallucinations accumulate. Nor should you automatically load too many memories into every session: they eat up context and pass a bad note on to all future sessions. All tests were run by Anthropic itself.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.