Anthropic and ETH Zurich: A Long CLAUDE.md Hurts Agent Performance and Raises Costs

CLAUDE.md and AGENTS.md were conceived as a quick way to explain a project's rules to an agent, but a new study shows that long context hurts more often than it helps. On a sample of 138 Python repositories, ETH Zurich researchers saw a drop in success rate and a rise in costs, especially with automatically generated files.

What the study found

The authors of Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? took 138 real repositories, collected 5694 pull requests and ran the tasks through four models: Claude Sonnet 4.5, Codex GPT-5.2, GPT-5.1 Mini and Qwen3-30B. They compared three scenarios: no special files, LLM-generated instructions, and AGENTS.md files maintained by humans. This matters: the study looked not at abstract demos but at real tasks from live codebases.

The main result for automatically created files was unwelcome. Success rate fell by 3% on average, while inference cost rose by more than 20%. Human-written files did better: they raised success by about 4%, but cost also rose by almost 19%. In other words, a context file turned out not to be a free accelerator. Even when it helps, the quality gain stays modest compared with the constant overpayment for tokens and extra agent steps.

Why a long file gets in the way

The most counterintuitive finding in the paper: descriptions of the project structure barely help the agent get its bearings. Sections on folders, architecture and stack look useful to a human, but the agent often finds everything anyway through grep, glob and reading files. If the information already sits in pyproject.toml, package.json, the linter config or the repository structure itself, a long explanation merely duplicates what the model can quickly discover on its own.

If the agent can learn it from the code itself, it is better to delete

it from the instructions.

The researchers also noticed that models given context files take more actions than the specific task requires. They re-read the instructions more often, run tests an extra time, and call the tools mentioned in the file more eagerly. The paper separately notes a rise in reasoning tokens for the GPT models. In other words, an instruction does not just add knowledge; it changes the agent's behavior: it starts following rules even where they do not help solve the current task.

What is worth keeping

The study does not argue for abandoning CLAUDE.md or AGENTS.md entirely. The conclusion is rather that such files should be short and contain only what the agent cannot reliably infer from the code or configs. The less decorative text, the lower the risk that the model fixates on pointless rituals instead of completing the actual request.

• A non-standard test-run command
• The package manager, if it is not obvious
• Custom scripts, tools and deployment quirks
• Naming rules, if they cannot be quickly inferred from the code
• A pointer to .env.example or another critical entry file

Another study complicates the picture: there, Codex with AGENTS.md showed a speedup and lower token consumption. But its sample was much smaller, and the correctness of results was assessed only to a limited extent. So the overall conclusion remains cautious for now: short, practical context sometimes helps, while a long file with an overview of architecture, stack and general rules easily turns into expensive ballast. ETH Zurich also barely touches on code maintainability and conformity to a project's style, so the debate over the usefulness of such files is not yet closed.

What this means

The practical takeaway is simple: treat CLAUDE.md as a list of the agent's fixed mistakes, not as an encyclopedia of the project. If an instruction does not help avoid a specific failure, does not describe a non-standard command and adds no unique context, it is better to remove it. For teams that use coding agents heavily, this is a direct reason to trim context files, cut token spending, and check on their own tasks which lines actually improve results and which only create expensive noise.