Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

AGENTS.md helped Codex follow project rules: 0 violations out of 13 versus 5 out of 13

A new AI agent session doesn't remember yesterday's agreements: it puts SQL directly into the service, adds an extra cancelled_at column, and forgets to write the action to the audit log. Tech Lead Sergey Proshchaev ran 18 Codex sessions on a small Python service. Without rules, the agent violated 5 of 13 project decisions; with an AGENTS.md file or a skill, it violated 0 of 13. That's why you should only write into the file what the agent would otherwise get wrong on its own.

AI-processed from Habr AI; edited by Hamidun News
AGENTS.md helped Codex follow project rules: 0 violations out of 13 versus 5 out of 13
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
The skill gave no advantage over AGENTS.mdthe root file just became shorter
Vercelwithout an explicit reference in AGENTS.md, the skill was not invoked in 56% of cases
ETH Zurichmodel-generated files raised task costs by 20–23%

You spend a week building a project with an AI agent and agree that SQL lives only in db.py, errors go through AppError, and canceling a subscription changes the status field and writes an audit log entry. The session closes.

The next day, a new session adds a revenue report, puts conn.execute directly into the service, creates a cancelled_at column, and forgets about the audit. The tests are green, the report calculates correctly, but the team's decisions stayed in yesterday's chat.

Tech Lead Sergey Proshchaev tested on 18 Codex sessions whether an AGENTS.md file fixes this. It's a plain text file with project rules that the agent reads at startup.

The result: without rules, the agent violated 5 of 13 verifiable decisions; with AGENTS.md or with a skill, meaning a separate directory of instructions, it violated 0 of 13.

What the experiment showed

The author ran Codex CLI 0.166.0 on a small Python service. The task: subscriptions with prices in rubles, cancellation, and a revenue report. Each session started from scratch, in an isolated directory, with no conversation history. Three variants were compared: no rules, AGENTS.md, and a separate skill. Each variant was run three times.

In the first round there were eight checks. Without rules, the agent broke two in every session: it named the money column price_kopecks instead of the agreed name, and it put SQL in the service even though the neighboring users.py accesses the database through db.py. The rest it inferred from the code on its own: storing money in kopecks, getting time through clock, raising AppError.

In the second round, the author added five decisions that go against the model's habits. Without rules, in all three sessions the agent created cancelled_at instead of status, did not call db.add_audit even though the function existed in the project, and wrote error codes in the old style bad_price. These are the same 5 violations out of 13 checks. With AGENTS.md and with the skill, there were no violations at all. Acceptance tests passed in every variant, including "no rules." Regular tests don't see such agreements, so a separate script, check.py, was needed to parse the code via ast.

The author honestly removed two indicators from the tally. The agent used UUIDs even without rules, because the signature required a string id. The log format was not violated, because the new module had no logs at all.

How to write rules

Step 1: For each decision, ask whether the agent would do it that way on its own. Check this not by guesswork but by running a session without rules. Step 2: A rule should have a reason and an observable sign of violation, and the forbidden thing should be named explicitly (cancelled_at, execute). If existing code doesn't follow the rule, say so in the rule itself, otherwise the agent will treat whatever it sees nearby as the model.

Step 3: Put decisions that apply to any change in the root AGENTS.md. Codex reads the global file, then one file per directory along the path to the working directory. The overall limit project_doc_max_bytes defaults to 32 KiB. Rules for a single module are better kept in the AGENTS.md of that module's directory.

Step 4: Move long material into a skill, but reference it from AGENTS.md. The agent sees only the skill's header and decides for itself whether to load the rest. In Vercel's measurement on Next.js 16 documentation, the skill was not invoked in 56% of cases, and the share of solved tasks stayed at 53%, the same as without documentation. An explicit line in AGENTS.md raised it to 79%, and a compressed index placed directly in the file gave 100%. In Proshchaev's experiment the reference was always present, and the skill was opened in six out of six sessions.

Step 5: Check rules in a clean environment. The author's global AGENTS.md contains 62 KB of personal rules and about a hundred skills, so a "no rules" variant would not truly be without rules without isolation. For testing, point HOME and CODEX_HOME to a temporary directory. Step 6: Turn decisions into automated checks, and test the checks themselves on deliberately broken code.

Limitations and cost

The experiment is small: one model, one task, three sessions per variant. The violation rate can't be extrapolated from it to other agents. The skill gave no measurable advantage over AGENTS.md; its benefit is organizational only. The check.py script catches only pre-known forms of violations: SQL routed through a different helper would slip past it.

A context file costs money. According to the ETH Zurich study "Evaluating AGENTS.md," model-generated files did not raise the share of solved tasks, while cost rose by 20–23%. Files written by developers gave +2.4% solved tasks, statistically insignificant, with cost increases of up to 19%. Agents start exploring and testing more, and that also consumes tokens.

The practical takeaway: take the last three fixes you made by hand after the agent's work, write them down as rules with a sign of violation, and run one session in a clean environment. The rules the agent breaks without the file and follows with it are what make up your AGENTS.md. Everything else only adds costs. Format support depends on the tool: Claude Code reads CLAUDE.md, Gemini CLI reads GEMINI.md by default, so check the documentation for your own agent.

Story

Key Questions Answered

What the experiment showed?

The author ran Codex CLI 0.166.0 on a small Python service. The task: subscriptions with prices in rubles, cancellation, and a revenue report. Each session started from scratch, in an isolated directory, with no conversation history. Three variants were compared: no rules, AGENTS.md, and a separate skill. Each variant was run three times. In the first round there were eight checks.

How to write rules?

Step 1: For each decision, ask whether the agent would do it that way on its own. Check this not by guesswork but by running a session without rules. Step 2: A rule should have a reason and an observable sign of violation, and the forbidden thing should be named explicitly (cancelled_at, execute). If existing code doesn't follow the rule, say so in the rule itself, otherwise the agent will treat whatever it sees nearby as the model. Step 3: Put decisions that apply to any change in the root AGENTS.md.

What are the key details on Limitations and cost?

The experiment is small: one model, one task, three sessions per variant. The violation rate can't be extrapolated from it to other agents. The skill gave no measurable advantage over AGENTS.md; its benefit is organizational only. The check.py script catches only pre-known forms of violations: SQL routed through a different helper would slip past it. A context file costs money.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…