Garry Tan Deletes the /investigate Skill: 50.0% vs. 47.2% Success Rate, and Fixes Cost 43% More
A skill is a set of instructions for Claude Code, such as a step-by-step bug investigation routine. Garry Tan deleted one such skill, /investigate, from gstack: across 18 bugs with 20 attempts each, the skill-equipped runs fixed 50.0% and the runs without it fixed 47.2%. The 2.8-point gap falls well short of the 10-point threshold, so it is noise, and each successful fix with the skill cost 43% more. When a new model ships, test your own skills the same way.
AI-processed from TG: @sergiobulaev (Сергей Булаев AI); edited by Hamidun News
A skill for a coding agent is a ready-made set of instructions that you plug into Claude Code so it works the way you want. For example: "when fixing a bug, first find the cause, then write the fix." The problem is that models keep getting smarter, and an instruction that helped yesterday may now only bloat the prompt and burn money.
Garry Tan tested this on his own gstack repository and deleted the /investigate skill: with the skill, the model fixed 50.0% of bugs, without it 47.2%, and each successful fix with the skill cost 43% more.
If your project keeps a set of prompts, rules, or skills for an agent, this testing method can help you figure out which ones to throw out.
What happened with /investigate
gstack is a repository of skills for Claude Code, with about 140 thousand stars and more than 50 skills. One of them, /investigate, walked the model through a step-by-step bug investigation. It used to help. Now, according to the author, Opus 5.5 fixes bugs just as well without that routine. Each new model can do more than the last, while old skills start to get in the way and cost money.
The team didn't leave "the skill isn't needed" as a gut feeling. They measured it. With the skill, the share of successful fixes was 50.0%; without it, 47.2%. A gap of 2.8 percentage points against a pre-set threshold of 10 points is noise, not a real advantage. On top of that, each successful fix with the skill cost 43% more. The skill was no better and more expensive, so it was cut.
Now bugs are fixed directly with a short routine: one check of the cause, a failing test before the edit if the cause is unclear, then the fix and a rerun.
A testing recipe that works for any skill
The method is simple, and you can carry it over to your own project.
First: collect real tasks on which the model, without the skill, succeeds in roughly half the cases. On tasks that are too easy, it solves everything anyway; on tasks that are too hard, it solves nothing. In both cases you can't see a difference between the variants, and the test shows nothing.
Second: run each task many times on two branches, with the skill and without it. At gstack that meant 18 bugs with 20 attempts each, or 360 runs per branch. The model answers differently from one run to the next, so a single run isn't enough.
Third: before you start, write down what gap you consider significant. gstack's threshold is 10 points. This guards against self-deception: if you decide after seeing the numbers, any difference can be declared a win.
Fourth: track two numbers at once. They are the share of successful runs and the cost of each successful run. A skill can slightly raise the success rate but double token consumption. Or, as here, give no gain at all and still add 43% to the cost. Without the second number, an effect like this is easy to miss.
How the check was built into the tooling
The team built the same approach into gstack itself. The /eval-plan skill prepares such an eval in advance, meaning a test plan: a set of tasks, the number of runs, and a threshold. And /write-skill requires adding to a skill only what the model doesn't do on its own. If the model already handles the task, there is no need to write an instruction.
That gives you a working cycle. A new model ships, and you run the eval again. If there's a gap, the skill stays. If there's no gap, the skill goes. This keeps your set of instructions from growing into dead weight that you pay for with every request.
What to keep in mind
The result applies to these specific 18 bugs, this specific model, and this specific skill. On a different stack, in a different language, or on harder tasks, the skill may turn out to be useful. The conclusions rely on the author's post and the pull request description; no one has independently reproduced the numbers. A 2.8-point gap means "within noise," not proof that there is no effect at all.
The practical takeaway for teams building agents: treat a skill as a hypothesis, not an asset. Every skill should have test tasks, a threshold, and a date of its last check. Then, after each model update, you remove what the model already knows how to do and save tokens without losing quality.
- Nov 18, 2026AI agent left SOC 27 minutes to respond versus 53 for a junior attacker
- Nov 18, 2026ArisGlobal cut its AI bill by 40% by caching the shared prompt instead of the PDF
- Nov 18, 2026After a Ban, the Author Launched an Agent Team in Kimi, Codex and OpenCode: Runs from $0.30
- Nov 18, 2026TokenEconomizer plugin cuts AI coding agent costs by 18.4–30.9% without changing the model
Key Questions Answered
What happened with /investigate?
gstack is a repository of skills for Claude Code, with about 140 thousand stars and more than 50 skills. One of them, /investigate, walked the model through a step-by-step bug investigation. It used to help. Now, according to the author, Opus 5.5 fixes bugs just as well without that routine. Each new model can do more than the last, while old skills start to get in the way and cost money. The team didn't leave "the skill isn't needed" as a gut feeling. They measured it.
What are the key details on A testing recipe that works for any skill?
The method is simple, and you can carry it over to your own project. First: collect real tasks on which the model, without the skill, succeeds in roughly half the cases. On tasks that are too easy, it solves everything anyway; on tasks that are too hard, it solves nothing. In both cases you can't see a difference between the variants, and the test shows nothing. Second: run each task many times on two branches, with the skill and without it.
How the check was built into the tooling?
The team built the same approach into gstack itself. The /eval-plan skill prepares such an eval in advance, meaning a test plan: a set of tasks, the number of runs, and a threshold. And /write-skill requires adding to a skill only what the model doesn't do on its own. If the model already handles the task, there is no need to write an instruction. That gives you a working cycle. A new model ships, and you run the eval again. If there's a gap, the skill stays. If there's no gap, the skill goes.
Build & Ship Production AI Agents Without Boilerplate Code
Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.