Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)

Garry Tan Deletes the /investigate Skill: 50.0% vs. 47.2% Success Rate, and Fixes Cost 43% More

A skill is a set of instructions for Claude Code, such as a step-by-step bug investigation routine. Garry Tan deleted one such skill, /investigate, from gstack: across 18 bugs with 20 attempts each, the skill-equipped runs fixed 50.0% and the runs without it fixed 47.2%. The 2.8-point gap falls well short of the 10-point threshold, so it is noise, and each successful fix with the skill cost 43% more. When a new model ships, test your own skills the same way.

AI-processed from TG: @sergiobulaev (Сергей Булаев AI); edited by Hamidun News
Garry Tan Deletes the /investigate Skill: 50.0% vs. 47.2% Success Rate, and Fixes Cost 43% More
Source: TG: @sergiobulaev (Сергей Булаев AI). Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceTG: @sergiobulaev (Сергей Булаев AI) ↗
Editor-in-Chief
Key metric #1The 10-point threshold was written down before the runs, so the conclusion couldn't be fitted to the results
360 runs per branch in total18 bugs, 20 attempts each
Key metric #3/write-skill lets into a skill only what the model doesn't already do on its own

A skill for a coding agent is a ready-made set of instructions that you plug into Claude Code so it works the way you want. For example: "when fixing a bug, first find the cause, then write the fix." The problem is that models keep getting smarter, and an instruction that helped yesterday may now only bloat the prompt and burn money.

Garry Tan tested this on his own gstack repository and deleted the /investigate skill: with the skill, the model fixed 50.0% of bugs, without it 47.2%, and each successful fix with the skill cost 43% more.

If your project keeps a set of prompts, rules, or skills for an agent, this testing method can help you figure out which ones to throw out.

What happened with /investigate

gstack is a repository of skills for Claude Code, with about 140 thousand stars and more than 50 skills. One of them, /investigate, walked the model through a step-by-step bug investigation. It used to help. Now, according to the author, Opus 5.5 fixes bugs just as well without that routine. Each new model can do more than the last, while old skills start to get in the way and cost money.

The team didn't leave "the skill isn't needed" as a gut feeling. They measured it. With the skill, the share of successful fixes was 50.0%; without it, 47.2%. A gap of 2.8 percentage points against a pre-set threshold of 10 points is noise, not a real advantage. On top of that, each successful fix with the skill cost 43% more. The skill was no better and more expensive, so it was cut.

Now bugs are fixed directly with a short routine: one check of the cause, a failing test before the edit if the cause is unclear, then the fix and a rerun.

A testing recipe that works for any skill

The method is simple, and you can carry it over to your own project.

First: collect real tasks on which the model, without the skill, succeeds in roughly half the cases. On tasks that are too easy, it solves everything anyway; on tasks that are too hard, it solves nothing. In both cases you can't see a difference between the variants, and the test shows nothing.

Second: run each task many times on two branches, with the skill and without it. At gstack that meant 18 bugs with 20 attempts each, or 360 runs per branch. The model answers differently from one run to the next, so a single run isn't enough.

Third: before you start, write down what gap you consider significant. gstack's threshold is 10 points. This guards against self-deception: if you decide after seeing the numbers, any difference can be declared a win.

Fourth: track two numbers at once. They are the share of successful runs and the cost of each successful run. A skill can slightly raise the success rate but double token consumption. Or, as here, give no gain at all and still add 43% to the cost. Without the second number, an effect like this is easy to miss.

How the check was built into the tooling

The team built the same approach into gstack itself. The /eval-plan skill prepares such an eval in advance, meaning a test plan: a set of tasks, the number of runs, and a threshold. And /write-skill requires adding to a skill only what the model doesn't do on its own. If the model already handles the task, there is no need to write an instruction.

That gives you a working cycle. A new model ships, and you run the eval again. If there's a gap, the skill stays. If there's no gap, the skill goes. This keeps your set of instructions from growing into dead weight that you pay for with every request.

What to keep in mind

The result applies to these specific 18 bugs, this specific model, and this specific skill. On a different stack, in a different language, or on harder tasks, the skill may turn out to be useful. The conclusions rely on the author's post and the pull request description; no one has independently reproduced the numbers. A 2.8-point gap means "within noise," not proof that there is no effect at all.

The practical takeaway for teams building agents: treat a skill as a hypothesis, not an asset. Every skill should have test tasks, a threshold, and a date of its last check. Then, after each model update, you remove what the model already knows how to do and save tokens without losing quality.

Story

Key Questions Answered

What happened with /investigate?

gstack is a repository of skills for Claude Code, with about 140 thousand stars and more than 50 skills. One of them, /investigate, walked the model through a step-by-step bug investigation. It used to help. Now, according to the author, Opus 5.5 fixes bugs just as well without that routine. Each new model can do more than the last, while old skills start to get in the way and cost money. The team didn't leave "the skill isn't needed" as a gut feeling. They measured it.

What are the key details on A testing recipe that works for any skill?

The method is simple, and you can carry it over to your own project. First: collect real tasks on which the model, without the skill, succeeds in roughly half the cases. On tasks that are too easy, it solves everything anyway; on tasks that are too hard, it solves nothing. In both cases you can't see a difference between the variants, and the test shows nothing. Second: run each task many times on two branches, with the skill and without it.

How the check was built into the tooling?

The team built the same approach into gstack itself. The /eval-plan skill prepares such an eval in advance, meaning a test plan: a set of tasks, the number of runs, and a threshold. And /write-skill requires adding to a skill only what the model doesn't do on its own. If the model already handles the task, there is no need to write an instruction. That gives you a working cycle. A new model ships, and you run the eval again. If there's a gap, the skill stays. If there's no gap, the skill goes.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…