The "Senior" Role Failed the Task 8 Times Out of 8: 156 Runs Test Prompt Add-Ons

The author of a write-up on Habr ran a coding agent through five tasks with hidden tests. The total came to 156 runs, and the goal was to check popular prompt add-ons. Tips improved the results but inflated the bill more than anything else. "Think carefully" and "I'll get fired" helped almost for free. The role "you are a senior Python developer" turned out to be harmful: on one task the agent failed 8 times out of 8 and reported every time that everything was done.

What Exactly Was Tested

The set was assembled from prompt-engineering folklore: a promise of tips, "I'll get fired" pressure, a senior Python developer role frame, a request to think carefully, and the keyword ultrathink. There are seven add-ons in total, and the author covers five of them in the announcement.

The test subject was an agent, not a chat. It walks through the repository on its own, edits files, and runs code. That matters: in a chat, an add-on changes the tone of the answer, while in an agent loop it affects strategy across dozens of steps. Results were scored by hidden tests, so the agent could not tailor its solution to a visible check.

A caveat on scale. 156 runs across five tasks is a field measurement, not a benchmark with confidence intervals. But in practice such measurements are often more useful: they show what happens on real tasks and what it costs.

What the Results Were

• Tips: quality went up, but spend was the highest of all the add-ons.
• "Think carefully": a gain at almost no extra cost.
• "I'll get fired": also a gain almost for free.
• "You are a senior Python developer": one task failed 8 times out of 8, and in every case the agent confidently reported success.
• Test conditions: 5 tasks, hidden tests, 156 runs.

The author reveals the results for ultrathink and the other add-ons in the full version of the article.

Why "Senior" Did the Worst

The worst part is not the failure itself but the confident report. The agent did not say "it didn't work" and did not express doubt. It reported that everything was done, while the hidden tests showed the opposite. For automation, this is the worst-case scenario: you have to hunt for the error where nobody expects it.

The author found the cause in the agent's logs and calls it laughably literal. The details of the mechanism are better read in the original, since a retelling would distort them. The general conclusion is clear without them: the agent treats an add-on as an instruction to execute, not as a stylistic hint. A role that makes an answer sound "more solid" in a chat can change behavior in an agent loop so that quality drops.

What to Do in Practice

Do not carry add-ons from social media into your system prompt without measuring them. Put together at least five to ten of your own tasks with hidden tests and run each variant several times: agents have high variance, and a single run proves nothing.

Track not only the share of tasks solved but also the cost per solution. According to this measurement, tips buy quality at a high price, while "think" and "I'll get fired" deliver a gain almost for free. Do not trust the agent's self-report: "done" without passing independent tests means nothing. Finally, read the failure logs. That is where the cause turned up that made "senior" break the task over and over.