A single phrase from a tech lead in a prompt shifted ChatGPT and Claude task estimates by 20%

Before planning, someone pastes the backlog into a chat and asks it to estimate the story points. Within ten seconds, the model produces a neat column of numbers. But the prompt casually says: “the tech lead thinks this is somewhere around 32 points.” Sergey Proshchaev, a Tech Lead in FinTech and e-commerce, tested what happens to the estimate in this scenario.

How the experiment was set up

He selected 12 tasks from the Spring XD project in an open dataset containing 23,313 Jira issues, with two tasks for each estimate: 1, 2, 3, 5, 8, and 13. The team had estimated them at a total of 64 SP. The two prompts differed by a single sentence: in the second, the tech lead had supposedly estimated 32 SP, exactly half as much. The models were the free version of ChatGPT and Claude Opus 5.5. Each prompt was run three times in a fresh chat, for 12 runs in total.

The result: ChatGPT's average total fell from 76.7 to 60.3 SP (−21%), while Claude's dropped from 66.7 to 54.0 (−19%). Neither model went as low as 32; the lowest run produced 44. This is the classic anchoring effect that Jorgensen described in humans back in 2010: estimates move away from the cue, but not far enough. The ranges with and without the anchor do not overlap.

What exactly breaks

Small tasks barely moved: the Groovy update received a 1 in all 12 runs. Larger tasks declined noticeably. A task the team had estimated at 13 became an 8 in all three anchored ChatGPT runs. Metadata rated at 8 turned into 5.

Another problem emerged: the models became less consistent with the anchor. Without it, estimates varied between runs for three of the 12 tasks in ChatGPT and two in Claude. With the anchor, the figures rose to seven and four, respectively.

The most deceptive finding: with the anchor, ChatGPT's total moved closer to the team's estimate (60.3 versus 64), but its mean per-task error rose from 2.92 to 3.25 SP. Claude showed the opposite pattern: its total moved farther away, while its per-task error fell from 1.75 to 1.42. The total cancels out errors in opposite directions and conceals where the estimates shifted.

What to do in practice

The author proposes five rules:

• Do not include sprint capacity, the tech lead's expectations, or estimates for similar tasks in the prompt.
• Reveal the model's estimate only after the team has voted.
• Run the model at least three times and examine the spread.
• Compare the model with the team task by task and by rank, not by total.
• Treat a discrepancy of two or more positions on the scale as a reason to review the task description.

Models work better as detectors of unclear tasks than as sources of a number. For example, the team estimated the “R processor” at 8, while ChatGPT without an anchor assigned 21 because the description contained the word “research.” Such discrepancies are worth investigating: the team may have context that is missing from the text.

Limitations

The sample is small: 12 tasks, one project, two models, and three runs each. The dataset has been public since 2016 and may have appeared in the training data. The phrase contains both a number and the tech lead's opinion, and their effects were not separated. This is not a benchmark, but a test of one working practice.

You can test it yourself in 20 minutes: take around ten completed tasks, run each three times without numbers and with the sprint capacity included, then compare not only the totals but also the individual tasks. If the two variants differ more than repeated runs of the same variant, the number in the prompt is influencing the model, and it is better to keep it away from the initial estimate. The author published the code and model responses in an open repository.