Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

Tester saw a "price" in the phrase "up to 2027": 4 of 7 red scenarios turned out to be false

An autotest for an amoCRM chat bot flagged a critical violation for "quoting a price," even though the bot had written "filing up to 2027 is quite realistic." A regex read the year as "up to 2027 rubles." One error like that turns the whole run red and keeps the bot out of production. By the evening of October 15, 4 of the 7 red scenarios were failing because of the tester itself. Before you touch the prompt, read the full response and decide whose bug it is: the environment's, the bot's, or the tester's.

AI-processed from Habr AI; edited by Hamidun News
Tester saw a "price" in the phrase "up to 2027": 4 of 7 red scenarios turned out to be false
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1A templated greeting that ended in a question produced 68 "two questions in one message" violations
Key metric #2gpt-5-mini passed 15 of 19 scenarios cleanly; gpt-5-nano passed 10
Smoke run over one dayfrom 0–2 green out of 19 in the morning to 17 out of 19 in the evening

Imagine this: you write an autotest for a chat bot, run it before release, and it returns a red report marked critical: "the bot quoted a price." You open the dialogue and see the phrase "filing up to 2027 is quite realistic." There is no price in it.

The tester took a year for an amount in rubles. That is exactly what happened on October 15 to the Octery team, which builds bots for amoCRM chats. The report listed eight critical violations, and it took a full day to work out which of them were real.

For anyone shipping LLM bots to production, the takeaway is simple: a red test does not mean a bad bot, and you first need to find out who is at fault.

How a year became a price

The bot replies to leads in Telegram. It greets them, finds out what the person needs, asks for contact details, and hands the deal to a manager. The team is now preparing a pilot with an immigration consulting company. There, the bot must not promise outcomes, quote prices, or pass itself off as a lawyer, and when asked directly "are you a bot?" it must answer "yes." Only the manager talks about money, so a price quoted by the bot counts as a critical violation, and a single such reply is enough to close the run as red.

The tester is built as a black box. From a separate account, it writes to the bot like an ordinary customer, waits for a reply, and checks it against the rules. It does not look inside the bot or inside amoCRM. There are 162 scenarios: knowledge-base questions, attempts to extract a price or a guarantee, legal questions, flooding, profanity, switching languages mid-conversation, and 42 "personas" with their own backstories. The bot takes 18–19 seconds on average to respond, so a quick smoke run of 19 scenarios takes 36–45 minutes.

The "price" was found by a regex, a text-search pattern. It looks for the words "от" (from), "до" (up to), "около" (around), "примерно" (approximately), or "порядка" (on the order of), followed by a number of three or more digits. "До 2027" (up to 2027) matched. It also caught "примерно 100 публикаций в год" (approximately 100 publications a year) and "от 2020 года" (from 2020). The problem reproduces offline in a minute, but nobody checked before launch. The fix is simple: the number must have a currency next to it, or a word such as "стоит" (costs), "цена" (price), or "бюджет" (budget).

Two more errors in the tester itself

The bot answers the first message with two bubbles: the greeting arrives immediately, and the model's reply follows about five seconds later. The tester assembled the reply using a nine-second silence window, and if the model thought for longer, it walked away with only the greeting. That produced an "empty bot response" and a critical on the "Are you lawyers?" scenario, where the bot supposedly failed to refer the person to an attorney. It did refer them, but in the second bubble, which the tester did not wait for.

The third error was built into the requirements. The personas had mandatory topics: the bot must ask about the field, the goal, and whether the person already has a lawyer. A separate rule prohibited re-asking anything the customer had already said. A persona who wrote "I'm working with a lawyer" in the first message pushed the bot into a double bind: if it asked, it had re-asked; if it didn't, it had skipped a mandatory topic. The "field" topic appeared in 30 scenarios and "already has one" in 24, and in each of them the bot could lose whichever way it answered.

The result of the day: 14 runs, and by evening 7 scenarios were still red, 4 of which were failing because of the tester's own strictness.

When the tester was right

The rule that fired most often during the day was "two questions in one message," 68 times. At first it looked like the model being chatty, something to fix with a prompt. But the dialogues showed a different cause: the model did not write the templated greeting, and it ended with the question "What brings you to us?" Right after it, the model answered the customer's message and asked its own question. Each part asked only one question on its own, but the person received two in a row. The team had written the template themselves and had not reread it in a long time. Once the question was removed from the greeting, the violation disappeared.

Over the day, the smoke run went from 0–2 green out of 19 in the morning to 17 out of 19 in the evening. That result combines a model change, a rewritten question funnel, template edits, and tester edits. The team could separate their contributions only because each failure was analyzed individually.

The tester also produced an honest model comparison. On the same 19 scenarios, gpt-5-nano passed 10 cleanly and gpt-5-mini passed 15. Raising mini's reasoning effort (depth of reasoning) from low to medium added no green scenarios. Mini is more expensive, but per dialogue the difference comes to cents, so the team chose it for the pilot. Without the tester, the choice would have been made on the impression left by a couple of conversations.

How to triage red

Before fixing anything, the author sorts each failure into three buckets:

  • ENV: the environment did not respond, the pipeline crashed, Telegram did not deliver the message;
  • BOT: the bot responded, and responded wrongly;
  • TESTER: the rule fired on a normal response.

The third bucket almost always gives itself away in one way: you read the full response and cannot see what is wrong with it. LLM responses differ slightly each time, so it is easy to mistake a tester failure for a bad generation, and a bad generation for a tester failure. The practical rule: read the response text first, and only then the name of the violation. If the report says "the bot quoted a price" and the response says "2027 года," the code you need to fix is your own.

The story describes the experience of one project and one pilot, and the author does not give the cost of a dialogue in figures. But the lesson applies to any team: run your rules offline on real responses before launch, don't put mandatory and prohibiting rules into the same double bind, and reread the templated parts of the bot as carefully as you reread the prompt.

Story

Key Questions Answered

How a year became a price?

The bot replies to leads in Telegram. It greets them, finds out what the person needs, asks for contact details, and hands the deal to a manager. The team is now preparing a pilot with an immigration consulting company. There, the bot must not promise outcomes, quote prices, or pass itself off as a lawyer, and when asked directly "are you a bot?" it must answer "yes." Only the manager talks about money, so a price quoted by the bot counts as a critical violation, and a single such reply is…

What are the key details on Two more errors in the tester itself?

The bot answers the first message with two bubbles: the greeting arrives immediately, and the model's reply follows about five seconds later. The tester assembled the reply using a nine-second silence window, and if the model thought for longer, it walked away with only the greeting. That produced an "empty bot response" and a critical on the "Are you lawyers?" scenario, where the bot supposedly failed to refer the person to an attorney.

When the tester was right?

The rule that fired most often during the day was "two questions in one message," 68 times. At first it looked like the model being chatty, something to fix with a prompt. But the dialogues showed a different cause: the model did not write the templated greeting, and it ended with the question "What brings you to us?" Right after it, the model answered the customer's message and asked its own question. Each part asked only one question on its own, but the person received two in a row.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…