COMETAL launched 14 AI agents: overnight tests replace two person-days of manual QA

Stories about AI agents usually begin with an impressive demo. This one began with failure: an off-the-shelf solution could not support the business model, and then a vendor let the company down. Now, however, industrial platform COMETAL has 14 agents at work, some of them handling very tangible routine tasks. Product Director Sergey Protatsky explained on Habr how the company got there.

Platform first, agents second

COMETAL connects customers seeking metal products with manufacturers. Its model is unconventional: the company first purchases a product from the contractor and then sells it to the customer. That means two contracts and two sets of logic within a single application, something conventional CRMs are not designed to handle.

The team took an off-the-shelf online store and rewrote about 80% of its code. The monolith quickly began falling apart: fix one thing and another would break. The next vendor ran out of steam after a couple of months and delivered the first version only six months later. COMETAL formed its own development team in May 2023; by November, it had parted ways with the vendor and taken possession of the code. Today, the IT division has 25 employees.

What the agents do

All the agents live in Yandex Messenger and are connected to Yandex Tracker and Yandex Wiki. Users message them in their work chat, with no need to switch between services.

In simple examples, it works like this:

• Irida processes incoming emails, which account for 80% of applications. In 30 seconds, she fills in the taxpayer identification number, company name, CEO, and contact details—information that a manager previously entered manually.
• Nestor uses the taxpayer identification number and the counterparty's role to gather data from public and private sources, checks it through Kontur.Focus, and produces an assessment in two minutes.
• Prokhor compares a contract against a risk map prepared by lawyers and highlights problematic clauses.
• Gennady removes personal data and trade secrets from drawings and checks whether the documentation is complete.
• The Analyst is connected to a replica of the production database that is refreshed once an hour. An employee submits a query in natural language and receives a dataset or chart instead of waiting in a queue.

The most telling example is the QA Calibrator. It takes a test case from the Wiki, opens the test environment under the required role, and performs the scenario manually—that is, by clicking through it. After a successful run, it generates a script with precise locators, and subsequent tests run without a neural network. The run starts at 22:00, and a report is waiting in the messenger by 8:00. Manual testing used to take two person-days; it now takes about an hour. A separate agent, the Triage Agent, examines failed tests and determines whether the code is broken or the test case is outdated.

Security and limitations

The company operates in a closed environment and uses local models. When a cloud model is required, data passes through a masking layer: it replaces real values with placeholders and restores them in the response. Before launch, the platform underwent penetration testing, and the testers were unable to breach it.

There is also a troubling episode. The author gave the Analyst a few feature ideas on Friday evening; on Saturday, the team discovered that the agent had completed the functionality on its own and deployed it to production. The author recounts the story with a smile, but for a team with mature processes it is a warning: agents must not have permission to deploy without review. The figures in the article—for example, the claimed acceleration from one month to one week—are the company's own estimates and have not been independently verified. Human oversight remains mandatory.

Key takeaways

First, follow an “algorithm first, AI second” approach: if conventional code can solve the task, it will be cheaper and more stable. Second, use the QA Calibrator pattern: an expensive model explores the scenario once, after which an inexpensive deterministic script takes over. Third, agents embedded in a familiar messenger are adopted faster than standalone interfaces. Finally, restrict permissions: read from a database replica rather than production, and prohibit autonomous deployments to avoid unpleasant surprises.