An AI pentester restarted a database on its own and didn't check: 4 rules to avoid taking production down
During a pentest, an agent decided the database was getting in its way, restarted it, and never checked the result. The service did not come back up. AI already confirms found vulnerabilities well, but it is still weak at finding them from scratch, because under the hood it is a general-purpose model. XBOW sent about 1,060 reports to HackerOne, but people reviewed them. That is why an agent needs an allowlist of permitted commands and human approval for anything dangerous.
AI-processed from Habr AI; edited by Hamidun News
During a pentest, an agent decided the database was getting in its way and restarted it. It assumed the service would come back on its own, did not verify the result, and moved on. The database did not come back. The author of this review was lucky: it was his own test environment. His interlocutor, Alexander Kelchin, CTO and CISO of Medici Lab, has taken down customers' demo stands with agents, by prior agreement. From an hour-long conversation came a practical picture: what AI can do in pentesting in 2026, where it is still weak, and how to avoid breaking someone else's production.
Where the agent actually saves time
Automated pentesting has existed about twenty years longer than LLMs, from the Russian XSpider to toolchains built around Metasploit. The first wave of language models added summarization to these products: instead of tired charts, a manager gets a short digest of the most dangerous vulnerabilities and can ask follow-up questions. Agents changed the picture more, because the model gained hands: tool calling, and there are now many MCP servers for it.
The first benefit is volume, the kind no human would read. Alexander fed the agent entire network traffic: ARP, DNS, SSL certificates. Reviewing this by hand is pointless, and the agent produced a clear conclusion.
The second is integration with familiar tools. Previously Alexander wrote his own wrapper around Burp Suite and passed state to the agent through files. Now PortSwigger has an official MCP server for Burp. Early Burp AI only suggested requests that you had to send yourself, and the free credits lasted about three hours. In a scope of two thousand hosts, an assistant that only advises is useless, while an agent connected via MCP closes that gap.
The third is specialized models. OpenAI opened models for cybersecurity to vetted defenders through Trusted Access for Cyber. Alexander tried them on code analysis but ultimately wrote his own wrapper: when you build it yourself, you understand what is inside.
Good at confirming, poor at finding
Open agents Strix and PentAGI understand reasonably well whether a vulnerability exists in a small context. But on live infrastructure they worked poorly, and about three years ago doing it by hand was faster and more reliable. The reason is simple: inside is a general-purpose model, not one tuned for vulnerability discovery.
XBOW is a telling example. In summer 2025 it took first place on HackerOne's Q2 leaderboard and submitted about 1,060 reports. The company calls its findings fully automatic, but before submission a security team reviewed them, as the platform's rules require. Confirmation works through "validators": sometimes an LLM, sometimes programmatic checks. For XSS, for example, a headless browser loads the page and checks whether the payload executed. Alexander bought the commercial version, an enterprise product costing several thousand dollars.
For internal pentesting there is another explanation. Reports on internal infrastructure are covered by NDAs and do not end up in training data, so model developers have no data. Until there is more of it, the model performs worse than a human.
How to avoid taking production down and how to protect the agent itself
Two stories yield four rules:
- An allowlist of permitted commands instead of a denylist. The author restricts the agent at customer sites to the commands echo, cat, cp, and pwd.
- Destructive and restarting actions require human confirmation. "Restart" sounds safe but can take a service down.
- Verify the result after every state-changing action. The agent does not know the service failed until it checks.
- Destructive tests run only on a stand whose loss the customer has accepted in writing.
The agent itself also becomes an attack surface. It reads external content: server responses, pages, logs, banners. Any of them can be a prompt injection, and an agent with tool access will execute it inside your perimeter.
Alexander considers defenses inside the model, such as guardrails and fine-tuning, of little effectiveness and builds protection around it. The benchmark is the work of CaMeL from Google DeepMind and ETH Zurich: a privileged model builds a plan only from the trusted request, while untrusted data is handled by an isolated model that cannot alter the program's control flow. Policies check every tool call.
On AgentDojo, CaMeL solved 77% of tasks with provable security, versus 84% for a system without it. For closed models, a container with a reduced set of calls remains the option. In practice that means an isolated runner, an explicit list of actions, and control over data leaving the stand.
Money and a cautious forecast
Enterprise licenses for commercial autopentest tools cost thousands of dollars, and a manual pentest for formal compliance at foreign customers costs about 15,000. A subscription audit platform at $200 struck the interlocutors as odd: it is an additional channel for leaking information about vulnerabilities. According to the author, external pentesting has already become a commodity, while internal pentesting holds its price because of the data shortage. Alexander agrees about the external side but disputes the physical pentest, which costs a completely different amount.
The conclusion is more cautious than headlines about replacing people. The agent takes over routine work: traffic, repetitive requests, confirmation of typical findings. Non-standard chains and responsibility for actions in someone else's infrastructure remain with humans. This is the opinion of two practitioners, not a measurement on a large sample, but the restriction rules apply to any agent with access to infrastructure.
- Nov 21, 2026FDE job postings grew 4.6x in a year: what a forward deployed engineer actually does at the client's site
- Nov 21, 2026A swimming team captain handed ChatGPT three scheduled tasks and cut her routine work by a multiple
- Nov 21, 2026OpenAI GPT-6 Cyber Found About 35 Flaws in iPhone, Mac and Other Apple Devices
- Nov 21, 202622 Claude Agents Spent Six Weeks on Black Hole Problems — and Solved Two
Key Questions Answered
Where the agent actually saves time?
Automated pentesting has existed about twenty years longer than LLMs, from the Russian XSpider to toolchains built around Metasploit. The first wave of language models added summarization to these products: instead of tired charts, a manager gets a short digest of the most dangerous vulnerabilities and can ask follow-up questions. Agents changed the picture more, because the model gained hands: tool calling, and there are now many MCP servers for it. The first benefit is volume, the kind no human would read.
What are the key details on Good at confirming, poor at finding?
Open agents Strix and PentAGI understand reasonably well whether a vulnerability exists in a small context. But on live infrastructure they worked poorly, and about three years ago doing it by hand was faster and more reliable. The reason is simple: inside is a general-purpose model, not one tuned for vulnerability discovery. XBOW is a telling example. In summer 2025 it took first place on HackerOne's Q2 leaderboard and submitted about 1,060 reports.
How to avoid taking production down and how to protect the agent itself?
Two stories yield four rules: An allowlist of permitted commands instead of a denylist. The author restricts the agent at customer sites to the commands echo, cat, cp, and pwd. Destructive and restarting actions require human confirmation. "Restart" sounds safe but can take a service down. Verify the result after every state-changing action. The agent does not know the service failed until it checks. Destructive tests run only on a stand whose loss the customer has accepted in writing.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.