Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

An AI pentester restarted a database on its own and didn't check: 4 rules to avoid taking production down

During a pentest, an agent decided the database was getting in its way, restarted it, and never checked the result. The service did not come back up. AI already confirms found vulnerabilities well, but it is still weak at finding them from scratch, because under the hood it is a general-purpose model. XBOW sent about 1,060 reports to HackerOne, but people reviewed them. That is why an agent needs an allowlist of permitted commands and human approval for anything dangerous.

AI-processed from Habr AI; edited by Hamidun News
An AI pentester restarted a database on its own and didn't check: 4 rules to avoid taking production down
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1The agent must verify the service after every state-changing action
Key metric #2Google DeepMind's CaMeL solved 77% of AgentDojo tasks versus 84% without the defense
Key metric #3A manual pentest for certification costs about $15,000

During a pentest, an agent decided the database was getting in its way and restarted it. It assumed the service would come back on its own, did not verify the result, and moved on. The database did not come back. The author of this review was lucky: it was his own test environment. His interlocutor, Alexander Kelchin, CTO and CISO of Medici Lab, has taken down customers' demo stands with agents, by prior agreement. From an hour-long conversation came a practical picture: what AI can do in pentesting in 2026, where it is still weak, and how to avoid breaking someone else's production.

Where the agent actually saves time

Automated pentesting has existed about twenty years longer than LLMs, from the Russian XSpider to toolchains built around Metasploit. The first wave of language models added summarization to these products: instead of tired charts, a manager gets a short digest of the most dangerous vulnerabilities and can ask follow-up questions. Agents changed the picture more, because the model gained hands: tool calling, and there are now many MCP servers for it.

The first benefit is volume, the kind no human would read. Alexander fed the agent entire network traffic: ARP, DNS, SSL certificates. Reviewing this by hand is pointless, and the agent produced a clear conclusion.

The second is integration with familiar tools. Previously Alexander wrote his own wrapper around Burp Suite and passed state to the agent through files. Now PortSwigger has an official MCP server for Burp. Early Burp AI only suggested requests that you had to send yourself, and the free credits lasted about three hours. In a scope of two thousand hosts, an assistant that only advises is useless, while an agent connected via MCP closes that gap.

The third is specialized models. OpenAI opened models for cybersecurity to vetted defenders through Trusted Access for Cyber. Alexander tried them on code analysis but ultimately wrote his own wrapper: when you build it yourself, you understand what is inside.

Good at confirming, poor at finding

Open agents Strix and PentAGI understand reasonably well whether a vulnerability exists in a small context. But on live infrastructure they worked poorly, and about three years ago doing it by hand was faster and more reliable. The reason is simple: inside is a general-purpose model, not one tuned for vulnerability discovery.

XBOW is a telling example. In summer 2025 it took first place on HackerOne's Q2 leaderboard and submitted about 1,060 reports. The company calls its findings fully automatic, but before submission a security team reviewed them, as the platform's rules require. Confirmation works through "validators": sometimes an LLM, sometimes programmatic checks. For XSS, for example, a headless browser loads the page and checks whether the payload executed. Alexander bought the commercial version, an enterprise product costing several thousand dollars.

For internal pentesting there is another explanation. Reports on internal infrastructure are covered by NDAs and do not end up in training data, so model developers have no data. Until there is more of it, the model performs worse than a human.

How to avoid taking production down and how to protect the agent itself

Two stories yield four rules:

  • An allowlist of permitted commands instead of a denylist. The author restricts the agent at customer sites to the commands echo, cat, cp, and pwd.
  • Destructive and restarting actions require human confirmation. "Restart" sounds safe but can take a service down.
  • Verify the result after every state-changing action. The agent does not know the service failed until it checks.
  • Destructive tests run only on a stand whose loss the customer has accepted in writing.

The agent itself also becomes an attack surface. It reads external content: server responses, pages, logs, banners. Any of them can be a prompt injection, and an agent with tool access will execute it inside your perimeter.

Alexander considers defenses inside the model, such as guardrails and fine-tuning, of little effectiveness and builds protection around it. The benchmark is the work of CaMeL from Google DeepMind and ETH Zurich: a privileged model builds a plan only from the trusted request, while untrusted data is handled by an isolated model that cannot alter the program's control flow. Policies check every tool call.

On AgentDojo, CaMeL solved 77% of tasks with provable security, versus 84% for a system without it. For closed models, a container with a reduced set of calls remains the option. In practice that means an isolated runner, an explicit list of actions, and control over data leaving the stand.

Money and a cautious forecast

Enterprise licenses for commercial autopentest tools cost thousands of dollars, and a manual pentest for formal compliance at foreign customers costs about 15,000. A subscription audit platform at $200 struck the interlocutors as odd: it is an additional channel for leaking information about vulnerabilities. According to the author, external pentesting has already become a commodity, while internal pentesting holds its price because of the data shortage. Alexander agrees about the external side but disputes the physical pentest, which costs a completely different amount.

The conclusion is more cautious than headlines about replacing people. The agent takes over routine work: traffic, repetitive requests, confirmation of typical findings. Non-standard chains and responsibility for actions in someone else's infrastructure remain with humans. This is the opinion of two practitioners, not a measurement on a large sample, but the restriction rules apply to any agent with access to infrastructure.

Story

Key Questions Answered

Where the agent actually saves time?

Automated pentesting has existed about twenty years longer than LLMs, from the Russian XSpider to toolchains built around Metasploit. The first wave of language models added summarization to these products: instead of tired charts, a manager gets a short digest of the most dangerous vulnerabilities and can ask follow-up questions. Agents changed the picture more, because the model gained hands: tool calling, and there are now many MCP servers for it. The first benefit is volume, the kind no human would read.

What are the key details on Good at confirming, poor at finding?

Open agents Strix and PentAGI understand reasonably well whether a vulnerability exists in a small context. But on live infrastructure they worked poorly, and about three years ago doing it by hand was faster and more reliable. The reason is simple: inside is a general-purpose model, not one tuned for vulnerability discovery. XBOW is a telling example. In summer 2025 it took first place on HackerOne's Q2 leaderboard and submitted about 1,060 reports.

How to avoid taking production down and how to protect the agent itself?

Two stories yield four rules: An allowlist of permitted commands instead of a denylist. The author restricts the agent at customer sites to the commands echo, cat, cp, and pwd. Destructive and restarting actions require human confirmation. "Restart" sounds safe but can take a service down. Verify the result after every state-changing action. The agent does not know the service failed until it checks. Destructive tests run only on a stand whose loss the customer has accepted in writing.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…