12 Pentesting Tools vs. Juice Shop: The Best AI Agent Found 5 Holes Out of 116

Every week a new "AI-powered pentesting agent" shows up on GitHub, promising "3600x speedup over a human" or "90% on benchmarks." The author of this Habr experiment tested those claims head-on. He took 12 tools: six AI agents, four classic DAST scanners and one hybrid setup. All of them attacked OWASP Juice Shop v20.3.0, which has 116 known vulnerabilities. The best result came not from an autonomous agent but from Claude connected to Burp Suite via MCP: 10 findings and a score of 9/10. The best purely autonomous agent, Strix, found five vulnerabilities.

How the test bench was set up

Juice Shop is built on Angular, Node.js/Express and SQLite. It contains 116 challenges in 16 categories that cover the entire OWASP Top 10. Difficulty ranges from 1 to 6 stars, from missing security headers to deserialization and XXE.

The test ran in black-box mode. Nobody got the source code. The AI agents were not told they were facing Juice Shop, so that the model would not pull ready-made solutions from the internet. The author evaluated effectiveness, speed, report quality, usability, local model support, operation from Russia without a VPN, and cost. The measurements were taken in September 2026 on Windows 11 and Kali Linux.

Six AI agents, and one that never got its turn

• Strix (open-source, Apache 2.0): a multi-agent system in a Docker sandbox that supports OpenAI, Anthropic, Google, OpenRouter and Ollama. The first run on Qwen 3.5-35b via OpenRouter ended in a crash: 3 hours of work and two info findings. The second took 58 minutes and produced 5 vulnerabilities (1 critical, 3 high, 1 medium). Strix confirmed the SQL injection with a working PoC. It costs ~$15 per scan, scored 6/10, and works from Russia. Local Qwen via LM Studio ran for more than 24 hours and found only 4 low-severity vulnerabilities.
• CAI from Alias Robotics (MIT + Proprietary, the project is archived, successor is CSI): 10 minutes, 4 findings (3 medium, 1 low), and it missed the critical SQL injection. It costs ~$1 per scan and scored 5/10. The claimed "3600x" stayed on paper.
• PentAGI: Docker sandbox, 20+ tools (nmap, metasploit, sqlmap), a web UI, and support for Ollama, OpenAI, Anthropic, Google, AWS Bedrock, DeepSeek and Qwen. The first run on OpenAI produced zero findings in 1h 50min. The second produced 6 findings (1 critical, 1 medium, 2 low) in 1h 32min. It costs ~$2, scored 5/10, and needs a VPN.
• PenliAgent: a commercial cloud agent (Free / $40 per month). Antivirus blocks the Windows installer, and the interface is slow on Kali. It got stuck on port scanning, collected 6 low/info findings in 1h 11min, and scored 4/10. It needs a VPN and has no local models.
• PentestGPT (published at USENIX Security 2024): run on Claude Opus. The three-command setup instructions didn't work, and the result came back as raw JSON. It did find the SQL injection, but the author credits that to the model, not the wrapper. 3 findings in 20 minutes, ~$20 per scan, score 2/10.
• T3MP3ST (AGPL-3.0, claims 90.1% on XBOW): ran on Gemini 3.1 Pro. In 12 minutes it produced 27 findings, all low/info, meaning the same security headers that Nikto and Nuclei catch. It costs ~$2, scored 2/10, and needs a VPN.
• Shannon: a white-box pentester that requires source code. It would not start in black-box mode and was left out of the scoring.

Classic tools and the hybrid: who caught the serious vulnerabilities

• Acunetix (Invicti): 1h 20min, 13 findings, including critical and medium ones and the SQL injection. It installs locally, produces PDF reports, and works from Russia. Score: 8/10.
• Burp Suite Pro (~$449 per year): 28 minutes, 15 findings, all low/info (headers, CORS), even with Active Scan++. It did not find the SQL injection. Score: 6/10, though it remains the gold standard for manual work.
• Nuclei (ProjectDiscovery, 10 000+ templates): 11 minutes, 24 informational findings. Free, score 2/10.
• Nikto: 4 minutes, 3 info findings, score 1/10.
• Claude + Burp Suite MCP (~$20 per month): Claude receives the request history, the site map and Burp's scanner results, then builds its own test requests and writes the report. 30 minutes, 10 findings: 2 critical, 3 high, 5 medium. Score: 9/10. Downsides: cloud only, and it doesn't work from Russia without a VPN.

What this means in practice

The takeaway from the table is an uncomfortable one for marketing. The number of findings says almost nothing about quality: T3MP3ST's 27 and Nuclei's 24 turned out to be noise, while Strix's 5 included a confirmed critical SQL injection. Instability is also still normal for this class of tools. Strix and PentAGI both failed on the first attempt, and PentAGI burned tokens for nothing.

For teams that need to keep data local, the picture is bleak for now. Local Qwen in Strix spent a full day and produced four weak findings. From Russia without a VPN, only Strix, Acunetix, Burp, Nuclei and Nikto work.

The practical setup today: Acunetix as an automated baseline layer, Burp for manual work, and Claude via MCP when you need intelligent analysis. Autonomous agents are worth keeping as an experiment, not as a replacement for a pentester.