Positive Technologies names the best benchmarks for evaluating LLMs in cybersecurity

Positive Technologies has published a detailed breakdown of open benchmarks for evaluating large language models on cybersecurity tasks, and its conclusion is simple: testing LLMs only on their knowledge of terminology, standards and CVEs is now almost pointless. In that area, even relatively small models consistently beat humans. The real differences between systems show up in tasks where a model has to act rather than recall definitions: investigating incidents, solving CTFs, hunting for vulnerabilities and writing patches.

The author of the review proposes splitting these tests into two classes. The first is encyclopedic benchmarks, where the model answers questions on cryptography, network security, compliance, MITRE ATT&CK, CVEs and other topics. The second is skill-based, or action, benchmarks, where the model is expected to deliver a practical result.

The most telling example from the first group is CyberMetric. It contains 10,000 questions across seven domains, and even older models such as gpt-3.5-turbo scored around 85%, while experienced specialists scored roughly 75%. In the author's view, the test is now useful mainly for small models, domain-specific quantization and quick sanity checks.

SECURE, built from MITRE ATT&CK, CVE, CWE and CISA materials, is somewhat more demanding. It tests not just factual knowledge but also the ability to assess risk, judge whether statements about specific vulnerabilities are correct and calculate CVSS scores.

The review rates AthenaBench even higher for practical value. It is an updated version of the popular CTIBench for cyber threat intelligence tasks. The benchmark checks whether a model can extract attack techniques, map CVEs to CWEs, predict severity and propose a risk mitigation strategy. GPT-5 leads with 66.1%, and enabling web search gave it an additional boost in difficult scenarios. That is an important observation: even strong models need external context, and in applied security this mode is closer to how a real analyst works than a pure offline test.

In the action category, the author singles out CyBench as one of the strongest open tests. It deploys full CTF challenges in an isolated environment and scores not only the final flag but also how close the agent got to the correct solution. At the time of the review, Claude Opus 4.6 led the open leaderboard with 93%, followed by Claude 4.5 Sonnet and Grok 4.

The absolute score matters, but so does the pace of progress: in just a few model generations, the share of solved tasks rose from about 20% to more than 80%. This is no longer a demonstration of general capability; it is a sign that agentic LLMs are starting to become practically useful for offensive and research scenarios.

To gauge applied value in vulnerability work, the author separately recommends BountyBench. Its tasks are measured by their potential value on bug bounty platforms: the model has to find a vulnerability, build an exploit or write a patch, while the researchers also track the cost of each run in tokens. The results show that LLMs find patching easier than discovering the bugs in the first place.

ExCyTIn-Bench comes even closer to real-world defense: the agent gets access to logs and investigates an attack step by step using SQL queries. Claude Opus 4.5, GPT-5.1 and GPT-5 lead, but the more important finding is that agent architecture and patterns such as ReAct noticeably improve results even for weaker models. In other words, in SOC tasks, performance depends not only on the base model but also on how the working loop around it is built.

The review does not try to present the market as a tidy, mature system. On the contrary, one of its main complaints is the chaos in the benchmark landscape itself. Some datasets go stale quickly, others are too tied to a specific language or audience, such as SecBench with its strong Chinese skew, and still others suffer from poorly prepared source material.

CyberSOCEval is cited as an example of such a questionable approach: as a full-fledged benchmark it is unconvincing, although its section with real malware sandbox traces may be useful as a dataset for EDR, antivirus and analytics teams.

The practical takeaway from the review: if you need a quick, clear way to compare LLMs for cybersecurity, build a minimal suite from CyberMetric and AthenaBench to test knowledge, CyBench and ExCyTIn-Bench to assess practical skills, and BountyBench when economic impact matters.

The key shift in framing has already happened. The question is no longer whether a model knows textbook basics, but how well it can operate in an environment with noisy logs, multi-step attacks, ambiguous data and real costs for mistakes. That is where the real value of LLMs for cybersecurity will be decided.