Devin Security Gets Claude Mythos 5.2: 73.5 Points vs. 32.7 for Opus 5.5

Hunting for vulnerabilities in a large codebase is like cleaning an apartment where the owner brings in new stuff every day. Static analyzers are noisy, pentests happen once a quarter, and the security team always has a queue of a hundred tickets. Cognition proposes to tune this process with a model: Claude Mythos 5.2 is now available in Devin Security.

What exactly was launched

Devin Security is a managed scanning service. Inside, "frontier" cybersecurity models are at work, but the security team does not need separate access to them. You give it a repository, Devin runs a scan and returns the findings. Mythos 5.2 is not yet available to everyone: only to eligible enterprise customers, while others are invited to submit an application.

Separately, Cognition promises credits for free scans. In one place the post mentions $100M+, in another "hundreds of millions of dollars" in Devin credits. How these reconcile is unclear, so look at the terms in the application rather than the headlines. The company also does not disclose its selection criteria.

What the benchmark showed

Cognition ran the models on an internal set of 25 repositories in C, Python, Go, PHP and TypeScript. The frameworks include Django, Flask, Laravel and Next.js. Each repository is pinned to a commit where publicly known vulnerabilities have not yet been fixed. There are 1,781 of them in total.

Previously the benchmark checked one thing: whether Devin found a single predefined bug. Now each repository contains many bugs, so precision can be measured as well — the share of findings that are real. The final metric is F2 on a 0–100 scale. It accounts for both recall (how many known holes were found) and precision, but gives recall more weight. Each model was run three times per repository and the results were averaged.

Results:

• Claude Mythos 5.2 — 73.5
• Claude Mythos 5.1 — 56.9
• Claude Opus 5.5 — 32.7

The gain over the previous Mythos version is about 29%. Opus 5.5 trails by more than half. For a single model generation, that is a serious jump.

Why the numbers deserve caveats

First, the benchmark is internal, and it was calculated by the vendor of the service. There is no independent replication, and the dataset itself has not been published.

Second, the vulnerabilities used were ones reported publicly. The model could have encountered their descriptions or patches in its training data. The post does not discuss this, and that is my guess, but such leakage is worth checking. Finding known bugs is easier than finding unknown ones in your private code.

Third, F2 rewards recall. For security the logic is clear: a missed hole costs more than an extra check. But false positives have a price, and your triage team pays it. A single overall score does not show how much junk you will have to read by hand. It is good that Cognition included precision in the calculation, but those figures are not given separately in the post.

What to do in practice

If you are in the enterprise segment and responsible for security, submit an application. Free credits are a reason to test the service without a pilot budget.

Do not judge the result by a single score. Take two or three of your own repositories that already have known, closed bugs, and measure two things: how many of them the scan found and what share of the report you had to throw out as noise. The second figure shows what triage will cost.

Find out in advance where your code goes: how data is stored, who sees the reports and what happens to them after the scan. For private repositories this matters more than the score difference.

And do not write off the pentest. A model that finds more known classes of bugs is a useful additional layer. But business-logic vulnerabilities and non-obvious attack chains were not tested by this benchmark.

The main takeaway: the "model + managed service" combination is getting noticeably cheaper for defenders. The open question is how much 73.5 points on someone else's test will translate into findings on your code. You can check that in a couple of weeks, while the credits are being handed out.