Dafny accepts 99.1% of Opus 4.7's HumanEval code, but only 1.9% is bound to the specification

When a code-generation pipeline reports that "N% of solutions are verified," that number can hide an almost empty guarantee. Prasenjit Dey, Ashish Jagmohan and Ravi Kokku have published a paper on arXiv titled "Verified but Vacuous." They propose breaking "verified LLM-written code" into three independently measurable levels, and show with data that these levels diverge sharply. In one experiment, the Dafny verifier accepted 99.1% of HumanEval solutions, yet only 1.9% had a postcondition bound to the specification.

Three failure surfaces

The authors call them surfaces A, B and C. Surface A: the verifier accepted the proof (for Dafny, a zero exit code). This is a necessary condition, but the weakest one. Surface B: whether the statement being proved is tied to anything meaningful. At least one postcondition (ensures) must reference a predicate or function from the paired specification. Surface C: whether the specification itself reflects the user's original requirement.

The audit runs as a chain: B is computed on the samples that passed A, and C on the samples that passed B. The authors note that these are conditional samples, not a logical hierarchy of properties.

For surface B, they built an ensures-binding gate, a static AST traversal that takes milliseconds per component. For surface C, they use the Post2Exe check from the VeriContest benchmark: the postcondition is compiled into an executable predicate and run against positive and negative tests. If the predicate accepts an answer that a negative test should reject, the specification is incomplete.

Two examples on the task "return the maximum of an array" show the difference. The verifier accepts the method r := 0 with no ensures: there is simply no functional guarantee. If an ensures references the predicate r >= 0, the binding formally exists, but the specification is too weak, and only a behavioral check catches it.

What the Li et al. reproduction showed

The team reproduced the Li et al. pipeline, in which Dafny serves as an intermediate representation, on Claude Opus 4.7. They tested three Python benchmarks of LeetCode-level tasks: HumanEval, LeetCodeDataset and LiveCodeBench v6. The verifier accepted 92–99% of samples, while the share bound to a predicate from the specification stayed very small. On HumanEval it was 1.9%. Another 3.4% of samples had a postcondition that referenced no predicate from the specification. By the authors' estimate, the gap between acceptance and binding is roughly forty-fold.

The share of samples with a postcondition grows with difficulty: from 5.3% on HumanEval to 10.61% on LiveCodeBench v6 (1 032 of 9 731; 10.9% on the subset of 9 448 samples that passed tests). The model writes more postconditions on harder tasks, but the number bound to the specification does not increase as a result.

The authors stress that this is not a defect of the Li et al. pipeline. The few-shot prompt in their paper contains a method with no body and no postcondition, plus a fixture that checks expectations at runtime. With this kind of input, verifier acceptance only means the remaining static obligations are correct, not that the code is functionally correct. The error, in their words, lies in how such a pipeline is usually described.

Even by construction, verified does not mean correct

The authors' own pipeline, VCG, is built differently: a Planner splits the task, a SpecAgent writes Dafny specifications, an ImplementationAgent writes implementations, and a ProverAgent runs a verify-and-repair loop with the ensures-gate. Verified components are transpiled to Rust. Binding to the specification holds for 100% of components there by construction. Even so, 11.48% of samples on MBPP-Rust and 14.82% on the LC slice of VeriContest failed hidden tests. The authors call this an operational approximation of conformance to requirements, not a precise diagnosis of specification errors.

On the LC slice of VeriContest, VCG's end-to-end End2End metric is 7.79% (535 of 6 870). The published external figures are 5.51% for GPT-5.5 and 1.88% for a single run of Opus 4.7. That is a difference of 2.3 percentage points and a factor of 4.14. But task budgets, the number of repair attempts and test harnesses differ (VCG uses ten samples per task, the external comparison one), so the authors do not claim a like-for-like advantage.

Limitations and practical takeaway

The paper gives point estimates without confidence intervals. The ensures-gate itself is crude: it may reject a meaningful inline postcondition, while letting through a tautology with an impressive-sounding name. The reported 27% of structural failures in the pre-release audit of VCG and 98% for Li@Opus have not been checked by independent labeling for semantic vacuity.

The practical recommendation is concrete. Any pipeline that issues certificates should publish at least one of metrics B or C alongside verifier acceptance, and preferably both. B alone leaves a semantic blind spot, as with VCG. C alone leaves a syntactic one, as with Li@Opus. For teams planning to ship LLM-produced code with "formal verification" to production, this is a ready-made checklist of what to ask the vendor.