IBM: Asking the Client Questions Before Generation Raised Smart Contract Accuracy From 54% to 73%

Agents built on large language models can turn a plain-English description into code that compiles and runs. But whether that code does what its author wanted is usually not measured in any serious way. For smart contracts, the question is pressing: these are programs on a blockchain that hold and transfer money themselves, and they cannot be fixed after launch.

Researchers at IBM T.J. Watson Research Center took 600 existing Solidity contracts, had a model describe each one in words, and then asked AI agents to write the contract again.

Accuracy was measured as the share of transactions on which the new contract behaves the same way as the original. The best pipeline scored 72.9% on 500 held-out contracts.

One-shot generation scored 53.9%, and the authors' own previous pipeline scored 39.3%.

What helped most was asking the client questions before writing any code.

What They Actually Measured as "Understanding"

The first part of the work examines the measuring instruments themselves, and the findings are uncomfortable. The researchers manually introduced a single wrong rule into real contracts and checked whether a reviewer would notice.

• The static analyzer Slither, which is tuned to known classes of vulnerabilities, raised a new finding in only 3 of 200 cases where an access check had been removed. • A language model acting as a judge and reading the code finds many bugs, but it flags 80% of untouched contracts. If it is required to produce an executable proof, false alarms drop, but it catches only 30%. • A reference model in Python, written from the description, catches 94.3% of typical bugs such as a removed modifier or a flipped comparison. But the same kind of model, written from function names alone, catches 87.5%.

That last figure matters. It means most of what ordinary tests mistake for understanding the task is knowledge of common patterns, which any strong model already has. So the authors went on to evaluate only rules specific to a particular contract.

Replication accuracy was also computed strictly. From each contract, the researchers generated 60 random sequences of 8 calls. The same calls were run on the original and the copy, and a step counted only if the outcome matched and the same observable values changed. A missing function or a different signature meant a failed step.

Where the Intent Gets Lost

A short description of 140 words leaves out many details. The check showed that 76 of 100 such descriptions contain at least one direct contradiction of the contract, for example "the administrator can open the auction" when in fact anyone can. On average, 14.9 rules are missing. A full description of 1700 words eliminates this problem.

In the experiment, the reference model caught 65.9% of contract-specific bugs from the short description and 87.5% from the full one. The gap was 21.6 points. Contracts written by the generator (GPT-5.4-mini, 92 contracts) showed the same pattern:

• short description — 55.6%; • plus answers to an interviewer agent's questions — 79.4%; • plus answers to ten fixed questions — 76.0%; • full description — 91.0%.

With Claude Sonnet 5, the result rose from 56.1% to 75.1%, and with Gemini 3.5 Flash, from 51.8% to 74.5%. An example from the paper: the description of a liquidity management contract lists states and operations but does not say who can call them. The model added an owner role, and the contract matched only 25% of transactions. A single client answer, "all operations are open to any account," raised the match to 100%.

Checks after generation barely help: all the tested stages combined add about one point. An automated loop that rewrites code based on a security auditor's findings subtracts 12.8 points, because it cuts out needed behavior along with the risky behavior.

How Many Questions You Need, and the Caveats

The pipeline with questions first interviews the client using a checklist. It then compares the contract against an independently written reference model, and shows each discrepancy to the client as a blind choice between two options. For security findings, it only reports them and does not rewrite the code. Three to five well-chosen questions are enough to recover 60–75% of the gain.

The main caveat: the client was played by GPT-5.4, which had the original contract in front of it. The gain of 19–23 points across three generator families is therefore an optimistic estimate of what a live author would deliver. In addition, the originals are imperfect too: in the source dataset, 35.1% of 21 976 contracts contain stubs and 27.8% have function bodies consisting only of comments; these were filtered out. The gain held on 24 descriptions without contradictions, although subgroups cannot be told apart on a sample that small.

For engineers, the takeaway is simple. If an agent writes code that moves money, it is worth investing first in clarifying requirements rather than in additional verification stages. Automated "repair" based on security reports should be measured separately: it can break needed behavior. The authors have published the test harness, the tasks and all the scores.