IBM study: asking the client questions first lifts smart-contract transaction fidelity to 72.9%

Agentic systems that turn an English description into code have become routine, and the code usually compiles. But almost nobody uses validated tools to measure whether the program's behavior matches the author's intent. Researchers at IBM T.

J. Watson Research Center studied this on Solidity smart contracts: they cannot be patched after deployment and they handle money, so mistakes are costly. The study's main metric is transaction fidelity, the share of replayed transactions on which the generated contract behaves the same way as the original.

A pipeline that first asks the client questions scored 72.9% on 500 held-out contracts. Single-prompt generation scored 53.

9%, and the authors' earlier seven-stage pipeline scored 39.3%. Most of the intent is lost before generation, and clarifying requirements helps most, not extra verification stages.

How fidelity was measured

The authors took 600 real contracts from the FSM-SCG dataset. For each one, Gemini 2.5 Flash wrote a short description of about 140 words and a full requirements document of about 1,700 words. The main metric is implementation fidelity, meaning how accurately the original's behavior is reproduced. For each contract, they ran 60 random sequences of 8 steps each in a local EVM virtual machine. A step counts only if both the call outcome and all observable values match. The authors note that the metric reflects the author's intent only to the extent that the original contract embodies it.

They first tested the evaluation tools themselves on injected bugs. The Slither static analyzer flagged a new issue after only 3 of 200 removed access checks. An LLM judge that reads the code marks 80% of untouched contracts as problematic, and requiring executable proof cuts the share of bugs it finds to 30%. A reference model in Python, written from the description, catches 94.3% of typical bugs such as removed modifiers. But the same model, written from function names alone, catches 87.5%. So a significant part of classic evaluations rewards knowledge of common conventions, not understanding of the description.

Where intent gets lost

On 100 fresh contracts and 279 contract-specific bugs, the reference model found 65.9% of bugs from the short description and 87.5% from the full document. The gap was 21.6 points. An extractor agent and a critic agent changed nothing. A checklist of ten fixed client questions closed the gap almost entirely: 89.4% versus 88.9% for the full document.

For the GPT-5.4-mini generator, transaction fidelity is 55.6% with the short description, 76.0% with checklist answers, 79.4% with interviewer answers and 91.0% with the full document. The checklist has a weakness: it describes behavior but not argument types and stored values, so it implements 88.7% of the required signatures versus 94.8% for the interviewer. The gain held across other model families: Claude Sonnet 5 rose from 56.1% to 75.1%, and Gemini 3.5 Flash from 51.8% to 74.5%. Across the three model families, the authors estimate the benefit of simulated-client answers at 19–23 points.

Descriptions are unreliable too. An auditor model found at least one contradiction with the original in 76 of 100 descriptions, for example a nonexistent access rule. So clarification often corrects false statements rather than just filling gaps. Among the 24 descriptions without contradictions, the checklist gain persists, but the sample is too small for a rigorous comparison.

How many questions are needed and what doesn't work

Three to five carefully chosen questions recover 60–75% of the total gain. Post-generation stages add no more than one point, and an automatic security-fix loop costs 12.8 points: it cuts out needed behavior along with the vulnerabilities. The final pipeline first asks the client questions, then checks the contract against an independent reference model and shows the client the discrepancies as a blind choice. It only flags security findings and does not rewrite the code.

The limitations are significant. The client is simulated by GPT-5.4, which reads the original contract, so the gain is an optimistic estimate of what real authors would provide. Real people will answer less completely and less accurately. The intervals do not account for variation between runs, and 36 criteria were tested without correction for multiple comparisons. The authors are publishing the test harness, the 600 tasks and all scores.

The practical takeaway for anyone building code-generation agents: invest in clarifying requirements before generation, and do not trust LLM judges or automatic security "repair" without testing them on injected bugs.