Evidence Chain Evaluation: An AI Fact-Checker That Is Allowed to Say "Not Sure" Instead of Issuing False Verdicts

On July 23, 2026, researchers published the Evidence Chain Evaluation (ECE) framework on arXiv. It is a fact-checking system that can answer "not sure" instead of a forced "true/false," and it holds 97.8% accuracy on the claims for which it does issue a verdict.

What Is Evidence Chain Evaluation

Evidence Chain Evaluation is a selective fact-checking framework: it allows an AI agent to abstain from a decision when evidence is weak, fragmentary, or contradictory. The system under test is a tool-using agent that gathers evidence through web search, scientific search, and executable checks, then returns a structured verdict with a confidence score and source metadata.

• Accuracy on answered claims (selective accuracy): 97.8% on the ECE-Bench benchmark
• Standard accuracy: 91.6%; coverage: 93.7%
• The system deferred 6 of 95 claims instead of guessing
• 5 of the 6 deferred cases involved the lowest-reliability sources (level L4)
• Agent tools: web search, scientific search, executable checks; the code is open on GitHub

Why the Right to Stay Silent Matters

Large language models can show high fact-checking accuracy, but a forced binary choice hides a reliability problem: the system issues a confident verdict even where there is almost no evidence or the evidence is contradictory. ECE addresses this with an "uncertain" verdict, an abstention that triggers specifically in weak-evidence zones rather than being spread across all claims.

The concentration of abstentions in unreliable zones is the authors' key argument. According to the paper, 5 of the 6 deferred cases involved sources at the lowest reliability level (L4). This supports the claim that abstention works as a safety valve, not as a way to dodge hard questions.

"Abstention works as a safety-oriented mechanism for handling

epistemically weak evidence," the arXiv paper states.

Where ECE Falls Short of Baselines

ECE does not outperform the strongest retrieval baseline on aggregate calibration metrics: Expected Calibration Error, Brier score, and AURC. The authors say so openly. ECE's advantage lies elsewhere: the system offers a clear selective-prediction trade-off, with very high accuracy on answered claims at the cost of refusing only 6 of 95 cases.

In other words, ECE sacrifices completeness for reliability where it matters most. For fact-checking, where a falsely confident verdict is more dangerous than an honest "I don't know," that trade-off is often justified.

What This Means

An AI system's ability to honestly admit a lack of evidence is a practical step toward trust in automated fact-checking. ECE shows that calibrated refusal can be built into an agent and directed precisely at the places where sources are weakest, without dragging down overall accuracy.

Frequently Asked Questions

What is Evidence Chain Evaluation?

It is a selective fact-checking framework in which an AI agent can return an "unsure" verdict instead of a forced "true/false." The system gathers evidence through web search, scientific search, and executable checks, and attaches a confidence score and source metadata to each verdict.

How accurate is ECE?

On the ECE-Bench benchmark, the system showed 97.8% accuracy on answered claims, 91.6% standard accuracy, and 93.7% coverage, deferring 6 of 95 claims.

Where can I get the ECE code?

The framework's code is open: the authors published it in the cheshireyang/ECE repository on GitHub.