Evidence Chain Evaluation: как ИИ-фактчекеру разрешили отвечать «не уверен» вместо ложных вердиктов

Researchers published the Evidence Chain Evaluation (ECE) framework on arXiv on July 23, 2026 — a fact-checking system that can answer "uncertain" instead of a forced "true/false" and holds 97,8% accuracy on claims for which it still issues a verdict.

What Evidence Chain Evaluation Is

Evidence Chain Evaluation is a selective fact-checking framework: it allows an AI agent to abstain from a decision when the evidence is weak, scattered, or contradictory. The system under test is a tool-using agent that gathers evidence via web search, scholarly search, and executable checks, then returns a structured verdict with a confidence score and source metadata.

• Selective accuracy on answered claims — 97,8% on the ECE-Bench benchmark
• Standard accuracy — 91,6%, coverage — 93,7%
• The system deferred 6 out of 95 claims instead of guessing
• 5 out of 6 deferred cases involved sources of the lowest reliability tier (level L4)
• Agent tools — web search, scholarly search, executable checks; the code is open on GitHub

Why the Right to Stay Silent Matters

Large language models can show high fact-checking accuracy, but a forced binary choice hides a reliability problem: the system issues a confident verdict even where evidence is nearly absent or contradictory. ECE addresses this with an "uncertain" verdict — a refusal that fires precisely in zones of weak evidence, rather than being spread across all claims.

The concentration of refusals outside reliable zones is the authors' key argument. According to the paper, 5 out of 6 deferred cases involved sources of the lowest reliability tier (L4). This confirms that abstention works as a safeguard, not as a way to dodge hard questions.

"Abstention functions as a safety-oriented mechanism for handling

epistemically weak evidence," the arXiv paper states.

Where ECE Falls Short of Baseline Methods

ECE does not outperform the strongest search baseline on aggregated calibration metrics — Expected Calibration Error, Brier score, and AURC. The authors admit this directly. ECE's advantage lies elsewhere: the system offers a clear selective-prediction trade-off — very high accuracy on answered claims at the cost of deferring just 6 cases out of 95.

In other words, ECE sacrifices completeness for reliability where it matters most. For fact-checking, where a false confident verdict is more dangerous than an honest "I don't know," that trade-off is often justified.

What This Means

An AI system's ability to honestly admit a lack of evidence is a practical step toward trust in automated fact-checking. ECE shows that calibrated refusal can be built into an agent and directed exactly where sources are weakest, without collapsing overall accuracy.

Frequently Asked Questions

What Is Evidence Chain Evaluation?

It is a framework for selective fact-checking in which an AI agent can return an "uncertain" verdict instead of a forced "true/false." The system gathers evidence through web search, scholarly search, and executable checks, and attaches a confidence score and source metadata to the verdict.

How Accurate Is ECE?

On the ECE-Bench benchmark, the system achieved 97,8% accuracy on answered claims, 91,6% standard accuracy, and 93,7% coverage, deferring 6 out of 95 claims.

Where Can I Get the ECE Code?

The framework's code is open: the authors published it in the cheshireyang/ECE repository on GitHub.