GRAPHEVAL: New Graph Framework Reveals That Self-Consistency Masks Reasoning Errors in LLMs

Researchers published a paper on arXiv in July 2026 with the GRAPHEVAL framework for assessing the reliability of language model reasoning. Main finding: the standard Self-Consistency method systematically fails to detect logical errors because it compares only final answers, completely ignoring intermediate reasoning steps.

Why Self-Consistency fails to catch reasoning errors

Self-Consistency (SC) is one of the most popular ways to improve language model reliability. The principle is simple: the model generates multiple solution variants for one task, then majority voting selects the most frequent final answer. This filters random mistakes, but leaves a blind spot.

Several chains of reasoning can reach the same answer in completely different — including erroneous — logical paths. The model appears confident, consensus is reached, but the structure of reasoning remains incoherent. The authors call this phenomenon "confident hallucinations": the model does not express uncertainty, yet its intermediate steps don't withstand logical coherence checks.

What GRAPHEVAL proposes

The framework reconsiders the task of uncertainty quantification as measuring the integral logical reliability of the entire reasoning space, not just final answers. The authors introduce two new tools:

• GRCS (Graph Reasoning Coherence Score) — a metric capturing semantic-structural consensus in graph representation of reasoning; simultaneously detects pathological mode collapse and confident hallucinations.
• GSC (Graph Self-Consistency) — a decoding strategy based on medoid selection: instead of voting on the final answer, it chooses the reasoning chain most structurally similar to all others.

GRCS proved to be the only metric consistently negatively correlated with reasoning incorrectness — for both large and small models.

What the experiments showed

The authors tested the framework on models of different scales and identified several key patterns:

• For small models, GSC clearly reveals how much SC overestimates accuracy through "lucky guesses": models often reach the correct answer via an erroneous path, and SC doesn't notice this.
• For more powerful models, GSC preserves or improves accuracy while simultaneously increasing logical reliability of reasoning.
• Adversarial medoid ablation confirmed: the path chosen by GSC is "load-bearing" — forcing deviation from it reduces reasoning reliability and in some cases worsens answer accuracy.

What this means

Standard accuracy metrics and Self-Consistency mask a fundamental problem: language models can guess the right answer without being able to reason correctly. GRAPHEVAL provides tools that distinguish "ability to guess" from "ability to think" — and this changes the framework for comparing and evaluating language models in academic and applied contexts.

Frequently asked questions

How does GSC differ from standard Self-Consistency?

Standard SC takes the answer of the majority from all generated solution variants. GSC selects the reasoning chain that is structurally most similar to all others (medoid) — this allows evaluating the logical path to the answer, not just the final result.

Is GRAPHEVAL applicable to models of any size?

The authors tested the framework on models of different scales. GRCS showed stable negative correlation with reasoning incorrectness across all tested models. GSC effect varies: for small models it reveals SC's overestimated accuracy, for more powerful ones it improves both metrics simultaneously.