Mycelium: Researchers Break Down Hallucination Benchmarks Along Three Axes

LLM hallucination detectors are typically compared using a single number on a single benchmark. In their meta-study Mycelium (arXiv 2610.33608, published October 24, 2026), Julia Belikova and Elisei Rykov show why that single number says less than it appears to. Benchmarks define hallucination differently, obtain model outputs through different methods, and label errors at different levels. The authors argue that detection should be treated as a measurement task that depends on the benchmark's conditions, and they introduce a new metric, Span Coverage F1.

One Term, Many Different Tasks

Hallucinations matter for evaluating and deploying large language models — how reliably they're caught affects RAG systems, assistants, and enterprise pipelines. New detectors usually arrive with a results table across several datasets. But if one dataset labels factual errors without context while another checks faithfulness to a given source, these are two different tasks sharing the same name. A high score on the first promises almost nothing about the second.

Mycelium breaks down how choices made during benchmark construction, annotation, and evaluation shape the resulting prediction task. The authors then map the mechanisms and assumptions of the detectors themselves against the conditions under which those detectors were actually measured.

Three Axes for Classifying Benchmarks

The authors sort all existing benchmarks along three dimensions:

• Provenance: how the model outputs were obtained and the conditions around the evidence — that is, where the texts came from and what evidence the verification relies on.
• Typology: which failures of factuality and faithfulness to the source are reflected in the benchmark's labels.
• Granularity: at what level hallucinations are annotated, predicted, and scored. This can be a full response, a sentence, or an individual span of text.

Different positions along these axes change the task itself. As a result, the authors conclude, comparisons between benchmarks have real limits, and a detector's generalization to other conditions must be tested explicitly rather than assumed.

Span

Coverage F1: When Annotation Boundaries Decide Everything

A separate section is devoted to span-level evaluation — that is, text fragments labeled as hallucinations. Here, the outcome depends heavily on exactly where the annotator drew the boundaries. One benchmark might flag a short noun phrase; another flags the entire sentence. A detector that found the same error will receive different scores depending on how closely its boundaries match.

As a complement to existing metrics, the authors propose Span Coverage F1. It's based on containment: it checks whether the predicted span covers the labeled one, rather than measuring only exact overlap. The metric is designed to be complementary — it doesn't replace standard measures, but helps separate genuine detection quality from annotation artifacts.

How to Read Detector Scores

A practical takeaway for teams choosing or training detectors:

• Before comparing two scores, check whether the benchmarks match on data provenance, error types, and annotation level. If they don't, the numbers are measuring different things.
• For production use, pick the benchmark closest to your actual scenario. For RAG, faithfulness to the source matters most; for open-ended questions, context-free factuality matters most.
• If a detector operates at the span level, check how boundaries are defined in the annotations, and add a containment-based metric like Span Coverage F1.
• Test for generalization: run the detector on benchmarks with different conditions and report the spread, not a single number.

The code and research materials are available in the lmeribal/mycelium repository on GitHub. The authors are asking for feedback and discussion — the work is meant to serve as a foundation for fairer comparisons in a field where a number in a table often carries more weight than the context in which it was obtained.