Mycelium: Researchers Sort Hallucination Benchmarks Along Three Axes

LLM hallucination detectors are usually compared by a single number on a single benchmark. In Mycelium, a meta-study (arXiv 2610.33608, published on 24 October 2026), Julia Belikova and Elisei Rykov show why that number says less than it appears to. Benchmarks define hallucination differently, obtain model outputs in different ways, and annotate errors at different levels. The authors propose treating detection as a measurement problem that depends on benchmark conditions, and they introduce a new metric, Span Coverage F1.

One Term, Many Different Tasks

Hallucinations matter for evaluating and deploying large language models: RAG systems, assistants, and enterprise pipelines all depend on how reliably they are caught. New detectors usually arrive with a table of results on several datasets. But if one dataset labels factual errors without context and another checks whether an answer is faithful to a provided source, these are two different tasks that share a name. A high score on the first promises almost nothing about the second.

Mycelium examines how choices made during benchmark construction, annotation, and evaluation shape the final prediction task. The authors then match the mechanisms and assumptions of the detectors themselves against the conditions under which those detectors were measured.

Three Axes for Classifying Benchmarks

The authors sort all existing benchmarks along three dimensions:

• Provenance: how model outputs and evidence conditions were obtained, that is, where the texts came from and what evidence the verification relies on.
• Typology: which factuality and faithfulness failures the benchmark labels capture.
• Granularity: the level at which hallucinations are annotated, predicted, and evaluated. This can be the whole response, a sentence, or an individual text fragment.

Different positions on these axes change the task itself. The authors therefore conclude that comparisons across benchmarks have limits, and that a detector's generalization to other conditions should be evaluated explicitly rather than assumed.

Span

Coverage F1: When Annotation Boundaries Decide Everything

A separate section covers span-level evaluation, where a span is a fragment of text labeled as a hallucination. Here the outcome depends heavily on exactly where the annotator drew the boundaries. One benchmark marks a short noun phrase, another the whole sentence. A detector that finds the same error will receive different scores depending on how the boundaries line up.

As a complement to existing metrics, the authors propose Span Coverage F1. It is based on containment: it accounts for whether the predicted fragment covers the annotated one, not only for the degree of exact overlap. The metric is designed to be complementary. It does not replace the familiar measures, but helps separate real detection quality from annotation artifacts.

How to Read Detector Scores

Practical takeaways for teams that choose or train detectors:

• Before comparing two scores, check whether the benchmarks match in data provenance, error types, and annotation level. If they do not, the numbers measure different things.
• For production, pick the benchmark closest to your use case. For RAG, faithfulness to the source matters; for open-ended questions, factuality without context.
• If a detector works at the span level, look at how the annotation defines boundaries, and add a containment-based metric such as Span Coverage F1.
• Test generalization: run the detector on benchmarks with different conditions and report the spread, not a single number.

The code and research materials are in the lmeribal/mycelium repository on GitHub. The authors invite feedback and discussion. The work is meant as a basis for more honest comparisons in a field where the number in a table often weighs more than the context in which it was obtained.