Omni-Woodpecker Cuts Hallucinations in Omni-LLMs Without Retraining: +12.9 on CMM

Omni models that listen to audio and watch video at the same time have learned to reason convincingly, but they also err convincingly. A group of researchers from Nanjing University (Zijian Liu, Chaoyou Fu, Yan Yang, Caifeng Shan, Ran He) has proposed Omni-Woodpecker, or OWP. The method reduces cross-modal hallucinations without any retraining. On Qwen2.5-Omni-7B, it adds 12.9 points on the CMM benchmark and 8.6 on AVHBench. The paper appeared on arXiv under number 2610.27103 and made it into Hugging Face Daily Papers.

Where the audio-visual link breaks in omni models

A cross-modal hallucination occurs when the audio and visual tracks say different things, or when one of them gives no clear cue. The model picks not what was actually captured in the input but what "usually happens" statistically: a familiar combination of objects and sounds from the training data outweighs the facts of the specific clip.

Most existing remedies work at the output level: they adjust the token distribution during decoding or filter the finished answer. The OWP authors consider this treating symptoms. By the time of decoding, evidence from the input and learned priors are already mixed together in the fused representations, and they are hard to pull apart at the output.

Three steps inside the hidden states

OWP moves the intervention from the output to the internal representations and works in three stages:

1. Conflict detection. The method identifies contradictions between modalities by comparing modality-specific, complementary views of the input. 2. Orthogonal decomposition. In the hidden states, the method separates the component grounded in real evidence (grounding evidence) from the competing priors. 3. Adaptive rebalancing. The fused representations are recalibrated: the contribution of misleading priors is suppressed, while ignored evidence is amplified. All of this happens before decoding.

The logic is as follows: if you can isolate the direction in latent space along which the model "fills in the blanks," you can weaken it surgically, without touching the weights and without launching a new training run. Hence the training-free label, which matters more to practitioners than any metric: no annotation and no GPU hours for fine-tuning are needed.

What the measurements showed

The authors tested the method on two architectures and several sets of tasks:

• Qwen2.5-Omni-7B, CMM benchmark: +12.9 points.
• Qwen2.5-Omni-7B, AVHBench benchmark: +8.6 points.
• Qwen2.5-Omni-7B, four general OLLM reasoning benchmarks: a consistent gain of 0.67 to 2.25 points.
• VideoLLaMA2-AV, the same two hallucination benchmarks: +4.6 and +3.9 points.

The ratio is telling. On dedicated hallucination tests the gain is double-digit; on general reasoning tasks it is modest but non-negative. Suppressing priors does not break the model's abilities, which is a common price of methods like this. The transfer to VideoLLaMA2-AV, which has a different architecture, suggests the approach is not tied to Qwen's design, although the gain there is lower.

Who can use it right now

The code is published in the Omni-Woodpecker repository (about 94 stars at the time of publication), so you can test the method on your own stack. The biggest winners will be teams building products on omni models where errors are costly: video call and meeting recording analysis, video surveillance with audio, automatic video descriptions, and media monitoring assistants. In those settings, the model most often "hears" something that isn't in the frame, and vice versa.

The practical order of steps is simple. First, run OWP on your own data with deliberately conflicting audio and video, not just on public benchmarks. Then measure latency and memory use: analyzing conflicts across individual modalities probably adds compute at inference. Finally, compare it with decoding methods from the previous generation, since they can be combined with intervention in hidden states. The experiments in the paper are limited to 7B-class models, and independent tests will show how the method behaves on larger omni systems.