POPE audit: the choice of "yes/no" tokens shifts LLaVA-1.5 F1 by 6.13 points

The authors of a preprint tried to reproduce a recent hallucination-reduction result on the POPE benchmark and ran into a problem with the evaluation itself. Base LLaVA-1.5-7B scored an F1 of 82.21 on the adversarial split. That is 1.21–8.67 points above the baselines they planned to compare the method against: 81.00, 79.30 and 73.54. A spread that wide looked suspicious. The cause turned up in the code: the way the "yes" or "no" answer is pulled from the logits shifts the metric on its own.

Where answer extraction breaks

The implementation with two-token readout compares the logits of two vocabulary IDs: 3582 ("yes") and 1217 ("no"). Across 9000 questions in all three splits, the model never generated either token under greedy decoding. It outputs " Yes" (3869) and " No" (1939) instead. SentencePiece adds a space byte after the "ASSISTANT:" prompt suffix, so the word gets a different ID.

The authors replaced the readout with a table of eight tokens that covers the forms with a leading space. LLaVA-1.5-7B produced these results:

• adversarial: F1 0.8221;
• popular: F1 0.8498;
• random: F1 0.8713;
• shift on adversarial from changing the readout: 6.13 points.

From that last figure, the old readout would have given roughly 76.1 on adversarial. This is our own calculation; the abstract does not give that value. The gain from changing the protocol exceeds the one claimed in five of the six papers surveyed. None of them states how it extracted the answer.

What the authors do not claim

The conclusion is cautious. The reference run used 4-bit NF4 quantization, so it is impossible to say what share of the claimed gains would hold up under identical readout and identical precision. The question stays open until both parameters match. The paper does not accuse anyone of inflating results. Its point is that comparisons without a stated protocol cannot be treated as valid.

On the corrected baseline, the authors ran nine inference-time corrections of their own. None of them beat it. They also examined why the adversarial split resists logit-level intervention. In a sample of 3000 questions, wrong answers received more attention to the image than correct ones: Cohen's d = −0.46, p = 1.4×10⁻²⁴. This contradicts the assumption behind methods that strengthen grounding in the image. Those methods assume the error comes from too little attention to the image.

A protocol problem, not a single-model problem

To check that the issue is not specific to LLaVA, the authors took four more VLMs from three tokenizer families. Carrying over the LLaVA-1.5 IDs unchanged drops F1 to zero on two models with non-overlapping vocabularies. A readout derived from the specific model's tokenizer stays in the 0.80–0.92 range. So the benchmark is sensitive to tokenization details for any model and prompt combination.

What to do when comparing models

Practical takeaways for anyone who publishes or reads results on POPE and similar yes/no benchmarks:

• Check which tokens actually appear under greedy decoding before computing any metrics.
• Build the table of valid answers from the specific model's tokenizer, including forms with a leading space and different capitalization.
• State the readout method and the weight precision (for example, NF4 or fp16) in the experiment description.
• Recompute other people's baselines with your own code instead of copying numbers from tables.

Until the protocol is fixed, a difference of one or two F1 points on POPE may reflect the choice of tokens rather than the quality of the hallucination-mitigation method.