Linear Head on Qwen3-VL Beats GPT-4o at Scoring Answers Without Human Labels

Researchers have shown that a multimodal reward model does not need annotators. A linear head on an intermediate layer of a frozen Qwen3-VL learns to rank a web caption of an image above a corrupted version of that caption, and nothing else. At 32B, the same recipe, with no retuning, matches or beats GPT-4o on three benchmarks and leads on VL-RewardBench by 6.6 points. The number of trainable parameters is a few thousand, and the compute is roughly 1,000 times lower than for fine-tuning.

How the recipe works

Reward models for vision-language are usually trained on labeled preference pairs. That labeling is expensive, annotators disagree with each other, and quality is then measured as accuracy on a benchmark. The authors take humans out of the training loop. They take an ordinary image caption from the web, corrupt it (the abstract does not disclose details of the sampler), and train the head to prefer the original. Labeled data is needed in only one place: to choose which layer to take features from.

That choice turned out to be decisive. An intermediate layer scores 5.8 points higher than the last one. The final layers are apparently too specialized for next-token prediction, while the middle of the network better preserves the difference between a correct and an incorrect description of an image.

What the numbers show

The main results from the paper:

• at 32B, the model matches or beats GPT-4o on three benchmarks;
• on VL-RewardBench the lead is 6.6 points;
• only a few thousand parameters are trained, at a cost of about 1/1000 of fine-tuning;
• with an equal budget of pairs, perturbations give a frozen head 3–5 points more than curated RLAIF-V pairs;
• the recipe transfers across benchmarks without retuning.

Because the backbone is frozen, training the head is a convex problem. The result does not depend on a lucky initialization or random optimization noise, which makes it easier to reproduce.

A benchmark does not tell you what you actually measured

The most interesting part of the paper concerns the benchmarks themselves. The reward is defined by the perturbation sampler, not by a leaderboard rank. The authors trained two heads on non-overlapping halves of the same perturbations. On the benchmark, the two scored the same. On hallucination evaluation, they diverged by 6.4 points. Two models with the same leaderboard number behave differently where it matters for a product.

There is also a counterexample. The head was retrained on the policy's own candidates, labeled by a verifier. Hallucinations in captions, as measured by CHAIR, fell by 5.1 points, and the share of successful image generations rose by 5.4. On all reward benchmarks, however, this head scored worse. The gain on the target task came with a drop in the table, because the table was measuring something else.

Practical takeaway

A team doing RLHF or RLAIF for a multimodal model would do well to try the cheap option first: a linear probe on the middle layers of a frozen model, trained on automatically generated "original vs. corruption" pairs. It costs next to nothing compared with fine-tuning and gives a strong baseline without hiring labelers. The layer should be chosen on a small labeled sample; it is the most sensitive parameter.

The second takeaway concerns evaluation. A single benchmark will not show what behavior is encoded in the reward. If you need a reward model to fight hallucinations, test it separately on hallucinations and, where possible, fine-tune it on the policy's own candidates with a verifier. A caveat: all figures come from an arXiv preprint and have not yet been independently verified, and the details of the perturbation sampler are best checked in the full text of the paper.