Qwen3-1.7B Fine-Tuned on Raw, Uncaptioned Video Gains 5.1 Points on Image Benchmarks

In the arXiv preprint 2610.11019, researchers took Qwen3-1.7B, ran it through a mid-training stage on raw video from YT-Temporal-1B, and then fine-tuned it on image-text pairs in the same way as the control. The model with the video stage scored 2.9 points higher on average on four video benchmarks and 5.1 points higher on ten image benchmarks. Text skills did not drop: 48.9 versus 48.0 across 14 tests. This stage used no captions and no text loss at all.

How the experiment works

Multimodal LLMs are usually trained on image-text pairs or annotated video. Raw web video is rarely used to fine-tune an existing language model. The authors tested whether that can change.

Frames are encoded into continuous visual tokens, and the language model learns to predict the next visual token. It is the same next-token prediction principle, except the model predicts frames instead of words. This stage has no text component at all.

The control is set up carefully. The model with video mid-training and the model without it received the same image-text instruction tuning. The only difference between them is the intermediate video stage. That means the gain can be attributed to that stage rather than to differences in the final tuning.

What the numbers show

Main results compared with the model without mid-training:

• video benchmarks: +2.9 points on average across four tests;
• image benchmarks: +5.1 points on average across ten, including perception, document and chart tasks;
• text benchmarks: 48.9 versus 48.0 on average across 14 tests;
• base model: Qwen3-1.7B, data: clips from YT-Temporal-1B.

The gain on images turned out to be noticeably larger than on video itself. It is particularly interesting that the image set includes documents and charts, tasks far removed from YouTube clips. Presumably, next visual token prediction teaches the model general visual representations that then transfer to static images. The authors do not state this interpretation explicitly in the abstract; it is our reading of the numbers.

Another result concerns training dynamics. Improvements on images and video appear in the first 30% of training, then plateau and fluctuate by less than 0.5 points. Judging by these curves, a longer run on additional data adds almost nothing.

Finally, the authors compared their approach with caption prediction. Caption prediction did not outperform next visual token prediction. This means video mid-training can stay fully self-supervised, with no compute spent on automatically captioning clips and none of the noise that generated captions introduce.

Who this is useful for

For teams building their own multimodal models, the takeaway is practical. Huge volumes of unlabeled video can serve as a cheap intermediate stage between text pretraining and instruction tuning. It needs no annotation pipeline and no separate captioning model. The early plateau suggests the budget for this stage can be capped.

Some caveats apply for now. This is a preprint, and it was tested on a single small 1.7B-parameter model and a single video dataset. The 0.9-point difference on text tests points to an absence of degradation rather than an improvement. Whether the effect scales to larger models and other video sources is not clear from the abstract. That is what should be tested next.