Hugging Face explains how to fine-tune multimodal embedding and reranker models

Hugging Face has shown something important for applied AI: you don't have to swap a multimodal retrieval model for a bigger one to get a noticeable quality gain. In a new Sentence Transformers guide, the team walks through how to train and fine-tune embedding and reranker models that work not only with text but also with images, audio and video. The main idea is simple: if a company already has a general-purpose multimodal checkpoint, it can adapt it to its own task and gain more than it would by moving to a heavier general-purpose model.

As a practical example, the authors took the Visual Document Retrieval task, where a text query must find the right document page in the form of a screenshot. In this scenario the model has to understand not just words but also page structure, tables, charts, captions and visual layout.

For the experiment they used Qwen3-VL-Embedding-2B and fine-tuned it on the English subset of the LlamaIndex dataset. The original set contains about 500 thousand multilingual query-image pairs; the version prepared for the experiment keeps 53,512 English examples. The first 10 thousand records were used for training and the next 300 for evaluation.

The pipeline itself differs little from ordinary text training in Sentence Transformers. The article stresses that the trainer, training arguments and dataset loading stay the same, and the key differences relate to multimodality: the model is loaded with processor_kwargs and model_kwargs to control image-processing quality, computation precision and the attention implementation; data can contain strings, images, audio, video or dictionaries with several modalities; and preprocessing runs automatically through model.preprocess().

For the main task, the author used CachedMultipleNegativesRankingLoss with mini_batch_size=1, so a large VLM could be trained without running out of memory while keeping the benefits of a large effective batch size through gradient caching.

MatryoshkaLoss gets separate emphasis. This wrapper around a base loss function trains the model so that useful information concentrates in the early dimensions of the embedding. In practice, this lets you shrink the vector size at deployment without a sharp drop in search quality. For Qwen3-VL the full embedding size is 2048 dimensions, but after this training the model loses almost no quality even when truncated to 512 dimensions. Moreover, the final version's configuration was saved with truncate_dim=1024, so by default it outputs vectors half the size of the full ones, lowering storage and index requirements.

The results look convincing even without long caveats. After one epoch, the fine-tuned version scored NDCG@10 of 0.947 on the evaluation set, while the base Qwen3-VL-Embedding-2B scored 0.888. In the comparison table, this 2-billion-parameter model beat not only its original version but also larger systems, including Qwen3-VL-Embedding-8B at 0.923 and several other current multimodal solutions. The author also showed that at 512 dimensions the fine-tuned model holds at 0.945, staying almost at its peak, and even at 64 dimensions it retains more than 92% of its maximum quality. For teams that count index cost and latency, this is not a detail but a practical argument for the approach.

Finally, Hugging Face notes that the same stack can also train multimodal reranker models. It uses CrossEncoderTrainer and specialized loss functions, and in the any-to-any reranker example the model is trained to decide whether an image matches a text, returning a binary score. This matters because in real search systems the retriever and reranker often work as a pair: the first quickly selects candidates, the second precisely reorders the results.

What this means: the era of "take the biggest multimodal checkpoint and hope for the best" is ending. Hugging Face shows a more pragmatic path: take an available model, fine-tune it on your own domain, stay compatible with the familiar Sentence Transformers pipeline and, if needed, shrink the embeddings without a noticeable drop. For teams building search over documents, catalogs, media archives or internal knowledge bases, this is a direct signal: the quality of multimodal search is increasingly determined not by model size as such, but by the quality of domain tuning.