Researchers Extend DINOv2's CLS Token with Regions of Interest for Image Retrieval

The authors of preprint arXiv:2610.10991 propose a way to get more out of DINOv2-reg than a single [CLS] vector provides. To the model's five semantic tokens (the global [CLS] and four register tokens), they add local ROI tokens. Retrieval uses a multi-vector scheme inspired by ColBERT. Instead of one descriptor per image, the system produces a small set of them, and it does not need to store embeddings for all patches. The code is published on GitHub.

Why a single descriptor falls short

The standard image retrieval pipeline built on a vision transformer is simple: run the image through the model, take [CLS], and compare by cosine similarity. That is enough for coarse semantic search ("find beaches"). Fine-grained tasks, such as closely related animal species, equipment models, or similar products, are harder. The differences sit in small details, while a global vector has to pack the entire content of the frame into itself at once. The authors state directly that this compression hurts retrieval quality.

Other approaches tackle the problem with external detectors, bounding boxes, or saliency modules. That means extra components, extra sources of error, and additional annotation. This work takes a different route.

How the method works

The method builds on DINOv2-reg, a variant of DINOv2 with register tokens. The authors observe that during training these tokens start encoding objects and their parts on their own. Each of the five semantic tokens ([CLS] and four registers) acts as a "cue." For each cue, the method selects a "buddy," the patch token that best matches the spatial region the cue describes. An N×N window of patches around the buddy is then aggregated into a local ROI token.

One important detail: the regions of interest are found automatically, purely by matching semantic tokens to their spatial representations. There are no external boxes and no saliency-extraction modules.

Then the ColBERT idea comes in. The query and the document are each represented as sets of vectors, and similarity is computed through token-by-token alignment (late interaction). This preserves fine-grained matching, but the index does not balloon to hundreds of patch embeddings per image.

• Backbone: DINOv2-reg, five semantic tokens ([CLS] + 4 register).
• Each "cue" gets its own ROI token, built from an N×N region of patches.
• No detectors, bounding boxes, or saliency modules are needed.
• Retrieval: multi-vector with token-by-token alignment, as in ColBERT.
• The code is open; the repository is listed in the abstract.

What the experiments showed

The authors draw three conclusions. First, register tokens do store useful fine details and complement [CLS]. Second, automatically pooled ROI tokens further improve discrimination between similar classes. Third, multi-vector retrieval with a small number of tokens outperforms the single-vector DINOv2-reg baseline and remains "tractable" for large-scale search.

One limitation up front: the abstract gives no concrete metrics, with no Recall@K or mAP gains and no benchmark names. The size of the effect can only be judged from the full text and the paper's tables. The claim of "tractable" scale also needs to be verified on your own data volumes.

Where it can be applied

The method is of practical interest to teams building visual search over catalogs: e-commerce, museum collections, biological and medical archives. In those settings, small differences separate the right result from a nearly right one. A plus of the approach is that it builds on top of an existing DINOv2-reg: retraining the backbone from scratch is not necessarily required, and it is enough to add a token-extraction step.

The cost is known in advance: instead of one vector per image, you will have to store several. If an ROI token is built for each of the five cues, that comes to about ten vectors (an estimate based on the method description; the exact number depends on the configuration). The index will grow, and the vector database must support multi-vector queries with MaxSim-style scoring. Before adopting it, it makes sense to run the repository on your own dataset and weigh the accuracy gain against the increase in storage cost and latency.