Glance: a single 9B model picks its own RAG passages and beats a reranker

A typical RAG pipeline keeps two models in memory. First, a reranker sorts the passages returned by search; then a generator reads the best ones and writes the answer. Researchers from the airun-ai project propose Glance: the generator scores the passages itself, so no separate reranker is needed. They added a LoRA adapter (a small set of fine-tuned weights) and a 256-dimensional "pointer head" to a frozen 9-billion-parameter model. Each additional "skill" add-on of this kind takes about 0.1 GiB. These are early results: the team tested two retrieval datasets, one question-answering dataset and one model family.

One pass instead of twenty

A classic cross-encoder (for example, bge-reranker-v2-m3) encodes each "query–passage" pair separately. With 20 candidates, that means 20 passes. Glance works differently.

• The model reads all candidates once and stores an internal cache (prefill).
• Short probes of about 30 tokens each branch off this cache: "Does passage N contain information that answers the query?"
• The probes cannot see each other, but each one sees all the passages and can compare them.
• The pointer head converts the model's state at the probe into a relevance score. No text is generated in the process.

There is also a Glance-Activated variant. In it, the adapter is switched on only for the probe tokens, the passages are encoded with the base weights, and the answer continues from the original cache. In the main mode, the selected passages (about 1.5 thousand tokens) are re-read with the adapter switched off.

The base model, Xiaomi's MiMo-V2.6-Distill-Qwen-9B, is a hybrid: Gated DeltaNet linear-attention layers alternate with standard attention layers in a 3:1 ratio. The linear layers keep a single sequential state, so caches of individual passages cannot be stitched together in arbitrary order. The authors chose to copy the branches from the end of the shared context, and this method is exact.

What the measurements show

The evaluation used nDCG@10: the higher the relevant documents rank in the top ten, the closer the value is to 1. The rerankers sorted the top 20 from BM25 (classic text search).

• Without training on retrieval data, the adapter scored 0.715 on SciFact (scientific claims), versus 0.729 for twenty separate calls of the same model. It was also 3.6 times faster. The cross-encoder scores 0.721 on SciFact.
• On NFCorpus (medical texts), the same mode scored 0.341 versus 0.326 for the cross-encoder. The authors suggest that comparing passages against each other helps.
• After 39 minutes of training on MS MARCO (4000 queries, one B200 GPU, 21 GiB peak), the results rose to 0.741 on SciFact and 0.345 on NFCorpus. The lead over the cross-encoder was 0.020 and 0.019. The authors did not test significance on these datasets and call any difference below 0.02 indicative only.

On HotpotQA (questions that require combining facts from two passages), the team took 300 questions with 10 passages each. The cross-encoder found the relevant passages more often: top-2 recall was 0.863 versus 0.762. But the answers built on Glance's selection turned out better: F1 0.718 versus 0.683 for the untrained adapter, and this gap is not statistically significant. The abstract claims a gain of +0.045 F1 for the fine-tuned version. If the adapter stays switched on during answering as well, F1 rises to 0.757, which is 0.032 below the ceiling with the two gold passages. The answers get shorter (2.5 words versus 3.1), which fits the HotpotQA format better.

Where the method hits its ceiling

The main advantage shows up on long candidate lists. The fine-tuned adapter keeps improving up to 200 candidates (43.4 thousand tokens): nDCG@10 is 0.717 versus 0.659 for the cross-encoder on the same 100 SciFact queries. At 100 candidates, the gap is 0.060 and latency is 1.05 s per query. The untrained adapter degrades at larger sizes, and at 65 thousand tokens its quality falls to 0.505.

The costs are as follows:

• Memory grows faster than context length, from 15 to 56 GiB, because each probe copies the attention cache.
• 200 candidates do not fit into a single call, so they are processed in batches, which takes 15.8 s per query.
• Combining several queries in one call does not speed things up: throughput drops from 2.87 to 1.57 queries per second.
• The cross-encoder remains much faster: 31–60 ms per query versus roughly 410–420 ms for the trained Glance on a B200.

The authors propose a shared block cache without copies, which would make memory linear and allow contexts on the order of 128 thousand tokens.

Who this is for

Glance's main gain is in memory. If a server already hosts a large model for answering, you can skip the second reranker and switch adapters on demand. The outputs match what each model produces when loaded separately. For systems where latency in the tens of milliseconds matters more, a dedicated cross-encoder remains the better choice. If you need to rank hundreds of candidates or pick the context the generator actually uses, the approach is worth trying. The adapters and code are on Hugging Face in the airun-ai/GlanceRAG-MiMo-9B-Reranker repository.