Apple speeds up vector search for RAG by up to 8 times at 99% recall

A user asks how to get a refund, but the support knowledge base returns articles about payments. The language model receives the wrong context, and a polished answer is of little help. Apple researchers propose training separate networks to learn where to look for matches in a particular database. For teams with large archives, this is an opportunity to examine how many resources search itself consumes and whether they can reduce the number of documents scanned.

The network learns where to search

In RAG, documents are first converted into numerical vectors that represent their content. The user's question undergoes similar processing. The system then finds documents whose vectors best match the question vector and passes the retrieved passages to the language model.

Comparing a query against every document is expensive. One option is to divide the database into groups: first select a few promising groups, then search within them. This shortcut comes at a cost: choosing the wrong group can leave the best result outside the search.

The Apple paper describes two ways to train this stage. SupportNet estimates how good a match can be found within each group. KeyNet predicts the vector of the best match and passes it to a conventional search index in place of the original question vector. These models can be used separately.

Imagine an online store's knowledge base. A customer writes: “I canceled my order. When will I get my money back?” The relevant guide might be titled “Refund processing times after cancellation.” In this hypothetical example, KeyNet should steer the search toward that guide's vector. It works with numbers: there is no need to generate the text of a potential answer.

Training examples are prepared in advance. For each question, the best match is calculated through an exhaustive search, and the network is then trained to reproduce that mapping. The costs shift to system preparation; the approach makes sense when the trained network serves many subsequent queries.

What an up to eightfold speedup means

In its experimental breakdown, Apple used around 2.6 million passages from Natural Questions. With the largest KeyNet tested, achieving 99% search recall took 0.23 ms, compared with 1.8 ms for the original query. The measurement includes query transformation and index search. Achieving the same recall required scanning roughly 11 times fewer vectors.

It is easy to misread this result. Recall of 99% means recovering the best match under the chosen vector metric. It does not promise a correct answer to the user in 99% of cases: the original document representations may themselves fail to capture the meaning of the question.

SupportNet produced a different result: at 95% routing recall, latency was 1.74 ms instead of 3.36 ms. These speedups cannot be added together: they are separate measurements of different search variants.

For a product, the absolute savings matter too. A reduction from 1.8 to 0.23 ms saves about one and a half milliseconds at this stage. If generating the answer accounts for most of the wait, users may barely notice the difference. At high query volumes, teams should still assess the impact on compute costs.

How to test the benefits on your own database

Start with the official AMIPS repository. The original post's claim that Apple did not release the code is incorrect: scripts are available for preparing vectors, computing training targets and training models, along with an example that runs on a small FIQA dataset.

A practical experiment should center on your own workload. Take a separate sample of real questions, record current search recall and latency, then compare them with the trained version. Separately calculate preparation time, memory requirements and update costs. Otherwise, the savings on queries can easily disappear into system maintenance.

The method is designed for a fixed database and familiar query types. A relatively stable reference knowledge base is therefore a sensible choice for a pilot. When documents are updated or questions change, you will need to reassess quality and decide whether retraining is necessary. The main criterion for adoption is straightforward: is this database searched often enough to justify the additional training and maintenance?