Cohere Keeps 98.4% of Search Quality With Fast Queries

RAG systems have a familiar engineering problem: fast vector search is good at gathering candidates, but it regularly pushes to the top fragments that resemble the question in wording rather than in meaning. If these are passed straight to a language model, it receives excess context, spends more tokens and is more likely to produce a confident but wrong answer.

Cohere has released Embed 5 and Rerank 5 on OpenRouter. The practical significance of the release is not that two more names have appeared in the model catalog. Now a full search pipeline can be built through a single API: create a vector index, find candidates, reorder them more precisely, and only then call the LLM. Separate accounts and SDKs for Cohere are not needed.

How the work is divided

Indexing happens once for the entire document corpus. The text is split into fragments, after which Embed 5 Pro turns each fragment into a vector. Documents are marked with the search_document mode, and the vectors are normalized so that similarity can be computed as a simple dot product.

When a user submits a query, Embed 5 Fast works in search_query mode. Pro and Fast share a common vector space, so the index does not have to be built a second time. According to Cohere, queries run through Fast against an index created by Pro retained 98.4% of the quality of a fully Pro configuration across 40 datasets. If Fast is used for both indexing and search, the figure falls to 96.6%.

Imagine a documentation base with instructions for creating, revoking and replacing API keys. A user asks how to change a key without stopping the service. Vector search finds the right instruction, but pages about creating keys, webhook secrets and log rotation may turn up alongside it. They are semantically close, although they do not answer the question.

Why a second stage is needed

Rerank 5 receives the question and roughly 10–20 retrieved fragments, reads each pair in full and returns a more accurate ordering. In Cohere's example, the correct instruction received a score of 0.808, while two close but useless results failed to pass the 0.3 threshold. Only a cleaned selection of three to five fragments is sent to the language model.

This split makes sense: first-pass retrieval is responsible for recall, and the reranker for precision. Running a heavy comparison across the whole index is too expensive, but rereading a short list of candidates is entirely feasible. If nothing is left after filtering, the LLM call can be skipped altogether and the system can honestly report that the documents contain no answer.

Cohere separately states that Rerank 5 scores are calibrated across queries and subject areas. This makes it possible to use a single relevance threshold. But 0.3 should not be treated as a universal setting: the threshold from the demonstration must be verified on your own data.

What to consider before deployment

The demonstration uses NumPy as storage, so it suits a prototype rather than a production workload. A working system will need pgvector, Qdrant, Pinecone, Weaviate or another vector database, index version control, and observability for latency, cost and result quality.

Fragment size also affects the outcome. Fragments that are too short lose context, while ones that are too long raise the cost of reranking. The 1,500 characters given in the example are only a starting point. Rerank 5 supports up to 32 thousand tokens for a query-and-fragment pair, but a large permitted context does not automatically make large fragments useful.

The practical next step is to take several dozen real user questions, manually mark the correct documents, and compare the top-five hit rate before and after reranking. At the same time, it is worth measuring the added latency and cost. If quality rises noticeably, the scheme of Pro for the index, Fast for queries and Rerank for final selection offers a clear way to speed up RAG without rebuilding the system from scratch.