Cohere: Reranking 100 Candidates Instead of 20 Raises the Load from 4,400 to 22,000 Tokens

Cohere has published a practical breakdown of how to choose between shared, pay-as-you-go inference and dedicated inference with reserved capacity for its Embed and Rerank models. The author, solutions architect Payal Singh, starts from a simple premise: comparing price lists is pointless until you understand how your application uses the model. A single embeddings call on 100 product descriptions of ~200 tokens each consumes about 20 thousand tokens. A single shopper's search query is a few words long. For billing, both count as one "request"; for hardware, they are completely different workloads.

One Model, Two Profiles

Cohere uses an online store as its example. First the catalog has to be indexed: each description passes through the model, and the resulting vector is stored. The first pass is a big job, followed by a stream of small batches as the product range changes. The application sends products in batches of about 100 descriptions per call, and no user is waiting on the result. What matters is finishing the job, not finishing it fast, so teams optimize for throughput and cost per unit of work.

A search query like "waterproof trail running shoes" is the opposite case. It is short, and a person is watching a loading indicator. Responsiveness matters here, and it comes at the cost of worse price-performance. In effect, one model is asked to do two different jobs. Infrastructure tuned for large batches will not necessarily deliver the response time you need across thousands of small, latency-sensitive requests.

Rerank: Retrieval Depth Works as a Multiplier

A reranker has its own geometry. Vector search returns, say, 50 candidates, and the model scores each query–document pair independently. The query is processed anew for every candidate, so the work grows in direct proportion to their number.

A typical call, by Cohere's estimate, is a query of about 20 tokens plus 50 products of ~200 tokens each. That comes to roughly 11 thousand tokens per search. If you increase retrieval depth, the picture changes without a single new user:

• 20 candidates: about 4,400 tokens per search;
• 50 candidates: about 11,000 tokens;
• 100 candidates: about 22,000 tokens.

The number of searches and the user experience stay the same, yet the workload needs five times as much compute. That is why Cohere suggests treating retrieval depth as an infrastructure decision, not just a quality-tuning setting.

Takeaways from the Key Points

The article's key points add up to the following logic:

• request shape matters more than request volume: a few large Embed calls can consume more compute than thousands of short search queries, so RPM as the sole metric for capacity planning is unreliable;
• dedicated inference is usually more cost-effective under steady, predictable load, while consumption-based pricing suits variable or spiky traffic;
• the break-even point depends on request size, token volume, and the number of rerank candidates, not just on infrastructure price;
• peak throughput is not the same as usable production capacity: you have to meet latency requirements while keeping costs reasonable.

How to Approach the Choice in Practice

The sequence follows Cohere's own: request profile, then traffic pattern, then utilization, and only then the deployment choice. Start by measuring real request sizes: how many documents per indexing batch, how many tokens on average, how many candidates go to the reranker. Then split the workload into background (indexing, batch recomputation) and interactive (search, where the user is waiting). The former can often stay in shared mode if it is irregular. The latter, if the traffic is steady, is a candidate for dedicated capacity.

Separately, recalculate cost when you change reranking depth: going from 20 to 100 candidates can shift the break-even point. And test capacity against your target latency, not against the datasheet maximum throughput. In the available excerpt, this part of the material cuts off at the section on traffic patterns, so Cohere does not give specific thresholds for switching between modes here.