TokenRouter speeds up token-level LLM routing by 2–64x over standard serving

One LLM for every task is a compromise. Calculating harmonics or parsing a specification calls for a model that is confident with formulas and tables, while a general-purpose model is enough for small talk. In production, this has long been handled with routing: at the session or request level, the system picks which model to call. The researchers argue the gains would be larger if you routed individual tokens instead of requests. The catch is that ordinary inference servers are not built for this. The TokenRouter team has proposed a system that fixes that.

Why route tokens

Imagine a small model writing an answer. While everything is trivial, it handles the work on its own and costs next to nothing. At a hard spot it loses confidence, and a large model steps in. This shifts the cost-quality frontier: you pay for the large model only where it is needed.

The paper gives the router R-Stitch as an example. After each decoding step, it computes the entropy of the next-token distribution. If the entropy is above a threshold (τ = 0.03 in the experiments), the token is considered uncertain, it is discarded, and the large model regenerates it. The remaining tokens stay with the small model.

On paper, this looks elegant. In practice, a request keeps jumping between models, and the server starts to stall.

Where ordinary servers break down

The authors name three problems.

• Step desynchronization. The models take different amounts of time per step. In a synchronous batch, everyone waits for the slowest one, and the fast ones sit idle.
• Batch admission delay. A request reaches the target model while it is busy with the previous batch, and has to wait for the next one. Gaps appear in the schedule, and batches break into small ones.
• Implementation complexity. Off-the-shelf engines have no interface for deciding "where next" at every step. You have to rewrite a lot of code and make it work with continuous batching and the prefix cache.

There is another problem: in an ordinary server, a token returned from another model is treated as a new request. This adds overhead for prefix matching and KV-cache updates.

How TokenRouter works

The authors' principle is to program from the request's point of view and execute from the model's point of view. To the client, the system looks like an ordinary server with a single LLM. Inside, each model runs its own sub-server with its own scheduler and a private KV-cache pool. The sub-servers work asynchronously and exchange peer requests.

The developer writes the algorithm through three functions:

• route(result) is called after every step. Based on the logits, hidden states and sampled tokens, it decides whether to stay local or hand the request to another model;
• send(req) builds the message for another model: the request ID, the suffix of tokens it has not seen yet, the status, and algorithm-specific fields;
• receive(peer_req) turns the message into a local request. By default, it simply accepts all the tokens it is given, and it rarely needs to be overridden.

To deal with admission delay, there is a scheduler with deferred batching: it waits until B requests accumulate in the buffer, and only then launches a step. The threshold B is chosen analytically: the authors model routing as a Markov chain that accounts for concurrency, routing probability and each model's step time. A B that is too small fragments batches, while one that is too large keeps requests waiting in the queue.

What the experiments showed

The team tested five algorithms: CITER, R2R, R-Stitch, Co-LLM and ME. Four work with the pair Qwen3-0.6B and Qwen3-32B, and ME works with three models (0.6B, 8B, 32B). The hardware was a server with 8×A100-80G. There were three workloads: short reasoning (AIME2024, output up to 2048 tokens), long reasoning (output up to 8192), and agentic tasks on SWE-Smith trajectories.

The result: throughput is 2.01–64.15 times higher, and end-to-end latency is 2.03–63.64 times lower than with standard serving. When output grew from 2048 to 8192 tokens, TokenRouter did not degrade, while standard serving lost 58.1–85.2% of its throughput.

What to do with this

If you are building a cascade from a small and a large model, the main takeaway is this: the problem is often not the routing algorithm itself but how the engine serves it. A naive implementation can eat almost all of the gain.

There are limitations, too. The results come from the Qwen3 family and from specific algorithms. Answer quality is not the main subject here: the paper is about serving speed, not about which router is best. The threshold B depends on the routing probability and the speed of the models, so you need to recalculate it for your own model pair. For teams running their own inference, this is a reason to look at the open code.