Latency-Aware LLM Query Routing: Up to 40% Better Accuracy-Cost Utility at the Same Latency

In July 2026, researchers published a paper on arXiv describing a latency-aware LLM query router that jointly optimizes latency, accuracy and cost. It delivers up to 40% higher accuracy-cost utility at the same latencies as standard load balancing.

What's wrong with current routers

Modern LLM query routers balance only response quality and monetary cost. They do not account for generation latency on a specific model instance. In practice, latency is handled by separate load-balancing policies such as round-robin or join-the-shortest-queue, and those policies ignore both model accuracy and inference cost. As a result, routing and load distribution operate blind to each other.

• The goal is to jointly optimize three parameters: latency, accuracy and cost
• Baseline load-balancing policies: round-robin, join-the-shortest-queue
• Gain in accuracy-cost utility: up to 40%
• Latencies stay at the level of standard approaches
• Source: arXiv preprint (cs.AI), July 2026

How request latency is estimated

The authors built a lightweight estimator that simulates autoregressive batch token processing in a serving framework and predicts TTFT (time-to-first-token) for each request. The difficulty is that latency depends on more than prompt length. It is also affected by the current prefill and decode load on the model instance, as well as the scheduling and batching policy of the framework itself. The estimator accounts for all of these factors and stays cheap enough to run in real time during routing.

Why it matters

The latency-aware router builds latency estimation directly into the routing decision and distributes requests across instances so that latency, accuracy and cost stay under control at the same time. According to the authors, this joint optimization raises accuracy-cost utility by up to 40% without making latency worse than conventional load balancing.

"Joint optimization yields up to a 40% gain in accuracy-cost utility

while maintaining the same latencies as standard load-balancing approaches," the paper's abstract on arXiv states.

For services that serve responses from dozens of models under load, this is a way to squeeze more quality out of every dollar spent without making users wait longer.

What this means

LLM query routing stops being a choice between "cheaper or more accurate" and gains a third axis: speed. For infrastructure teams, this offers a lever: the same fleet of models delivers more value with no increase in latency and no overspending on inference.

Frequently asked questions

What is TTFT (time-to-first-token)?

TTFT is the time until the first token of a response, a key latency metric for LLM services. The estimator in the paper predicts TTFT for each request by simulating batch token processing in a serving framework.

How much more effective is latency-aware routing than conventional

load balancing?

According to the authors, joint optimization of latency, accuracy and cost raises accuracy-cost utility by up to 40% at the same latencies as round-robin or join-the-shortest-queue.