Роутинг LLM-запросов с учётом задержки: +40% к accuracy-cost при тех же latency

Researchers published a paper on arXiv in July 2026 about a latency-aware LLM request router that jointly optimizes latency, accuracy, and cost, delivering up to a 40% gain in accuracy-cost utility at the same latency as standard load balancing.

What's wrong with current routers

Modern LLM request routers balance only response quality and monetary cost, but don't account for generation latency on a specific model instance. In practice, latency is managed by separate load-balancing policies — round-robin or join-the-shortest-queue — and these ignore both model accuracy and inference cost. As a result, routing and load distribution operate blind to each other.

• The goal is to jointly optimize three parameters: latency, accuracy, and cost
• Baseline load-balancing policies: round-robin, join-the-shortest-queue
• Accuracy-cost utility gain: up to 40%
• Latency stays at the level of standard approaches
• Source: an arXiv preprint (cs.AI section), July 2026

How request latency is estimated

The authors built a lightweight estimator that simulates autoregressive batch token processing in a serving framework and predicts TTFT (time-to-first-token) for each request. The challenge is that latency doesn't depend only on prompt length: it's also affected by the current prefill and decode load on the model instance, as well as the framework's own scheduling and batching policy. The estimator accounts for all these factors while remaining cheap enough to run in real time during routing.

Why this matters

A latency-aware router embeds the latency estimate directly into the routing decision and distributes requests across instances so as to simultaneously keep latency, accuracy, and cost under control. According to the authors, this joint optimization improves accuracy-cost utility by up to 40% without degrading latency compared to ordinary load balancing.

"Joint optimization delivers up to a 40% gain in accuracy-cost utility

while maintaining the same latency as standard load-balancing approaches," the paper's abstract on arXiv states.

For services that serve responses from dozens of models under load, this is a way to squeeze more quality out of every dollar spent, without making the user wait longer.

What this means

LLM request routing is no longer a choice between "cheaper or more accurate" — it adds a third axis: speed. This gives infrastructure teams a lever: the same fleet of models delivers more value without increased latency and without overpaying for inference.

FAQ

What is TTFT (time-to-first-token)?

TTFT is the time until the first token of a response, a key latency metric in LLM services. The estimator from the paper predicts TTFT for each request by simulating batch token processing in the serving framework.

How much more effective is latency-aware routing than ordinary load balancing?

According to the authors, jointly optimizing latency, accuracy, and cost improves accuracy-cost utility by up to 40% at the same latency as round-robin or join-the-shortest-queue.