Blink Removes the CPU From LLM Serving and Cuts Time to First Token by Up to 8.47×
While a model generates a response, the GPU often sits idle, waiting for the CPU. The CPU accepts requests, assembles batches, and decides what to compute next. Blink moves that work to a SmartNIC and the GPU itself, taking the CPU out of the serving loop. In the best case, time to first token at P99 (the latency that 99% of requests stay under) fell by 8.47×, and energy per token dropped by 48.6%. If your server slows down under CPU load, check whether the GPU is idling.
AI-processed from X: @aiwithmayank; edited by Hamidun News
Imagine sending a query to a chatbot and staring at a blank screen, even though the server has an expensive GPU that is fully ready to compute. It is not waiting for data or calculations. It is waiting for a command from an ordinary CPU.
The researchers show that in systems such as vLLM, SGLang, and TensorRT-LLM, this CPU-based scheduler often becomes the bottleneck. Their system, Blink, works without it in steady-state operation: in the best case, time to first token (the first fragment of the response) fell by 8.47×.
This refers to P99, the latency within which 99% of requests complete, which is close to the worst case a user sees in practice. In addition, each token consumes 48.6% less energy.
Where the GPU Loses Time
Modern LLM serving engines are built in a similar way. The CPU accepts incoming requests, groups them into batches (sets of requests processed together), plans what to compute next, and manages the memory for the KV cache. This is the memory where the model stores the part of the conversation it has already processed, so it does not have to recompute it.
A clinic makes this easy to picture. The CPU acts like a receptionist: it registers patients, assigns them to rooms, and tells the doctor who is next. The doctor, meaning the GPU, may be free and ready, but without the receptionist's instructions will not start seeing patients. While the processor is busy with something else, the expensive hardware sits idle.
On a real server, the CPU always has other work: logging, monitoring, the network stack, neighboring services. When the researchers deliberately loaded the host with heavy competing tasks, conventional systems slowed by up to two orders of magnitude, roughly a hundredfold in the worst case.
How Blink Works
The idea is to remove the middleman from steady-state serving. The design has three parts:
- A SmartNIC, meaning a network card with its own processor, receives each request and, using RDMA (direct memory access without CPU involvement), places it straight into GPU memory;
- A persistent kernel running inside the GPU assembles batches and schedules work on its own;
- The same kernel manages the KV cache, and the server's CPU takes no part in the serving loop.
In clinic terms, the doctor watches the registration queue and decides for themselves whom to call next. No one phones them over the intercom.
What the Tests Showed
The authors compared Blink with vLLM, SGLang, and TensorRT-LLM. All the figures below are stated as "up to", meaning they describe the best conditions:
- time to first token (P99) is up to 8.47× lower;
- latency for each subsequent token (P99) is up to 3.40× lower;
- throughput during the generation phase is up to 2.1× higher;
- energy consumption per token is 48.6% lower.
Stability deserves separate mention. When the host CPU was overloaded, classic systems degraded sharply. Blink stayed flat, because inference no longer depends on the overloaded host.
Limitations and What to Do in Practice
The result should not be overstated. Blink does not remove the CPU from the server entirely: the processor is still needed for everything else; it simply leaves the critical path while the model is generating. The system still depends on the GPU and the SmartNIC, so this is not another config flag but an architectural choice that requires specialized hardware. This is not necessarily bad news for Nvidia: GPUs are not going anywhere. The numbers are the authors' own account, there are no independent measurements on production workloads in the source, and the "best case" figures should not be treated as typical.
The practical takeaway for anyone building inference infrastructure: before buying more powerful GPUs, check whether they are waiting on the CPU. If tail latency (P99) jumps noticeably as background load on the host grows, the problem is most likely scheduling rather than compute. A simple test is to load the host with unrelated tasks and watch how time to first token and throughput respond. If the picture worsens, the scheduler deserves attention before a new GPU does. It is also worth watching whether similar approaches appear in popular engines: moving scheduling onto the GPU and the network card looks like a logical next step if the load on LLM services keeps growing.
- Nov 21, 2026Anthropic ends first week on the stock market at a $1.6 trillion valuation, but falls 8.5% in a single day
- Nov 21, 2026TensorRT-LLM Beats vLLM on H100: 2780 vs 2400 Tokens/s on Llama 3.3 70B
- Nov 21, 20261C Developers Switch to AI Agents: McKinsey Sees Just a 3% Productivity Gain
- Nov 21, 2026Chinese Tech Giants Double AI Spending to $139B, While Labs Recover Less Than 8%
Key Questions Answered
Where the GPU Loses Time?
Modern LLM serving engines are built in a similar way. The CPU accepts incoming requests, groups them into batches (sets of requests processed together), plans what to compute next, and manages the memory for the KV cache. This is the memory where the model stores the part of the conversation it has already processed, so it does not have to recompute it. A clinic makes this easy to picture. The CPU acts like a receptionist: it registers patients, assigns them to rooms, and tells the doctor who is next.
How Blink Works?
The idea is to remove the middleman from steady-state serving. The design has three parts: A SmartNIC, meaning a network card with its own processor, receives each request and, using RDMA (direct memory access without CPU involvement), places it straight into GPU memory; A persistent kernel running inside the GPU assembles batches and schedules work on its own; The same kernel manages the KV cache, and the server's CPU takes no part in the serving loop.
What the Tests Showed?
The authors compared Blink with vLLM, SGLang, and TensorRT-LLM. All the figures below are stated as "up to", meaning they describe the best conditions: time to first token (P99) is up to 8.47× lower; latency for each subsequent token (P99) is up to 3.40× lower; throughput during the generation phase is up to 2.1× higher; energy consumption per token is 48.6% lower. Stability deserves separate mention. When the host CPU was overloaded, classic systems degraded sharply.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.