Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
InfrastructureInsight · On RadarX: @aiwithmayank→ original

Blink Removes the CPU From LLM Serving and Cuts Time to First Token by Up to 8.47×

While a model generates a response, the GPU often sits idle, waiting for the CPU. The CPU accepts requests, assembles batches, and decides what to compute next. Blink moves that work to a SmartNIC and the GPU itself, taking the CPU out of the serving loop. In the best case, time to first token at P99 (the latency that 99% of requests stay under) fell by 8.47×, and energy per token dropped by 48.6%. If your server slows down under CPU load, check whether the GPU is idling.

AI-processed from X: @aiwithmayank; edited by Hamidun News
Blink Removes the CPU From LLM Serving and Cuts Time to First Token by Up to 8.47×
Source: X: @aiwithmayank. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceX: @aiwithmayank ↗
Editor-in-Chief
Key metric #1Each subsequent token is up to 3.40× faster (P99), and throughput rises up to 2.1×
Key metric #2When the CPU was saturated, conventional engines slowed by up to two orders of magnitude, while Blink held steady
Key metric #3Blink requires a SmartNIC and a GPU; the host CPU stays in the server

Imagine sending a query to a chatbot and staring at a blank screen, even though the server has an expensive GPU that is fully ready to compute. It is not waiting for data or calculations. It is waiting for a command from an ordinary CPU.

The researchers show that in systems such as vLLM, SGLang, and TensorRT-LLM, this CPU-based scheduler often becomes the bottleneck. Their system, Blink, works without it in steady-state operation: in the best case, time to first token (the first fragment of the response) fell by 8.47×.

This refers to P99, the latency within which 99% of requests complete, which is close to the worst case a user sees in practice. In addition, each token consumes 48.6% less energy.

Where the GPU Loses Time

Modern LLM serving engines are built in a similar way. The CPU accepts incoming requests, groups them into batches (sets of requests processed together), plans what to compute next, and manages the memory for the KV cache. This is the memory where the model stores the part of the conversation it has already processed, so it does not have to recompute it.

A clinic makes this easy to picture. The CPU acts like a receptionist: it registers patients, assigns them to rooms, and tells the doctor who is next. The doctor, meaning the GPU, may be free and ready, but without the receptionist's instructions will not start seeing patients. While the processor is busy with something else, the expensive hardware sits idle.

On a real server, the CPU always has other work: logging, monitoring, the network stack, neighboring services. When the researchers deliberately loaded the host with heavy competing tasks, conventional systems slowed by up to two orders of magnitude, roughly a hundredfold in the worst case.

How Blink Works

The idea is to remove the middleman from steady-state serving. The design has three parts:

  • A SmartNIC, meaning a network card with its own processor, receives each request and, using RDMA (direct memory access without CPU involvement), places it straight into GPU memory;
  • A persistent kernel running inside the GPU assembles batches and schedules work on its own;
  • The same kernel manages the KV cache, and the server's CPU takes no part in the serving loop.

In clinic terms, the doctor watches the registration queue and decides for themselves whom to call next. No one phones them over the intercom.

What the Tests Showed

The authors compared Blink with vLLM, SGLang, and TensorRT-LLM. All the figures below are stated as "up to", meaning they describe the best conditions:

  • time to first token (P99) is up to 8.47× lower;
  • latency for each subsequent token (P99) is up to 3.40× lower;
  • throughput during the generation phase is up to 2.1× higher;
  • energy consumption per token is 48.6% lower.

Stability deserves separate mention. When the host CPU was overloaded, classic systems degraded sharply. Blink stayed flat, because inference no longer depends on the overloaded host.

Limitations and What to Do in Practice

The result should not be overstated. Blink does not remove the CPU from the server entirely: the processor is still needed for everything else; it simply leaves the critical path while the model is generating. The system still depends on the GPU and the SmartNIC, so this is not another config flag but an architectural choice that requires specialized hardware. This is not necessarily bad news for Nvidia: GPUs are not going anywhere. The numbers are the authors' own account, there are no independent measurements on production workloads in the source, and the "best case" figures should not be treated as typical.

The practical takeaway for anyone building inference infrastructure: before buying more powerful GPUs, check whether they are waiting on the CPU. If tail latency (P99) jumps noticeably as background load on the host grows, the problem is most likely scheduling rather than compute. A simple test is to load the host with unrelated tasks and watch how time to first token and throughput respond. If the picture worsens, the scheduler deserves attention before a new GPU does. It is also worth watching whether similar approaches appear in popular engines: moving scheduling onto the GPU and the network card looks like a logical next step if the load on LLM services keeps growing.

Story

Key Questions Answered

Where the GPU Loses Time?

Modern LLM serving engines are built in a similar way. The CPU accepts incoming requests, groups them into batches (sets of requests processed together), plans what to compute next, and manages the memory for the KV cache. This is the memory where the model stores the part of the conversation it has already processed, so it does not have to recompute it. A clinic makes this easy to picture. The CPU acts like a receptionist: it registers patients, assigns them to rooms, and tells the doctor who is next.

How Blink Works?

The idea is to remove the middleman from steady-state serving. The design has three parts: A SmartNIC, meaning a network card with its own processor, receives each request and, using RDMA (direct memory access without CPU involvement), places it straight into GPU memory; A persistent kernel running inside the GPU assembles batches and schedules work on its own; The same kernel manages the KV cache, and the server's CPU takes no part in the serving loop.

What the Tests Showed?

The authors compared Blink with vLLM, SGLang, and TensorRT-LLM. All the figures below are stated as "up to", meaning they describe the best conditions: time to first token (P99) is up to 8.47× lower; latency for each subsequent token (P99) is up to 3.40× lower; throughput during the generation phase is up to 2.1× higher; energy consumption per token is 48.6% lower. Stability deserves separate mention. When the host CPU was overloaded, classic systems degraded sharply.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…