Hugging Face speeds up LLM inference by 22% with asynchronous batching

Hugging Face found a simple way to speed up LLM token generation by 22%: it made the CPU and GPU work at the same time instead of waiting on each other. It requires no model retraining and no new algorithms.

What was the problem

In ordinary synchronous batching, the CPU prepares the data, then the GPU computes it, then the CPU waits for the results and prepares the next batch. The GPU ends up idle for about 24% of the time while the CPU handles preprocessing. It is like an assembly line where someone checks the clock at every step. Even when the CPU runs at full load, it is slower than even the simplest GPU computation, so the graphics card just sits and waits for new data.

With synchronous batching, an 8-billion-parameter model at batch size 32 generated 8K tokens in 300 seconds, with GPU activity at only 76%.

How asynchrony solves the problem

The idea is very simple: run the CPU and GPU in parallel using CUDA streams. Three independent streams are used: one for transferring data to the GPU (H2D), one for the computation itself (Compute), and one for transferring results back (D2H). Each stream returns control to the CPU immediately, without blocking it.

CUDA events act as markers between the streams and guarantee the correct order:

• The H2D stream marks when the data has been loaded onto the GPU
• The Compute stream waits for that marker, computes, then sets its own
• The D2H stream waits for the computation to finish and pulls out the results

While the GPU processes batch N, the CPU is already preparing batch N+1 in a different memory buffer. No waiting; both run in parallel.

The main trick: carry-over

The difficulty is that when a request spans several batches, its output tokens from batch N become the input for batch N+1. Hugging Face solved this with placeholders: when preparing batch N+1, it puts zeros in place of the future tokens' inputs, and when batch N finishes, the zeros are replaced with the real values.

"This lets the CPU prepare the next batch without waiting for the results of the current one,"

Hugging Face explains.

Everything relies on CUDA graphs with a shared memory pool, which even saves VRAM.

The numbers speak for themselves

Before optimization: GPU active 76%, generating 8K tokens took 300.6 seconds. After: GPU active 99.4%, the same tokens took 234.5 seconds. The 22% speedup is close to the limit for this architecture (the theoretical maximum was 24%).

What this means

You do not need more expensive GPUs or model retraining to get a substantial speed gain. The code is already in the Transformers library, in the `ContinuousBatchingAsyncIOs` class. For companies that generate a lot of tokens, this could save millions on hardware.