Zero-Host-Copy: How to Run YOLOv8m at 165–215 FPS by Removing the CPU from the Hot Loop

You optimized the model, quantized it to INT8, and inference takes a couple of milliseconds. Yet FPS is still far from what you expected. It is a familiar story: the bottleneck is not the compute unit but everything around it. A write-up on Habr shows how to deal with this in practice.

Where Frames Are Really Lost

On a discrete GPU, every high-resolution frame that travels to system memory and back pays a toll over the PCIe bus. If intermediate results, such as bounding boxes for detection, are also shuttled back and forth, throughput drops noticeably. The faster the model, the more glaring this cost becomes: the model computes in 2 ms, while the copy takes longer than the computation itself.

The Zero-Host-Copy idea is simple: from the moment data is read from disk and decoded until the final encoding and write, it stays in GPU memory. Only the GPU processes it, and GPU → CPU copies and back are reduced to a minimum. A related idea, Zero-Copy, means passing a pointer to a data block between pipeline stages rather than the block itself.

The Silent Killer: cudaMalloc

The second problem is less obvious. Calling cudaMalloc triggers implicit synchronization: the CPU has to wait for the GPU, and the asynchronous pipeline stops being asynchronous. A single such call in the frame-processing loop can cause a slowdown that the profiler shows as mysterious pauses.

The solution is a three-phase lifecycle:

• Initialization. The application pre-allocates memory pools, buffers, and queues with an initial depth.
• Warm-up. The full processing queue is run. If additional buffers are needed, the system allocates them here.
• Active operation. After the first several frames, dynamic allocation is forbidden: no cudaMalloc and no new, only reuse of already allocated blocks.

In essence, this is a familiar technique from game engines and audio processing: do all the heavy work before the start, and in the hot loop only reuse.

The Results

The author implemented a pipeline without DeepStream in pure C++/CUDA. YOLOv8m at 1024x1024 resolution in INT8 delivers roughly 165–215 FPS on an RTX 3060 Ti and about 38 FPS on a Jetson Orin Nano Super. For an embedded platform, this is a very respectable result at this input size.

Limitations and Takeaways

The price of the approach is complexity. A bare-metal pipeline requires manual management of pools, stream synchronization, and buffer lifecycles. A mistake in pool size will lead to a crash or hidden degradation in the hot phase. Maintaining such a system is more expensive than building a solution on DeepStream or an off-the-shelf framework. Moreover, the figures were obtained on a specific model and specific hardware, and the article, judging by its announcement, does not compare against ready-made stacks under the same conditions, so the gain should be verified on your own data.

Practical takeaways for those building video analytics or working with edge devices:

• Before optimizing the model further, measure how much time goes to copies and allocations.
• Find all cudaMalloc, new, and implicit synchronizations in the hot loop and move them into initialization.
• Design the pipeline so that a frame never leaves the GPU: hardware decoding, preprocessing, inference, postprocessing, encoding.
• Build in a warm-up phase and verify that no allocations occur after it.

When the model is already fast, the gains lie in the engineering discipline around it. For real-time video streams, this is often the cheapest way to get a multiple-fold increase in FPS.