Cold-starting an LLM in Kubernetes: 200 of 294 seconds are not spent loading weights

The idea sounds compelling: requests to the model arrive only a few times a day, yet the GPU node has to be paid for 24/7. The logical move is to remove pods between requests and bring them up when load appears. This works for a conventional web service: a delay of a couple of seconds will not bother anyone. LLMs are different.

Why an LLM takes so long to start

After a model pod starts, it must download the image, load the weights into the GPU, compile the graphs, and warm up the engine. In July, the H Company team wrote that a new replica of its largest models could take up to 27 minutes to become ready. The company runs vLLM on Kubernetes GPU nodes and adds replicas as agent-driven load increases. The autoscaler can quickly raise the pod count, but scaling settings cannot accelerate all these preparation stages.

The author of the analysis expected weight loading to consume most of the time. Public benchmarks showed otherwise. In Tensorfuse's test, starting Llama 3.1 8B on an L40S took 294 seconds. Downloading the model and transferring the weights to the GPU took only 94 seconds. The remaining 200 seconds went to compilation, CUDA graph capture, engine initialization, and imports.

What can be cached

An important detail is that a substantial part of this work is repeated every time a pod starts, even though its results can be stored on the node. This is the main opportunity for acceleration. If compilation artifacts are kept on the node and weights are fetched in advance, a new pod does not have to start from scratch.

In the full article, the author promises to cover the practical details:

• where KEDA gets a load signal when there are no pods at all;
• what should be cached in advance;
• which timeouts allow the first request to wait for the model;
• manifests for KEDA HTTP Add-on, a preload DaemonSet, and vLLM configuration.

The load signal deserves special attention. When there are no pods, there are no pod metrics to collect, so a component is needed in front of the service. KEDA HTTP Add-on intercepts the request, starts a pod, and holds the connection until the model is ready. Timeouts are critical here: the client, gateway, and ingress must not terminate the connection before the model has warmed up.

Limitations and conclusion

Even after all optimizations, a cold start will not become a matter of seconds. If you run an interactive chat where users are unwilling to wait a minute, scale-to-zero is a poor choice. For background jobs, internal tools, batch workloads, and agents that can tolerate a delay, the approach can save a meaningful amount of money.

The practical takeaway for platform engineers is to examine where the time really goes before optimizing weight downloads. According to public benchmarks, more than two-thirds of startup time is spent on compilation and initialization, not the network. Caching these stages on the node delivers more value than a fast disk.

Finally, assess the workload profile honestly. If requests are infrequent and predictable, zero replicas can pay off. If users expect an answer immediately, it is better to keep one warm replica and pay for it than to pay for lost customers.