Runpod shows how GuideLLM helps squeeze the most out of vLLM

Runpod has published a guide on its blog about using vLLM with GuideLLM. The idea is simple: before you expose a model to users, run simulated real-world traffic against it and look at three numbers: time to first token, inter-token latency and throughput. These determine how many GPUs you need and how much you will pay per million tokens. The original post is short. It is more an announcement of a practical guide, so it contains no concrete benchmarks on specific cards. But the methodology itself deserves a closer look.

Why "it's fast on my laptop" means nothing

vLLM has long been one of the standard engines for serving open models. It uses continuous batching and PagedAttention to manage the KV-cache, and it works well with a handful of requests. Problems start under load. When dozens of requests with prompts of different lengths arrive at once, the cache fills up, the queue grows, and time to first token can increase several-fold.

A single request won't reveal this. You need a load generator that behaves like real users: requests arrive in waves rather than one at a time, and prompts and responses vary in length. That is what GuideLLM was built for. It is an open-source tool from the vLLM ecosystem. It connects to an OpenAI-compatible endpoint, sends synthetic requests or ones you define, and collects statistics.

What a run looks like

The setup on Runpod is typical. You spin up a pod or a serverless worker with vLLM, get the endpoint address, and launch GuideLLM from that machine or another one. Then you define a load profile: input and output length in tokens, request mode and duration. The usual options are a fixed rate, a Poisson stream, a maximum-throughput mode, and sequential execution, in which the server is never overloaded. A sweep is handy for a first look: it steps through increasing load and shows where the latency curve breaks. Exact flag names change from version to version, so check them against the documentation.

The output is a report with the following metrics:

• TTFT, time to first token: determines how responsive a chat feels;
• ITL, inter-token latency: determines how smooth streaming is;
• end-to-end latency for the whole request, which matters for APIs and agent chains;
• requests per second the server sustains before it degrades;
• tokens per second per card, the key figure for cost calculation;
• the number of concurrent requests at which queues start to grow.

The value lies in comparing these numbers. If TTFT stays within your target at a given load and then climbs, that is your working ceiling on the current configuration.

In practice: from metrics to a GPU bill

The procedure is as follows.

1. Describe your real traffic: average and maximum prompt length, response length and peak load. If you don't have logs yet, use conservative estimates and refine them later.

2. Set target values for TTFT and ITL. They will differ for a chat interface and for batch document processing.

3. Run a sweep on several configurations: different GPUs, different quantization, different maximum context length values and different shares of memory allocated to the KV-cache. 4.

For each configuration, find the maximum load at which the targets are met, and divide the card's hourly price by the resulting number of tokens. That is the real price of a token. 5.

For a serverless deployment, also check cold start and behavior during a sudden queue spike. A static benchmark will not show either.

There is a caveat. Synthetic load remains a model, not reality. If your users send long system prompts with a shared prefix, enabling prefix caching will change the picture, and the test needs to account for it. Results are also tied to the vLLM version, the drivers and the specific card, so it is worth repeating the run after updates.

The main benefit of a guide like this is discipline. Choosing a GPU and settings by feel usually ends in paying for headroom that nobody uses. Half an hour of load testing produces numbers you can put on a finance person's desk.