Runpod shows how to deploy Llama 3.1 8B on vLLM without your own servers

Runpod has published a step-by-step guide to deploying Llama 3.1 8B Instruct on the vLLM engine inside its Serverless platform. The 8-billion-parameter model, with a context window of up to 128 thousand tokens, becomes an API endpoint that wakes up on request and does not require a rented GPU around the clock. This is a how-to guide, not the release of a new product. But the topic is a practical one: this is how small teams now put open models into production without building their own infrastructure.

Llama 3.1 came out in July 2024 in three sizes: 8B, 70B and 405B. The 8-billion-parameter version is the most accessible. In FP16, its weights take up about 16 GB, so it fits on a single card with 24 GB of memory. You still need to budget separate headroom for the KV-cache at long contexts.

How the setup works

vLLM is currently one of the most widely used engines for inference on open models. It is built on PagedAttention: KV-cache memory is allocated in blocks, like pages in an operating system, which reduces fragmentation. Combined with continuous batching, where new requests join a batch that is already running, this raises throughput on a single card. For developers, there is a bonus: the vLLM server is compatible with the OpenAI API format, so client code needs minimal changes.

Runpod Serverless adds a management layer. The model is packaged into a worker container and requests go into a queue. The platform spins workers up and shuts them down on its own, and billing covers their running time to the second. For vLLM, Runpod offers a ready-made template configured through environment variables: the model name, a Hugging Face token and the maximum context length.

Key parameters of the setup:

• Model: Meta Llama 3.1 8B Instruct, about 8 billion parameters.
• Context: up to 128K tokens.
• FP16 weights: on the order of 16 GB.
• Engine: vLLM with PagedAttention and continuous batching.
• Interface: OpenAI-compatible API.
• Platform: Runpod Serverless, with worker autoscaling, including down to zero.

Pitfalls

The biggest one is cold start. If a worker has been shut down, it has to load about 16 GB of weights into GPU memory on the first request, and the user will notice. There are three ways to mitigate the problem: bake the model into the container image, use caching on the platform side, or keep at least one worker active. The last option eats into the savings that are the reason to choose serverless in the first place.

The second issue concerns access. Llama 3.1 is distributed under a Meta license, and the model on Hugging Face is gated: you must accept the terms and pass a token to the worker. The third involves memory. By default, vLLM may reserve space for the entire 128K context, so it is worth limiting max model len to what your application actually needs. That leaves more room for concurrent requests.

The payoff of serverless depends on your load profile. The article gives no specific prices, so you will have to do the math yourself using Runpod's current rates. The rule of thumb is simple: with infrequent, spiky requests, per-second billing beats a dedicated card, while under consistently high load a dedicated server is usually cheaper.

Where to start

A short sequence is enough for a trial run:

1. Accept the Llama 3.1 license on Hugging Face and create an access token. 2. Create a Serverless endpoint from the vLLM template, specifying the model and the token. 3. Choose a GPU with headroom, starting at 24 GB, and limit the maximum context length. 4. Set the minimum and maximum number of workers, as well as the idle time before shutdown. 5. Point your existing OpenAI client at the new endpoint and measure time to first token on both a cold and a warm worker.

If cold start is critical for your product, measure it before migrating, not after. For internal tools and background jobs, it is usually tolerable. For chat interfaces with live users, you will most likely have to keep a warm worker.