Runpod's roundup: Llama-2 with a 32k-token context and SDXL 1.0 now publicly available

Cloud GPU provider Runpod has published the second issue of its weekly RoundUp digest. It leads with two items: Llama-2 with support for contexts of up to 32 thousand tokens, and the public release of SDXL 1.0. Meta's base Llama-2 is built for 4096 tokens, so the window grew eightfold. The third item, about a new Stability AI release, is cut off at the word Stable in the available announcement, so we will not guess about it.

A note on the source. The digest itself is short and consists almost entirely of links to other people's releases. Those releases date from the summer of 2023, while the feed shows a different publication date. The entry was most likely reindexed. For the reader, that means one thing: this is a roundup of things that have already shipped, not fresh news. Still, both items show clearly where the real bottlenecks are for GPU owners.

Llama-2 at 32k: where the long window comes from

Meta released Llama-2 in three sizes: 7B, 13B and 70B parameters. All three had a 4096-token context. The 32k versions came later and were built by third-party teams, including Together AI with its LLaMA-2-7B-32K model. The usual technique is position interpolation: the positional embeddings are stretched to cover a longer sequence, and the model is then fine-tuned on long texts. The architecture stays the same. Only the way the model distinguishes distances between tokens changes.

The practical benefit is clear. 32 thousand tokens hold roughly 80–100 pages of text, which means a long contract, a large code module or the transcript of a multi-hour meeting in one piece. There is no need to split it into chunks or to run a separate vector database search.

You pay for this in memory. Take Llama-2-7B in fp16: 32 layers, hidden dimension 4096. The KV-cache for one token takes 2 × 32 × 4096 × 2 bytes, or 512 KB. At 32 thousand tokens that comes to about 16 GB, and that is for a single request. The model weights themselves take about 14 GB. In total, one long session does not fit in 24 GB of VRAM, so you need a 40 or 80 GB card, or quantization.

SDXL 1.0: what users got

Stability AI's SDXL 1.0 became publicly available in late July 2023. The model works as a pair of components: the base network generates the image, and a refiner polishes the details. The base model has about 3.5 billion parameters, and the whole ensemble about 6.6 billion. That is noticeably more than Stable Diffusion 1.5, and the native resolution rose to 1024×1024. The weights were released under the CreativeML Open RAIL++-M license, so you can run and fine-tune them yourself.

For providers like Runpod, this translates directly into rental demand. People who have an 8 GB card at home move SDXL to the cloud and pay by the hour.

Key numbers

• Llama-2 context: 4096 → 32,000 tokens, roughly an 8x increase.
• Llama-2 sizes: 7B, 13B, 70B parameters.
• Llama-2-7B KV-cache in fp16: about 512 KB per token, about 16 GB at 32k tokens.
• Llama-2-7B weights in fp16: about 14 GB.
• SDXL 1.0: about 3.5 billion parameters in the base model, about 6.6 billion with the refiner.
• SDXL native resolution: 1024×1024.

What to check in practice

If you plan to run long context on rented GPUs, calculate memory in advance. For a 7B model with a 32k window, budget at least 40 GB per parallel session. For several simultaneous requests, you need an 80 GB card or an inference engine with paged KV-cache management, such as vLLM. Quantizing the weights to 8 or 4 bits frees up memory, but the cache has to be compressed separately.

SDXL is simpler. A 16–24 GB card is enough for 1024×1024 generation, and you can enable the refiner only where details matter. Before renting hardware for a long window, check whether plain RAG at 4k already solves your task. It is cheaper, and in many scenarios its quality is no worse.