Two engineers integrated SGLang with IBM Spyre in a week and boosted throughput 3.6-fold

Integrating a new accelerator with a language model server can easily become an engineering project in its own right. Memory management, attention computation, and request handling all need to work together, and the result must actually be faster than the original solution. A team of two engineers reported integrating SGLang with IBM Spyre in under a week using AI agents. The plugin came to roughly 1500 lines, with throughput 3.6 times that of the reference PyTorch inference implementation using torch-spyre. The authors report these results on the PyTorch blog.

The practical lesson lies in how the work was organized. The engineers already had hardware support and an example of another integration. They gave the agents existing code and specific interfaces to connect it to. For teams moving inference to different hardware, this offers a useful example of how to define a bounded task and build on existing work.

How the integration was assembled

Spyre is IBM's accelerator for enterprise inference. The torch-spyre library registers it as a PyTorch device and connects it to torch.compile. This makes some hardware functionality available through standard PyTorch mechanisms.

The second foundation was the existing Spyre integration with vLLM. It supplied attention implementations, tensor shape handling, input bucketing by size, and an approach to allocating memory for the KV cache. This cache stores intermediate data so the model does not have to recompute it when generating each subsequent token.

SGLang provided the integration points: device registration, attention implementation, KV cache management, and compute operation selection. AI agents adapted the code to these interfaces. The request scheduler and main serving loop were existing SGLang components.

Four small patches to SGLang itself were also needed. Two made existing execution paths accessible without CUDA dependencies; the other two registered the Spyre device and an attention implementation that does not require Triton. According to the authors, some of these changes will also benefit other accelerators.

Why a server needs to remember the start of a request

Imagine an enterprise assistant: every request begins with the same long instruction, followed by an employee's question. If the server can reuse computations for that shared beginning, it does not have to process that portion again for every request.

In SGLang, RadixCache handles prefix sharing. In this integration, it worked without Spyre-specific changes: the scheduler decides what to reuse before handing work to the accelerator. This illustrates how a server feature can carry over to new hardware along with the shared serving logic.

The code savings also had a concrete explanation. The existing vLLM integration for Spyre contained around 3500 lines. The authors attribute roughly two-thirds of the difference between it and the new plugin to reuse of SGLang's base attention class. Comparing plugin sizes alone therefore says little about AI productivity: the libraries' architecture accounted for a substantial part of the result.

What to check before putting it into production

The 3.6-fold throughput increase refers to the comparison with the reference PyTorch implementation. In a separate comparison with vLLM on Spyre, SGLang delivered comparable results on the measured workloads for Granite 3.3 8B and Gemma 4 26B A4B. There is not yet a basis for extending these findings to all models and request patterns.

The initial integration also relies on the existing compiler, runtime, and hardware operations. The authors describe support for dynamic shapes and Triton as next steps. Running the example requires installing and configuring the Spyre environment, torch-spyre, the SGLang plugin, and compatibility patches in advance.

For developers, the useful takeaway is how to frame an agent's task: provide a working implementation of the operations, specify the target interface, and validate the result on real hardware. Product managers should separately budget time for load testing and maintenance. A week to an initial integration is an impressive result, but readiness to serve a specific product still needs to be confirmed through your own measurements.