SGLang-TorchTPU nearly doubles Muse Spark Open’s token generation speed

A coding agent has read the repository, run a tool and paused before its next response. It is easy to blame the model’s size for that pause. But the server may be spending time fetching weights, communicating between accelerators and reprocessing familiar context. For a product team, this is a concrete problem: the user waits while infrastructure they are paying for handles preparatory work.

The SGLang-TorchTPU team described running Muse Spark Open, an 816-billion-parameter model, on a single host with four TPU v7 chips. The main single-request result: time per output token fell from 3.78 to 1.95 ms. The measurements and implementation details were published on the PyTorch blog.

Accelerators need data delivered on time

Imagine a kitchen where the cook stops before every task to wait for the next basket of ingredients. Buying a more powerful stove will not eliminate the wait. Another approach is to arrange deliveries so that the next basket arrives while the current dish is cooking.

The team applied a similar idea here: it fused the execution of 62 layers into one program and overlapped data transfers with computation. This makes it possible to prepare in advance what the next stage of work will need.

Putting this approach into practice starts with profiling. Suppose you run an internal assistant for a small team. Usually, several people interact with it at once, and each wants a smoothly streamed response. In this setting, it is worth examining how long each generation step takes and how much of that time is spent waiting for data.

A service with a large user base faces a different challenge: handling the queue without a sharp increase in latency. One impressive speed figure therefore does not tell you how many users the system can support. That requires a series of measurements at different loads under otherwise identical conditions.

An agent’s pause is a reason to retain context

The second result concerns memory. An additional cache in host RAM increased throughput on agent workloads by up to 77%.

Consider an assistant looking for a bug in a project. It reads files, runs tests, receives the output and returns to reasoning. Much of the conversation remains unchanged between these actions. If the server has retained the precomputed context data, the next turn can begin with less repeated work.

For a product team, this raises a useful question: what happens to session state while the user or tool is silent? A pause does not, by itself, mean the conversation is over. This is especially true when an agent is waiting for a build, an external search or the result of a long-running command.

A dedicated test scenario is a useful way to check this. Give the agent a large document, exchange several short turns with pauses and compare how long it takes for the response to start. Then repeat the experiment with memory full. This will make it clearer how much the service’s behavior depends on context retention and what happens when older sessions are evicted.

What to apply to your own service

A nearly twofold speedup does not automatically translate into budget savings. The fused program’s advantage shrinks as the number of concurrent requests grows. The authors also still plan to upstream some of the work into the project’s main branch.

Developers should start with three measurements: time to first token, the intervals between subsequent tokens and the number of completed requests at a given latency. Product managers should add a realistic user journey with tools and pauses. Founders should calculate the cost of a successfully completed task at an acceptable response time.

There is also a separate scenario for testing the queue: one user submits a huge document while another asks a short question. Check how long the second user waits. This experiment quickly reveals whether resource allocation delivers the promised quality of service.

The publication’s value lies in its set of testable engineering techniques. Before changing hardware, it is useful to establish exactly where the server sits idle, what work it repeats and which requests it lets occupy the queue for too long.