NVIDIA unveils ReplaySSM: a way to speed up decoding in hybrid Mamba and GDN models

NVIDIA has published a Developer Blog post on ReplaySSM, an attempt to rethink how the inference stack stores SSM-layer state when decoding hybrid models. The list of architectures shows how broad the topic is. Nemotron 3, Qwen3.5 through Qwen3.8, Kimi-K3 and GLM-5.3-Flash all follow the same pattern: a few attention layers alternate with state-space model layers. The portion of the article available to us cuts off in the introduction, so there are no concrete speedup figures here, and we will not invent any.

Why hybrids have become the norm

These models use one of three variants as their SSM layer: Mamba-2, Gated DeltaNet (GDN) or Kimi Delta Attention (KDA). The idea behind all three is the same. A classic attention layer stores a KV-cache that grows with every new context token. An SSM layer instead keeps a fixed-size state and updates it at every step. Memory does not balloon on long contexts, and decoding a token costs roughly the same whether the history holds a thousand tokens or hundreds of thousands.

A small number of attention layers remain to preserve accuracy on tasks that require exact lookup in the context. The result is a trade-off: most of the network is cheap to decode, while a smaller part is expensive but irreplaceable.

Where the state problem comes from

A fixed-size state has a downside. A KV-cache can be truncated: if speculative decoding rejects three of five tokens, you simply drop the last three entries. SSM state works differently. It folds the entire history into a single tensor, and once it has been updated, you cannot subtract an individual token from it. Rolling back requires either storing copies of the state for each step or being able to reconstruct it.

This raises questions that inference engineers have to answer for speculative decoding, prefix caching and continuous batching. How many state snapshots should be kept in memory? How often should they be taken? How much traffic between HBM and the compute units does that cost? The name ReplaySSM seems to point to a "replay instead of store" approach: rather than saving intermediate states, recompute them when needed. This is our interpretation of the name, not a confirmed detail from NVIDIA's text.

What the announcement tells us

• Model class: hybrids in which SSM layers alternate with a small number of attention layers.
• Affected families: NVIDIA Nemotron 3, Qwen3.5–Qwen3.8, Kimi-K3, GLM-5.3-Flash.
• SSM layer types: Mamba-2, Gated DeltaNet, Kimi Delta Attention.
• Key property of SSMs: a fixed-size state instead of a growing KV-cache.
• Stated topic: rethinking state storage for faster decoding.
• Speedup metrics, hardware and comparisons with baseline implementations: not specified in the available portion.

What inference-stack engineers should check

The fact that this appeared on NVIDIA's official blog matters: hybrids have gone from exotic to a main target of optimization, and their specific bottlenecks are getting dedicated solutions. If you run Qwen3.5+ or Nemotron 3 in production, it is worth reading the full article and answering a few questions.

First: how does your engine handle state rollback in speculative decoding, and how much memory goes to snapshots? Second: does prefix caching work for your SSM layers, or only for the attention layers? Third: at what batch sizes and context lengths do the authors measure the gain? For hybrids, the result depends heavily on these parameters.

Until the full text with measurements has been reviewed, treat ReplaySSM as a signal of the direction of the work, not as a ready-made number for capacity planning.