Qwen Team Releases FlashQLA: Up to 3× Faster Linear Attention on NVIDIA Hopper

The QwenLM team has released FlashQLA, an open-source kernel library that speeds up linear attention operations by up to three times on NVIDIA Hopper GPUs. The library targets two scenarios: large-scale language model pretraining and agentic inference on edge devices.

What FlashQLA is

FlashQLA optimizes the forward and backward passes of the Gated Delta Network (GDN) architecture in Chunked Prefill mode. GDN is a form of linear attention: its computational complexity is O(n) in context length, versus O(n²) for standard transformers. In practice, this means GDN-based models can handle very long contexts without an explosive growth in memory use.

The problem is that these theoretical advantages do not translate into real speed without efficient low-level kernels. FlashQLA fills that gap.

The name nods to FlashAttention, the library that made quadratic attention practical for long sequences through tiled memory optimization. FlashQLA tackles a similar problem for linear architectures: it provides the infrastructure layer without which a theoretically promising approach does not produce real numbers.

Where the 3× speedup comes from

The performance gain comes from deep optimization for NVIDIA Hopper (H100/H200), the GPUs that dominate modern cloud data centers. Hopper includes specialized units for recurrent and sparse computation, which fit GDN's requirements well.

The library covers several scenarios:

• Large-scale pretraining — a faster backward pass cuts training time and cost
• Edge inference — efficient execution without a powerful cloud GPU, which matters for on-device deployment
• Chunked Prefill — splitting a long input context into blocks lowers peak memory use
• Agentic inference — repeated model calls within a single flow without accumulating latency
• Hybrid architectures — compatibility with models that combine linear and standard attention

Before FlashQLA, developers working with GDN architectures saw weak benchmarks not because of flaws in the architecture itself, but because optimized kernels were missing. That created a false impression that linear attention was uncompetitive.

Why Alibaba and Qwen care

The Qwen team at Alibaba Cloud is one of the most active players in open-source LLM development. The Qwen model series has steadily expanded its capabilities: long context, multimodality, specialized versions for code and math, and tool-calling support.

FlashQLA is an infrastructure bet, not just a research artifact. Alibaba is wagering that linear and hybrid architectures will claim a meaningful niche in the next generation of LLMs, especially where long context and resource efficiency matter. Targeting Hopper rather than older GPU generations points to production scenarios, not lab conditions.

What this means

FlashQLA signals that linear architectures are moving from the research phase into the engineering phase. A speedup of up to 3× on current hardware makes GDN models genuinely competitive with transformers on long-context tasks and agentic inference. For developers working with non-transformer architectures, it means proper tooling has arrived, rather than only theoretical promises.