The Qwen team released FlashQLA: accelerating linear attention up to 3× on NVIDIA Hopper

The QwenLM team released FlashQLA — an open-source kernel library that accelerates linear attention operations up to three times on NVIDIA Hopper GPU architecture. The library targets two scenarios: large-scale language model pretraining and agent inference on edge devices.

What is FlashQLA

FlashQLA optimizes forward and backward passes for the Gated Delta Network (GDN) architecture in Chunked Prefill mode. GDN is a variant of linear attention: a mechanism with computational complexity O(n) over context length, unlike O(n²) for standard transformers. In practice, this means that GDN-based models can work with very long contexts without explosive growth in memory consumption.

The problem is that theoretical advantages don't convert to real speed without efficient low-level kernels. FlashQLA fills this gap. The name references FlashAttention — a library that made quadratic attention practical for long sequences through tile-based memory optimization. FlashQLA solves an analogous problem for linear architectures: it provides an infrastructure layer without which a theoretically promising approach doesn't yield real numbers.

3× Speedup: How It Works

The performance gain is achieved through deep optimization for NVIDIA Hopper (H100/H200) — GPUs that dominate modern cloud data centers. The Hopper architecture includes specialized units for recurrent and sparse computation logic, which aligns well with GDN requirements.

The library covers several scenarios:

• Large-scale pretraining — accelerated backward pass reduces training time and cost
• Edge inference — efficient execution without powerful cloud GPU, important for on-device deployment
• Chunked Prefill — splitting long input context into blocks reduces peak memory consumption
• Agent inference — multiple model calls in a single stream without accumulating latencies
• Hybrid architectures — compatibility with models that combine linear and standard attention

Before FlashQLA, developers with GDN architectures got weak benchmarks not due to architectural shortcomings, but due to lack of optimized kernels. This created a false impression of linear attention's non-competitiveness.

Why This Matters for Alibaba and Qwen

The Qwen team from Alibaba Cloud is one of the most active players in open-source LLM development. The Qwen model series consistently expands capabilities: long context, multimodality, specialized versions for code and mathematics, tool-calling support.

The release of FlashQLA is an infrastructure bet, not just a research artifact. Alibaba is investing in the idea that linear and hybrid architectures will occupy a significant niche in the next generation of LLMs — especially where long context and resource efficiency matter. The focus specifically on Hopper, not older GPU generations, signals a target on production scenarios, not lab conditions.

What This Means

FlashQLA signals that linear architectures are transitioning from research phase to engineering phase. 3× acceleration on current hardware makes GDN models truly competitive with transformers for long-context and agent inference tasks. For developers working with non-transformer architectures, this is the arrival of proper tooling — not just theoretical promises.