How NVIDIA's New Neural Network Saves Memory: Gated DeltaNet-2
NVIDIA has unveiled Gated DeltaNet-2, a new linear attention mechanism for large language models. The key difference: instead of a single scalar gate, the new architecture uses separate channels to control the erasure of old data and the writing of new data. The model was trained on 100B tokens with 1.3B parameters and outperforms competitors in language modeling and long-context tasks.
AI-processed from MarkTechPost; edited by Hamidun News
NVIDIA has built a new neural network, Gated DeltaNet-2, with a linear attention mechanism that saves memory by controlling the erasure of old data and the writing of new data separately, rather than through the single scalar gate used in earlier generations.
What is linear attention in neural networks?
Linear attention is a mechanism that compresses an unbounded KV cache into a fixed-size recurrent state. This makes it more efficient on long texts and significantly reduces memory consumption.
Why do neural networks need memory optimization?
Linear attention mechanisms compress an unbounded KV cache into a fixed-size recurrent state, which lets models handle long texts more efficiently and lowers memory consumption.
How does Gated DeltaNet-2 differ from older neural networks?
The main difference is that it controls the erasure of old data and the writing of new data separately, instead of the single scalar gate used in previous generations.
Key Questions Answered
What is linear attention in neural networks?
Linear attention is a mechanism that compresses an unbounded KV cache into a fixed-size recurrent state. This makes it more efficient on long texts and significantly reduces memory consumption.
Why do neural networks need memory optimization?
Linear attention mechanisms compress an unbounded KV cache into a fixed-size recurrent state, which lets models handle long texts more efficiently and lowers memory consumption.
How does Gated DeltaNet-2 differ from older neural networks?
The main difference is that it controls the erasure of old data and the writing of new data separately, instead of the single scalar gate used in previous generations.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.