NVIDIA Nemotron-Labs-3-Puzzle-75B: 37% Smaller Model Doubles Server Throughput

On July 9, 2026, NVIDIA released Nemotron-Labs-3-Puzzle-75B, a compressed version of the Nemotron-3-Super language model. NVIDIA's engineers used a technique called Puzzle, which alternates hardware-aware compression with knowledge-recovery phases. As a result, the model shrank by 37%, from 120.7B to 75.3B parameters, while server-class throughput rose 2.03x. On a single node with eight B200 accelerators, the model serves 100 tokens per second to each user. On a single H100, throughput at the maximum context of 1M tokens jumped from 1 to 8 concurrent requests.

How the Puzzle method works

Nemotron-Labs-3-Puzzle-75B was built with the new Puzzle method, which tackles a classic problem: a compressed model loses capabilities. NVIDIA's engineers found a fix: alternate two steps.

In the structural compression step, the algorithm removes or prunes neurons, guided by energy consumption and bandwidth on specific hardware. This is not simple weight pruning but an analysis of the hardware profile. A short knowledge-distillation phase follows, in which the original Nemotron-3-Super "teaches" the compressed version and restores the lost knowledge.

The process repeats iteratively: compression → distillation → compression → distillation. The result is a model that keeps its ability to answer hard questions, write code and analyze context, but needs 37% less memory.

• Original model: 120.7B parameters, of which 12.8B are active at once
• Compressed model: 75.3B parameters, of which 9.3B are active
• Architecture type: hybrid Mixture of Experts (MoE)
• Method: iterative hardware-aware compression and knowledge distillation
• Memory savings: roughly 37% reduction

Performance on different accelerators

The main advantage of Nemotron-Labs-3-Puzzle-75B is throughput, not the speed of an individual request. That matters for cloud APIs and enterprise services that serve thousands of users at once.

On the newest hardware, a node with eight NVIDIA B200 accelerators, the Puzzle model delivers 2.03x higher throughput than Nemotron-3-Super. Each user is guaranteed a processing speed of at least 100 tokens per second.

On the more affordable H100, the gain is even more impressive. With a 1M-token context, concurrency rose from 1 to 8 simultaneous requests. In other words, a single H100 now serves 8 times more users at the same latency.

Why this matters for the industry

The results of Nemotron-Labs-3-Puzzle-75B suggest the era of giant, unoptimized models is coming to an end. Companies value not model size but the ratio of quality to performance on specific hardware. NVIDIA has shown that even models with tens of billions of parameters lend themselves to rational compression.

The Puzzle method will likely become a standard for preparing LLMs for production deployment. Startups and mid-sized companies will get a tool to take powerful models and optimize them for their own data centers, whether cloud or edge devices.

What this means

NVIDIA is demonstrating that the future belongs to optimized models rather than giant ones. A 37% reduction in size alongside a 200% speedup is not a trade-off but clear progress. Companies deploying LLMs will get a way to save on compute without losing functionality.