NVIDIA releases Nemotron 3 Nano 4B, a compact hybrid model built to run on-device

NVIDIA has opened access to Nemotron 3 Nano 4B, a compact language model with 4 billion parameters designed specifically for edge deployment on Jetson devices, RTX GPUs and DGX Spark. It is NVIDIA's first 4B model built on a hybrid Mamba-Transformer architecture, with a focus on minimal memory use and fast inference.

A new-generation hybrid architecture

Nemotron 3 Nano 4B is built around a 42-layer design: 21 Mamba blocks, 4 Attention blocks and 17 MLP blocks. That mix is unusual for language models of this size, since most competitors rely purely on transformers. Mamba layers process long sequences with linear rather than quadratic memory complexity, which is the main source of the efficiency gains. The Attention blocks are placed strategically to preserve accuracy where global understanding of context matters.

Compared with its parent model, Nemotron Nano 9B v2, the embedding dimension is cut from 4,480 to 3,136, the number of Mamba heads from 128 to 96, and the number of layers from 56 to 42. The result: the lowest VRAM footprint in the 4B class when tested on an RTX 4070, and record-low Time-to-First-Token latency on long input sequences.

Four training stages

Nemotron 3 Nano 4B is not simply a trimmed-down 9B, but a model with its own four-stage training pipeline. The first is compression via Nemotron Elastic: a neural architecture search (NAS) with a trained router decided exactly where to prune the 9B network. The router worked along four axes: Mamba heads, hidden dimension, FFN channels and model depth.

The second is distillation to recover accuracy:

• Short context (8K, 63B tokens): 70% post-training + 30% pretraining data
• Long context (49K, 150B tokens): window extension for complex reasoning tasks

The third is supervised fine-tuning (SFT): two stages covering math, code, science, chat, agentic tasks and safety.

The fourth is three-stage reinforcement learning via NeMo-RL: from single-turn instruction following to multi-turn with JSON/XML outputs, and then to tool calling. The reasoning/non-reasoning data ratio is 50/50, with a progressively tightened KL penalty.

The numbers that matter

On a Jetson Orin Nano with 4-bit GGUF (Q4_K_M), the model generates 18 tokens/sec, twice as fast as Nemotron Nano 9B v2 on the same hardware. FP8 quantization via ModelOpt retains 100% of median accuracy while speeding up inference 1.8× compared with BF16.

"FP8 quantization achieved 100% median accuracy recovery with up to

1.8X latency/throughput improvement over BF16" — from NVIDIA's technical documentation.

On key benchmarks, the model leads its class:

• IFBench and IFEval — instruction following
• Orak — game intelligence: Super Mario, Darkest Dungeon, Stardew Valley
• Tool-use — tool calling and hallucination avoidance
• TTFT — minimal latency on long input sequences

The model comes in three variants: BF16 (full precision), FP8 (optimized for RTX and server GPUs) and GGUF Q4_K_M (for Jetson and Llama.cpp). The vLLM, TRT-LLM and Hugging Face Transformers engines are supported.

What this means

A 4B model with a twofold speed advantage over the 9B on Jetson changes the edge AI equation: robotics, IoT, local agents and game NPCs get an industrial-grade tool without expensive hardware and without sending data to the cloud. The open weights allow fine-tuning the model for a specific domain without licensing restrictions.