Developer Fits DeepSeek V3 into 190 Lines of JAX and Trains It on a MacBook Air in 2.6 Hours
The DeepSeek V3 technical report is hard to read, and the model's real code runs to thousands of lines. A developer squeezed the architecture into 190 lines of JAX, Google's library for neural networks, and trained a mini version with 36.2 million parameters on a MacBook Air M5 in 2.6 hours. It is a teaching setup: the model read 207 detective novels and writes smoothly, but meaninglessly. To understand how DeepSeek works, reading this code is the simpler route.
AI-processed from Hacker News AI; edited by Hamidun News
The DeepSeek V3 technical report is hard going: it is full of formulas, while the model's real code runs to thousands of lines with a pile of dependencies that obscure the core idea. Developer Vaibhaav Tiwari showed on Hacker News how to get around this. He fit the entire DeepSeek V3 architecture into 190 lines of JAX (Google's library for numerical computing and neural network training) and trained a scaled-down copy directly on a MacBook Air M5 in 2.6 hours. The result is a model with 36.2 million parameters instead of the original's 671 billion. It is not a strong model, but this code can be read like a textbook.
What's Inside These 190 Lines
The project is written without Flax, Equinox, or PyTorch. Parameters are plain dictionaries of arrays, and every part of the model is a function that can be read from top to bottom. The author reproduced DeepSeek V3's main techniques in a reduced form.
The first technique is Multi-head Latent Attention (MLA). Standard attention stores full keys and values for every token in the cache, which consumes memory on long texts. MLA stores a single small compressed vector per token and per layer, and reconstructs what it needs from that vector. A separate rotary (RoPE) key is shared across all heads and sits outside the compressed vector.
The second is the mixture of experts (MoE). Picture a call center: one generalist operator answers everyone, and there are 16 narrow specialists, with four of the best ones called in for each question. The same logic applies here: one shared expert is always active, plus 16 routed experts, from which four are selected for each token. As a result, 13.6 million of the 36.2 million parameters are active per token.
The third is balancing without an auxiliary loss function. To keep all the specialists in the call center loaded, each expert has a bias term that nudges the router toward idle experts. It affects only expert selection, not their weights. In addition, there is a small sequence-level balance loss with a coefficient of 1e-4, as in the paper.
The fourth is multi-token prediction (MTP). An additional module predicts not only the next token but the one after it, with a weight of 0.3 in the overall training loss. Computation runs in bf16 on top of f32 master weights.
How It Was Trained and What Came Out
The corpus consists of 207 detective books in the US public domain: novels by Dashiell Hammett (repeated three times in the data) and about 200 works from Project Gutenberg. In total there are 21.7 million tokens, with a BPE tokenizer of 8,192 tokens. Validation runs on held-out crime texts.
The main run: 12,000 steps, roughly 98 million tokens (about 4.5 epochs), 2.6 hours on a MacBook Air M5 with 24 GB of memory.
The most telling result is the balancing experiment. By step 2,000, with the bias term enabled, each expert received between 5.7% and 6.9% of tokens, against an ideal of 6.25%. Without it, the shares spread from 0.09% to 18.6%: some experts were overloaded while others barely worked. Yet validation loss was practically identical: 3.921 with balancing and 3.923 without. So the bias term evens out the load at no cost to quality, as DeepSeek's paper claims.
The model itself, by the author's account, is weak. Given the opening "The fog came in off the", it continues: "bay, and the great trees were spreading and the stars were shining". The grammar holds up, but the meaning is vague.
Limitations and Who Will Find This Useful
The author is upfront about what is missing. Every expert processes every token, and the outputs of unselected experts are simply zeroed out: the routing math is exact, but there is no sparse token dispatch and no expert parallelism. MLA has no weight absorption: keys and values are decompressed explicitly, which is easier to read but slower at inference. FP8 training, DualPipe, node-limited routing, YaRN context extension, and any post-training are not implemented. The scale is also incomparable: 6 layers against 61, and a 256-token context against 128 thousand.
Still, the project is easy to reproduce. It includes 39 tests covering causal masking, cache-versus-full-pass consistency, routing, MTP targets, bf16 master weights, and more. The code runs on Apple Silicon via jax-metal and is slower on CPU, for which a reduced config is provided. On an AMD RX 7600 XT with 16 GB, a full run takes about 50 minutes at roughly 35,000 tokens per second, with validation loss of 3.527 against 3.507 on the Mac.
One security warning: the checkpoint is saved as a Python pickle, and loading such a file can execute arbitrary code. Load it only from trusted sources, or better yet, train the model yourself.
The practical takeaway is simple. If you want to understand MLA, MoE, and MTP, you do not need to tackle production code: an evening with this repository and a laptop is enough. As a cheat sheet to DeepSeek's architecture it is more useful than many summaries, and as a testbed for experimenting with expert routing it is easier to work with than bulky frameworks.
- Nov 21, 2026Tencent Hy4 Fixes 76% of Real Code Vulnerabilities and Beats DeepSeek by 6 Points
- Nov 21, 2026DeepSeek Weights Show Traces of NVIDIA Blackwell GPUs: 445 of 469 Rows Match on B300
- Nov 21, 2026Jensen Huang Steps Down as NVIDIA CEO After 33 Years, CFO Colette Kress Becomes Interim CEO
- Nov 21, 2026AI Warehouse Clerk Broke a Hidden Rule: DeepSeek V4 Pro Failed in 18% of Cases, Gemini 3.8 Flash in 4.8%
Key Questions Answered
What's Inside These 190 Lines?
The project is written without Flax, Equinox, or PyTorch. Parameters are plain dictionaries of arrays, and every part of the model is a function that can be read from top to bottom. The author reproduced DeepSeek V3's main techniques in a reduced form. The first technique is Multi-head Latent Attention (MLA). Standard attention stores full keys and values for every token in the cache, which consumes memory on long texts.
How It Was Trained and What Came Out?
The corpus consists of 207 detective books in the US public domain: novels by Dashiell Hammett (repeated three times in the data) and about 200 works from Project Gutenberg. In total there are 21.7 million tokens, with a BPE tokenizer of 8,192 tokens. Validation runs on held-out crime texts. The main run: 12,000 steps, roughly 98 million tokens (about 4.5 epochs), 2.6 hours on a MacBook Air M5 with 24 GB of memory. The most telling result is the balancing experiment.
What are the key details on Limitations and Who Will Find This Useful?
The author is upfront about what is missing. Every expert processes every token, and the outputs of unselected experts are simply zeroed out: the routing math is exact, but there is no sparse token dispatch and no expert parallelism. MLA has no weight absorption: keys and values are decompressed explicitly, which is easier to read but slower at inference. FP8 training, DualPipe, node-limited routing, YaRN context extension, and any post-training are not implemented.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.