Open-source MiniMind trains a 64M-parameter language model in 2 hours for 3 yuan
Want to understand how a language model works from the inside, rather than fine-tuning someone else's through ready-made libraries? The open-source MiniMind project provides all the code in plain PyTorch for a model with 64 million parameters (weights), which is hundreds of times smaller than typical assistants. The SFT stage, where the model learns to hold a dialogue, takes about 2 hours on a single RTX 3090 graphics card and costs about 3 yuan in rental fees. That makes it cheap to test ideas like DPO, LoRA and RL here rather than on a large model.
AI-processed from GitHub Trending AI; edited by Hamidun News
Most developers know LLM training like this: you take a ready-made model, plug in a library, write a dozen lines, and everything fine-tunes "by itself." It's convenient, but a black box remains inside. The MiniMind project, which gained 188 GitHub stars in a single day on GitHub Trending, proposes the opposite: build and train a small language model entirely by hand.
The numbers in the headline mean the following. "64 million parameters" is the number of trainable weights in the model; typical assistants have tens or hundreds of billions, so MiniMind is hundreds of times smaller. "2 hours" is the actually measured time for the SFT stage (training the model to hold dialogue on "question — answer" examples) on a single NVIDIA RTX 3090 graphics card.
"3 yuan" is the cost of renting the GPU for those two hours.
The author, Jingyao Gong, compares conventional training wrappers to flying first class, and his own approach to assembling an airplane from a construction kit. He also criticizes online courses that explain LLMs superficially and with errors. For that reason, all the key algorithms in the repository are written in "bare" PyTorch, without high-level interfaces from third-party libraries.
What's inside and how to read "two hours"
The project covers the entire path: data cleaning and deduplication, pretraining (the model reads a corpus and learns to predict the next token), SFT, LoRA, DPO, reinforcement learning and distillation. LoRA is a lightweight fine-tuning method in which a small add-on to the weights is trained while the main model stays untouched; here it is implemented without the peft library. DPO trains the model on pairs of "good answer — bad answer." PPO, GRPO and CISPO are reinforcement learning algorithms in which another model or a verifiable rule scores the response. There is also Tool Use and Agentic RL (tool calling and training on the outcome of a multi-step scenario), and a switch between "think" and "answer immediately."
An important caveat: the two hours and three yuan apply only to the SFT stage. Pretraining is a separate step, but for those with a single card, the authors have prepared lightweight datasets: pretrain_t2t_mini at 1.2 GB and sft_t2t_mini at 1.6 GB. The full versions weigh 10 GB and more. The authors' test rig is eight RTX 3090 cards with 24 GB each, but the project supports both a single card and multiple cards (DDP, DeepSpeed), resuming after a crash, and charts in wandb and swanlab.
Everything needed for running and checking is included: evaluation on C-Eval, C-MMLU and OpenBookQA, context extension via YaRN, a minimal OpenAI API-compatible server, and a Streamlit chat interface. Ready-made weights run via llama.cpp, vLLM and Ollama, and can be connected to Open-WebUI and FastGPT.
What it teaches and where the limits are
The README also contains engineering observations that are useful beyond the project. The tokenizer here is intentionally small: in a tiny model, a large vocabulary would consume a noticeable share of the parameters, so the authors do not recommend retraining it. For small models, according to observations from the MobileLLM work, a "deep and narrow" architecture usually beats a "wide and shallow" one, although narrowness has a lower bound. And the MoE variant (where requests are routed among "experts") trains noticeably slower than a regular model when implemented natively in PyTorch: it can be accelerated only with specialized optimized libraries.
The authors describe the limitations themselves. A 64-million-parameter model knows few facts, and early versions sometimes produced nonsense instead of an answer. The tools in the training data are a few imitations, such as requests for the time, math and weather, so there is not yet a real ability to work with arbitrary APIs. If thinking and tool calling are enabled at the same time, the model usually does not show its reasoning: the data contains no joint examples.
The practical takeaway: MiniMind will not replace a production model, but it is an inexpensive testbed. It is handy for comparing DPO and GRPO, for understanding how your dataset behaves in SFT, and for teaching a team to read training code without library magic. The Apache 2.0 license allows free use of the code, and when you need to find where a pipeline breaks, the cheapest place to check is a model you can retrain in an evening.
- Nov 21, 2026California Sues OpenAI: Its AI Agents Escaped Control at Least 80 Times
- Nov 21, 2026OpenAI GPT-6 Cyber Found About 35 Flaws in iPhone, Mac and Other Apple Devices
- Nov 21, 202622 Claude Agents Spent Six Weeks on Black Hole Problems — and Solved Two
- Nov 21, 2026Google to split Gemini chat history from model training, so you can opt out without losing your history
Key Questions Answered
What's inside and how to read "two hours"?
The project covers the entire path: data cleaning and deduplication, pretraining (the model reads a corpus and learns to predict the next token), SFT, LoRA, DPO, reinforcement learning and distillation. LoRA is a lightweight fine-tuning method in which a small add-on to the weights is trained while the main model stays untouched; here it is implemented without the peft library.
What it teaches and where the limits are?
The README also contains engineering observations that are useful beyond the project. The tokenizer here is intentionally small: in a tiny model, a large vocabulary would consume a noticeable share of the parameters, so the authors do not recommend retraining it. For small models, according to observations from the MobileLLM work, a "deep and narrow" architecture usually beats a "wide and shallow" one, although narrowness has a lower bound.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.