Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseCode / SDK · ActionableGitHub Trending AI→ original

Open-source MiniMind trains a 64M-parameter language model in 2 hours for 3 yuan

Want to understand how a language model works from the inside, rather than fine-tuning someone else's through ready-made libraries? The open-source MiniMind project provides all the code in plain PyTorch for a model with 64 million parameters (weights), which is hundreds of times smaller than typical assistants. The SFT stage, where the model learns to hold a dialogue, takes about 2 hours on a single RTX 3090 graphics card and costs about 3 yuan in rental fees. That makes it cheap to test ideas like DPO, LoRA and RL here rather than on a large model.

AI-processed from GitHub Trending AI; edited by Hamidun News
Open-source MiniMind trains a 64M-parameter language model in 2 hours for 3 yuan
Source: GitHub Trending AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceGitHub Trending AI ↗
Editor-in-Chief
Apache 2.0 licensecode, data and model weights are open
The full cyclepretraining, SFT, LoRA, DPO, PPO/GRPO/CISPO, distillation
Key metric #3Includes an OpenAI-compatible server, and runs via llama.cpp, vLLM and Ollama

Most developers know LLM training like this: you take a ready-made model, plug in a library, write a dozen lines, and everything fine-tunes "by itself." It's convenient, but a black box remains inside. The MiniMind project, which gained 188 GitHub stars in a single day on GitHub Trending, proposes the opposite: build and train a small language model entirely by hand.

The numbers in the headline mean the following. "64 million parameters" is the number of trainable weights in the model; typical assistants have tens or hundreds of billions, so MiniMind is hundreds of times smaller. "2 hours" is the actually measured time for the SFT stage (training the model to hold dialogue on "question — answer" examples) on a single NVIDIA RTX 3090 graphics card.

"3 yuan" is the cost of renting the GPU for those two hours.

The author, Jingyao Gong, compares conventional training wrappers to flying first class, and his own approach to assembling an airplane from a construction kit. He also criticizes online courses that explain LLMs superficially and with errors. For that reason, all the key algorithms in the repository are written in "bare" PyTorch, without high-level interfaces from third-party libraries.

What's inside and how to read "two hours"

The project covers the entire path: data cleaning and deduplication, pretraining (the model reads a corpus and learns to predict the next token), SFT, LoRA, DPO, reinforcement learning and distillation. LoRA is a lightweight fine-tuning method in which a small add-on to the weights is trained while the main model stays untouched; here it is implemented without the peft library. DPO trains the model on pairs of "good answer — bad answer." PPO, GRPO and CISPO are reinforcement learning algorithms in which another model or a verifiable rule scores the response. There is also Tool Use and Agentic RL (tool calling and training on the outcome of a multi-step scenario), and a switch between "think" and "answer immediately."

An important caveat: the two hours and three yuan apply only to the SFT stage. Pretraining is a separate step, but for those with a single card, the authors have prepared lightweight datasets: pretrain_t2t_mini at 1.2 GB and sft_t2t_mini at 1.6 GB. The full versions weigh 10 GB and more. The authors' test rig is eight RTX 3090 cards with 24 GB each, but the project supports both a single card and multiple cards (DDP, DeepSpeed), resuming after a crash, and charts in wandb and swanlab.

Everything needed for running and checking is included: evaluation on C-Eval, C-MMLU and OpenBookQA, context extension via YaRN, a minimal OpenAI API-compatible server, and a Streamlit chat interface. Ready-made weights run via llama.cpp, vLLM and Ollama, and can be connected to Open-WebUI and FastGPT.

What it teaches and where the limits are

The README also contains engineering observations that are useful beyond the project. The tokenizer here is intentionally small: in a tiny model, a large vocabulary would consume a noticeable share of the parameters, so the authors do not recommend retraining it. For small models, according to observations from the MobileLLM work, a "deep and narrow" architecture usually beats a "wide and shallow" one, although narrowness has a lower bound. And the MoE variant (where requests are routed among "experts") trains noticeably slower than a regular model when implemented natively in PyTorch: it can be accelerated only with specialized optimized libraries.

The authors describe the limitations themselves. A 64-million-parameter model knows few facts, and early versions sometimes produced nonsense instead of an answer. The tools in the training data are a few imitations, such as requests for the time, math and weather, so there is not yet a real ability to work with arbitrary APIs. If thinking and tool calling are enabled at the same time, the model usually does not show its reasoning: the data contains no joint examples.

The practical takeaway: MiniMind will not replace a production model, but it is an inexpensive testbed. It is handy for comparing DPO and GRPO, for understanding how your dataset behaves in SFT, and for teaching a team to read training code without library magic. The Apache 2.0 license allows free use of the code, and when you need to find where a pipeline breaks, the cheapest place to check is a model you can retrain in an evening.

Story

Key Questions Answered

What's inside and how to read "two hours"?

The project covers the entire path: data cleaning and deduplication, pretraining (the model reads a corpus and learns to predict the next token), SFT, LoRA, DPO, reinforcement learning and distillation. LoRA is a lightweight fine-tuning method in which a small add-on to the weights is trained while the main model stays untouched; here it is implemented without the peft library.

What it teaches and where the limits are?

The README also contains engineering observations that are useful beyond the project. The tokenizer here is intentionally small: in a tiny model, a large vocabulary would consume a noticeable share of the parameters, so the authors do not recommend retraining it. For small models, according to observations from the MobileLLM work, a "deep and narrow" architecture usually beats a "wide and shallow" one, although narrowness has a lower bound.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…