Developer trains a 260M language model from scratch on an 8 GB RTX 4060

Most stories about "building your own model" turn out to be fine-tunes of someone else's weights. This one is different. The author of the dpp-gpt line did everything himself: tokenizer, pretraining, SFT, RL, and benchmarks. The hardware is a home RTX 4060 with 8 GB of memory. The latest version, 2.2 Pro with 259M parameters, does not outperform professional small models, but the project shows well what a full training cycle looks like without a lab budget.

What's inside and what it was trained on

The architecture will look familiar from Llama: RMSNorm, RoPE, SwiGLU, GQA. The model has 20 layers, a width of 1024, an 8192-token context, and its own BPE vocabulary of 16,384 tokens in which digits are separate tokens. Languages: Russian, English, French.

Pretraining took about 31B tokens. The mix was assembled carefully: Wikipedia in three languages (23%), CulturaX web text, Python code, math (14%), poetry, translations, and synthetic data for text-handling tasks.

The most interesting part is the math. A script generates it, and each answer is written out by place value: hundreds, tens, ones, then the total. Python verifies every example, so the data contains no errors. This is how the model learns to handle numbers up to 40 digits: it repeats an algorithm rather than guessing the answer.

The SFT set grew to 60M tokens (about 122,000 dialogues). DeepSeek generated almost all of it via OpenRouter, and code double-checked the answers containing numbers. The data cost roughly 3,000 rubles. To keep the model from forgetting its base knowledge, 32% pretraining data was mixed into the training.

RL on 8 GB: what worked and what broke

The final version was produced with GRPO. For each prompt, the model writes 8 answers, receives a reward, and learns from the differences within the group. The reward comes either from code (arithmetic, tests for Python, JSON, tool calls) or from a local judge, Gemma 4 E4B, for essays, poems, and facts.

One useful technique is difficulty tuning. If a task is always solved or never solved, all 8 answers get the same reward and there is no gradient. So such task types are sampled less often, while those where the model solves about half are sampled more often.

Classic reward hacking showed up too. In the first runs, code scored text answers using indirect signals, and the model learned to answer briefly. Essays and general questions broke. The author had to add the judge and rebalance the training.

Once the obvious problems were fixed, progress stalled: each new checkpoint won in some areas and lost in others. Simple weight averaging helped: 0.4 · GRPO8 + 0.3 · GRPO9 + 0.3 · GRPO5. This "soup" turned out better than any of the individual checkpoints.

Another setback: SFT hurt the math. On the author's own benchmark, the base model scored 47.6%, dropping to 26% after the first SFT and recovering to 36.8% after the data mix was tuned.

The numbers, unvarnished

On the author's own set of 2450 questions, the average rose from 8.3 (version 2.15) to 24.2. The model leads in arithmetic and tool calling, but on average it is only on par with Falcon-H1-Tiny 100M and behind Gemma 3 270M. The author adds a caveat: the set was built around his formats and favors his model.

On standard English tests, the picture is weaker. HellaSwag: 28.9%, against 39.3% for Gemma 3 270M and 42.9% for SmolLM2 135M. GSM8K: 1.8%, IFEval: 14.1%. The base model without SFT scores the same 28.8% on HellaSwag, which means pretraining sets the ceiling: 31B tokens versus trillions. RL did not move these tests; its gains are in arithmetic, tools, JSON, and instruction following.

The weak spots are also spelled out: multiplication of 5–6-digit numbers is correct in about 30% of cases, the model loses count between messages, follows commands like "answer in one word" poorly, and the "with thoughts" mode does not help yet. Tool calling works only in the format it was trained on.

What to take from this

First: for narrow tasks (structured output, function calling, arithmetic), verifiable synthetic data and code-based rewards give a noticeable gain even on a tiny model. Second: if an RL reward has a weak spot, the model will find it, so you need diverse checks and a judge. Third: averaging the weights of nearby checkpoints costs almost nothing and may beat the search for a "best" one. And fourth: on general knowledge, a small model with a small token budget is no competitor. The author's next steps are a 140M flash version with a corrected share of Python, and an enlarged vocabulary with local attention for speed.