Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

Qwen3-0.6B Fine-Tuned on a Mac mini: Errors Dropped to 3 out of 30 from 9

The tiny Qwen3-0.6B answers questions about a blog directly in the browser, but it used to invent details that were not in the retrieved snippets. After fine-tuning on a Mac mini M4 in 1 hour 20 minutes, wrong answers on 30 questions dropped to 3 from 9, and correct answers went from 17 to 16. On 15 off-topic questions, the model honestly declined to answer. For a small model, it is better to train on short answers grounded in the text than on summaries.

AI-processed from Habr AI; edited by Hamidun News
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Training4,500 examples from Claude Opus, one epoch, LoRA
Version v1 got worse8 correct answers out of 30 instead of 17
Key metric #3Validation loss kept falling, while correct answers went 18, 11, 9

Imagine searching your blog or documentation: the system finds four relevant snippets, and a small language model running right in the browser has to answer briefly based on them. It sounds convenient: no server, no token fees. But the 0.

6-billion-parameter model (Qwen3-0.6B) often makes up details that are not in the snippets. The developer set out to see whether fine-tuning could fix this.

The result: on 30 questions that the blog does answer, wrong answers dropped to 3 from 9. Answers consistent with the source fell to 16 from 17 for the original model, and the model declined to answer all 15 questions outside the blog's topics. The most useful part of the story is how the author got there: three of the four attempts failed.

How the experiment works

The model receives a short system prompt, four numbered snippets from semantic search, and a question. It must answer in no more than two sentences and only from these snippets. If the snippets contain no answer, it should say that the blog does not appear to cover the topic.

The evaluation set had 45 questions: 30 with an answer in the blog and 15 without one. Training the model on the author's own posts would have been cheating, and there are only 16 of them anyway. So the author took 2,193 openly licensed technical documents: the Go blog, the Rust blog, Rust RFCs, and Python PEPs. The documents were split into snippets and embedded the same way as on the site, so the model would get used to real search noise. Sometimes the answer is in the first snippet, sometimes in the fourth, and sometimes a nearly identical passage from another post sits alongside it.

Claude wrote the questions. In some cases the answer was among the retrieved snippets; in others the source document was removed so the model would learn to refuse. There were also "near misses": questions on a related topic that the snippets do not answer. Claude Opus then wrote the answers. This is distillation: a large model demonstrates the behavior, and a small model imitates it. Training ran with LoRA and the mlx-lm library on a Mac mini with an M4 chip and 24 GB of memory. One epoch over 4,500 examples took about 1 hour 20 minutes. Each version was compared against the original model exported to the same 4-bit format, so only the fine-tuning differed.

Three failures on the way to the result

The first version was worse than the baseline: only 8 of 30 answers were grounded in the source, versus 17. At least it produced no invented answers to off-topic questions. The model picked up two habits from Opus. The first was adding hedges such as "details are not in the blog." These appeared in roughly 10% of the training answers, but the fine-tuned model produced them in almost one of every three answers, often where the needed information was in the snippets. The second was packing in as many details as possible. As a result, "king" and "queen," which sit close together in the article, turned into "king" and "bicycle," which were supposedly close to each other.

The second version stripped the hedges with a script, but only 9 of 30 answers were correct. The most instructive part is the checkpoints saved after a quarter, a half, and a full epoch. Validation loss fell at each one, meaning the model copied Opus's wording ever more precisely. Yet correct answers on blog questions dropped: 18, 11, and 9. A model of this size reproduces the form of a rich answer but not its content, and the better it copies, the more it gets confused.

In the third version the questions and snippets stayed the same, and only the answers were rewritten: a single short sentence stating the fact, as close as possible to the source's wording. No merging of facts from different snippets and no comments about missing information. The average answer length fell from 180 to 123 characters.

The result: 16 of 30 answers matched the source (versus 17, a one-answer difference that falls within noise for a sample this size), and wrong answers dropped to 3 from 9. For example, on a question about a GPU for Llama 70B, the base model produced a paragraph of advice drawn from snippets about embeddings and AI agents, while the fine-tuned model declined. The price of caution: on 11 of the 30 questions that do have an answer in the blog, it also says the topic is not covered, compared with 4 cases for the base model.

For a search page where the snippets are visible below the answer, that is a reasonable trade: a few extra seconds of reading beats a confident falsehood.

The fourth version added 1,200 broad questions such as "What does the blog say about X?" to reduce unjustified refusals. Correct answers fell to 13 of 30, worse than the third version.

What to take away

  • Small models pick up the style of examples more easily than the skill behind it. Teach them only what they can reproduce from the content.
  • Don't rely on validation loss. Check checkpoints on your actual task.
  • Assign small models the extraction of a specific fact, not the synthesis of several sources.
  • Carelessly generated training data produces behavior other than what you intended.

The author has published the model, dataset, and code openly, and loading it in the browser now requires less data than the original did.

The limitations are worth remembering too. Thirty questions are too few for statistical conclusions, and the drop in errors from 9 to 3 should not be treated as a scientific result. The answers were judged by the same Claude model family that wrote the training examples, so the judge may have favored a familiar style. If you repeat this approach for your own documentation, rewrite the prompt and the refusal phrasing for your domain and regenerate the answers from scratch.

Story

Key Questions Answered

How the experiment works?

The model receives a short system prompt, four numbered snippets from semantic search, and a question. It must answer in no more than two sentences and only from these snippets. If the snippets contain no answer, it should say that the blog does not appear to cover the topic. The evaluation set had 45 questions: 30 with an answer in the blog and 15 without one. Training the model on the author's own posts would have been cheating, and there are only 16 of them anyway.

What are the key details on Three failures on the way to the result?

The first version was worse than the baseline: only 8 of 30 answers were grounded in the source, versus 17. At least it produced no invented answers to off-topic questions. The model picked up two habits from Opus. The first was adding hedges such as "details are not in the blog." These appeared in roughly 10% of the training answers, but the fine-tuned model produced them in almost one of every three answers, often where the needed information was in the snippets.

What to take away?

Small models pick up the style of examples more easily than the skill behind it. Teach them only what they can reproduce from the content. Don't rely on validation loss. Check checkpoints on your actual task. Assign small models the extraction of a specific fact, not the synthesis of several sources. Carelessly generated training data produces behavior other than what you intended.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…