Fine-Tuned 0.8B-Parameter Model Beats GPT-5.4 at Grammar Annotation

The team behind an English-learning platform fine-tuned small Qwen3.5 language models and launched the 0.8B-parameter version for all of the service's students. The model reads lesson transcripts and tags grammar concepts, which feed a grammar mastery tracker. On two hand-labeled benchmarks, this model and a 4B reference version beat prompted GPT-5.4 and GPT-5.6 Sol on both precision and recall. Serving costs fell roughly 16x. In an online experiment, student engagement rose 15.8%.

What problem were they solving?

Corrective feedback is among the best-supported factors in second-language acquisition. But in a live lesson, corrections are spoken and then lost in the stream of speech. They don't add up to a picture of which constructions a student already uses confidently and where they make the same mistakes again and again. Frontier models can extract that picture from transcripts of student-teacher dialogues. At the scale of a whole platform, though, this is expensive, because every lesson becomes a long request to a costly model.

The authors tackled this with fine-tuning. They took labels generated by teacher models, filtered them, and rebalanced the classes so that rare grammar concepts would not get lost. They then fine-tuned Qwen3.5 in two sizes on that data.

The contract lives in the weights, not the prompt

The main engineering trick is that the annotation rules are written into the adapter weights. Prompted models need a detailed instruction every time: what counts as a concept, how to pick the evidence fragment, when an answer counts as correct. Here, the instruction is baked into training. As a result, the 0.8B model runs with a compact, consistent prompt. The savings come from two places at once: the model is smaller, and the input context is shorter.

The comparison used nested criteria of increasing strictness:

• concept match;
• match of the text fragment the annotation relies on (evidence span);
• correctness of the judgment itself.

At all three levels, both fine-tuned models, the production 0.8B and the 4B used for comparison, achieved higher precision and recall than GPT-5.4 and GPT-5.6 Sol. Serving the 0.8B model costs roughly 16x less.

The team also tested the effect on the product. An online feature-level experiment produced these results:

• student engagement: +15.8%;
• scheduled lesson hours: +2.1%;
• GMV from new lessons: +13.2%.

What the abstract leaves out

The abstract does not disclose several important things: the absolute precision and recall values, the size of the training set, and the duration and design of the online experiment. It does not name the platform or say which models generated the training labels. The frontier models took part in the comparison with prompts only, without any task-specific fine-tuning. The benchmarks are domain-specific, so the result cannot be transferred to any text-annotation task. This is a published preprint, and there is no independent verification yet.

What you can take into practice

The task here is narrow and repeats millions of times: structured output on a clear schema. This is exactly where a small specialized model beats a general-purpose one. If your team has a stable response format and a stream of similar requests, it makes sense to calculate what a prompted frontier setup costs at real volume and compare that with a fine-tuned model of 1–4B parameters.

The workflow from the paper goes like this: first collect labels with a strong model, then filter them and balance the class distribution, then fine-tune an adapter and cut the prompt to a minimum. Evaluate the result on nested criteria, as the paper does, so you can see where the model loses quality: in choosing the class, in grounding to the text, or in the final judgment. Roll it out through an online experiment with business metrics, not on a single offline benchmark.