Developer fine-tunes a 4B-parameter model and hits 84.5% accuracy in one evening

A small open model with 4B parameters, fine-tuned in a single evening on a rented GPU, handles a noticeable share of routine business work. The author of a Habr write-up spent a few hundred rubles on training and got 84.5% accuracy on his own Russian-language tasks. The model handles almost two-thirds of the stream on its own at 95% correct decisions. On outside tasks it had not seen, the result is more modest: the top group, but not first place.

A task, not a chatbot

The author starts from a simple observation. A significant share of the tasks for which companies use AI comes down to choosing one answer from a set of options. Where to send a request. Who needs to approve an application. Whether a complaint is urgent. None of this requires a large generative model with long chains of reasoning. It requires a fast classifier with a clear cost of error, and it can run on a company's own infrastructure.

This is where a small fine-tuned model can compete with the cloud giants. It is cheaper to run, and the data never leaves the company's perimeter.

What the measurements showed

The model was compared with YandexGPT, GigaChat, and open alternatives. The experiment's parameters, according to the announcement:

• model size: 4B parameters, open weights;
• training cost: a few hundred rubles of GPU rental, about one evening;
• accuracy on the author's own tasks: 84.5%;
• automation: almost two-thirds of the stream at 95% correct decisions;
• on unfamiliar tasks: the top group, no claim of leadership;
• the model and code are published.

The gap between 84.5% and 95% is most plausibly explained by a confidence threshold. Confident answers go to automation, and doubtful ones go to a human. This is my interpretation, not a quote from the announcement. Such a setup is closer to real-world operation than a single accuracy figure across the whole sample.

The test on tasks that none of the models had seen deserves separate mention. Fine-tuned models easily look strong on data close to their training set. An honest measurement on unfamiliar material shows where specialization ends. A result of "top group, but not the leader" looks more credible than victory reports.

Limitations

I have seen only the announcement, not the full text, so some details remain unknown: which base model was used, which datasets, and how the samples were built. The benchmark is the author's own, and no one has yet confirmed independent reproducibility. The figures of 84.5% and 95% apply to the author's specific tasks and do not automatically carry over to your data.

How to put this to use

The author also promises a breakdown of pitfalls and a list of metrics worth demanding from a contractor. The announcement does not reveal the full list, but the logic of the experiment suggests a minimal set of questions for any deployment:

• What share of the stream goes to automation at a given accuracy, for example 95%?
• How does the model behave on data that was not part of its training?
• How much does an error cost in your process, and what confidence threshold follows from that?
• Has the solution been compared with cloud models on your own sample?

The practical move is to assemble a labeled sample of your own requests, run the open model from the publication on it, and compare the results with your current solution. If automation reaches half of the stream at an acceptable cost of error, you can move on to calculating the economics. If not, a few hundred rubles for GPU rental is a small price for the answer.