LANN-4 Author Shows Repeated Fine-Tuning of a Small LLM Hits a Ceiling

If you fine-tune the same small language model on examples for several rounds in a row, it does not get better from round to round. The first round gives a noticeable gain, then growth stops. That is the conclusion of the author of a series of Habr posts, whose experiments run on a model that fits on an ordinary home GPU. The next experiment will take about a day of GPU time, and before launching it the author asks readers to look for weak spots in the test design.

Why it all started

In the first article, the author explored whether a small model can get smarter over time without adding parameters. Instead of inflating the weights, he bet on three things: memory, careful fine-tuning, and verification of answers. In the second post, he explained where this leads. The experiments are steps toward the LANN-4 architecture, a neural network that learns throughout its "life." It is supposed to have a small, unchanging "brain" and memory kept outside it.

The author states up front that LANN-4 itself does not exist yet. The architecture has not been chosen or built, and the models in the experiments serve as test subjects. The publication contains no finished product and no benchmarks of the new system. It is a research report, and one with a negative result.

What did not work

The test was simple: does fine-tuning the same model on examples many times in a row produce a cumulative effect? It did not. The text paints the following picture:

• model size: fits on an ordinary home GPU;
• first fine-tuning round: a noticeable quality gain;
• later rounds: growth stops, and the ceiling is reached quickly;
• the author's conclusion: simple repeated fine-tuning does not turn a small model into a system that improves throughout its life;
• cost of the decisive test: about a day of GPU time.

The announcement does not reveal why growth stops; presumably the main text covers it. The continual learning literature usually points to forgetting of previously learned material, saturation on homogeneous data, and hitting the model's own capacity limit. But that is general context, not the author's claims. The announcement does not say which explanation applies here.

Why the result is useful

Negative results are rarely published in open AI experiments, even though they save other people's time. This one also bears on a practical question. The idea of a "small fixed brain plus external memory" is attractive: it promises learning without retraining all the weights, and operation on consumer hardware. But if the model itself stops absorbing new material after the first iteration, the entire load falls on the external memory and the answer-verification mechanism. That is where the author is now shifting his focus.

The format also deserves a mention: the author does not simply launch the decisive run, but publishes its design for critique in advance. A day on a single GPU is not that expensive, but on home hardware it is a day without other experiments. A flaw in the setup would cost more than the compute itself.

What practitioners can take from this

If you fine-tune small models for your own tasks, the post offers three takeaways. Do not expect repeated rounds on the same data to deliver comparable gains: measure quality after each round on a held-out set, and stop when the curve plateaus. Keep answer verification separate from the training data, or you may mistake a ceiling for progress, or the reverse. And if continual learning interests you, read the original article on Habr and try to find flaws in the design of the decisive experiment: the author asks for exactly that before the run.