Why Repeated Fine-Tuning Hits a Ceiling: The LANN-4 Study

This isn't an announcement of a finished architecture — it's an interim study with an honest account of what worked and what didn't. The author is asking for specific, substantive criticism. All rules and task sets were fixed in advance with checksums — the results can't be gamed after the fact — and they were independently recomputed using code written by someone other than the person who ran the experiment. That's a safeguard against self-deception.

LANN-4 is a neural network designed to learn continuously throughout its life, with a small, unchanging core and memory held externally, rather than growing with every update. It's a response to the industry's dominant trend of simply making models bigger. The author is interested in something else: how much usable capability can be squeezed out of a tightly bounded model size, so that every subsequent similar task becomes cheaper to solve.

Working principles: seven pillars of development

Development is guided by seven working principles — not a description of how the system works, but a set of guardrails:

• The system remembers the past across tasks instead of starting from a blank slate every time.
• Each task consumes exactly as much effort as it's worth, no more.
• Knowledge and experience live in a separate external memory rather than being baked into the model's weights.
• It learns new things without forgetting old ones.
• An answer, a hint, and a verified fact are three distinct things; an honest "I don't know" beats a confident fabrication.
• Any self-improvement is first checked in a sandbox and by an external judge — the model never edits itself unsupervised.
• Everything related to safety and permissions is handled externally and doesn't depend on the model's obedience.

Methodology: four rounds, three modes

The team used a single 200-task exam, identical at every stage, and ran the system through four rounds of fine-tuning. Three operating modes were compared:

• Accumulation (stacking) — the model is fine-tuned, and each subsequent round builds on the previous one.
• From scratch — each round starts from a clean model, with no accumulation.
• No training — the model simply retakes the exam (a control for randomness).

The key metric: does the share of correct answers climb from round to round?

Results: a jump, then a plateau

The share of correct answers in the accumulation mode: 53% → 59% → 54–57% → 54–57%. The first round produced a jump of roughly five percentage points. The second, third, and fourth rounds added nothing. The "from scratch" mode behaved identically — accumulated tuning gave no advantage. The control mode drifted slightly downward, as expected from noise. Performance on older tasks didn't degrade. An independent recomputation matched the original results exactly — the numbers held up.

One round of fine-tuning on verified solutions does produce a genuine gain, and it holds steady across all four rounds. But where does that gain come from? There are two honest explanations, and they point the project in very different directions: either the model absorbed the content of the correct solutions, or it simply adapted to the code's format.

The fork in the road: a placebo check

To tell these apart, the team ran a placebo check: they compared fine-tuning on correct "task + verified solution" pairs against fine-tuning on deliberately mismatched pairs, where the solution doesn't actually solve the task (they confirmed the mismatched pairs genuinely failed to solve the tasks). Everything else was kept identical: same number of steps, same examples, same conditions, same exam.

There's a gap in this check: it separates training on code-in-general from training on correct content, but only within a single task type. It doesn't catch overfitting to the code style of that same task type. Closing that gap requires a separate experiment on different task types — another day of GPU time.

Three questions for the community to weigh in on

The author is asking for substantive criticism on three points:

• Is an equal-cost placebo baseline enough to honestly call the first round's gain genuine learning? What's missing from the list of conditions?
• How can the overfitting-to-a-single-task-type gap be closed without doubling the GPU workload?
• Have others seen this same pattern — a jump on the first round, then a ceiling — when repeatedly fine-tuning a small model?

A negative result isn't a failure. It honestly narrows the field, showing where the capacity to accumulate learning does not lead. This result moves the project forward.