97 hours on a single GPU: a self-improving neural network experiment and the trap of pretty metrics

Imagine you have a language model and want to give it a new skill, such as solving differential equations or writing Rust code. The standard route is fine-tuning. That takes data and compute, and it inevitably leads to catastrophic forgetting, where the model loses old abilities as it picks up new ones. But what if you could plug in skills the way you install apps on a smartphone: quickly, modularly, with no side effects? An independent researcher set out to test exactly that idea and published a detailed write-up on Habr. The result: 97.5 hours on a single RTX 4090, 22 experimental iterations, and one of the most instructive disappointments in machine learning in recent memory.

Modular extension of language models isn't a new concept on its own. The industry has long discussed approaches such as LoRA adapters, mixture of experts, and various plugin architectures. The author wanted to go further: build a system in which the model doesn't just call external modules but effectively improves itself, folding new capabilities into its work without a full retraining cycle. For anyone working with limited compute, that sounds like the holy grail. And that describes the vast majority of independent researchers and small teams, who have no access to clusters of thousands of GPUs.

Technically, the architecture worked. The modules plugged in, the system ran stably, and the metrics on validation datasets looked convincing. The researcher went through two dozen iterations, refining the approach each time, and at some point the numbers became genuinely beautiful. Loss went down, accuracy went up, and the training curves showed exactly the dynamics any ML practitioner hopes to see. On paper, it looked like a success.

Then came the moment of truth: testing on real tasks. What happened next is familiar to many practitioners but rarely discussed openly. By formal metrics, the model showed a brilliant command of the "language of mathematics," yet it turned out to be completely unable to solve concrete math problems. It had learned to imitate the form without mastering the substance. It produced plausible-looking derivations, used correct notation, and built chains of reasoning that seemed logical at first glance, but the answers were wrong. This is a classic example of what the community calls Goodhart's law applied to machine learning: when a metric becomes a target, it ceases to be a good metric.

The case exposes one of the fundamental problems of modern machine learning: the gap between optimizing metrics and actual competence. Language models are extremely good at detecting statistical patterns and reproducing them. But reproducing a pattern and understanding the logic behind it are fundamentally different things. A model can learn that certain math expressions are usually followed by certain symbols without grasping why those symbols belong there. For a researcher watching the loss curve and accuracy, the difference stays invisible until the system hits a task that demands real generalization.

But the story doesn't end there, and the ending is what makes it truly interesting. According to the author, the model eventually "found a way out on its own": under certain conditions, the system began to exhibit behavior that had not been explicitly programmed. The details of this breakthrough deserve separate attention because they touch on one of the hottest topics in AI research: emergent behavior, where complex and unexpected problem-solving strategies arise from simple rules. Whether this is genuine emergence or just a lucky combination of architectural choices remains an open question, but the fact itself deserves close study.

The experiment matters less for its specific results than for the lessons it offers. First, it's a reminder of how fragile metrics are as a tool for measuring progress. Second, it shows that serious language model research is still possible on consumer hardware, albeit with significant limitations. Third, it highlights the value of publishing failures openly. An industry obsessed with record benchmarks and press releases about the next breakthrough badly needs honest accounts of how attractive ideas collide with reality. Those are the stories that move science forward: not victory announcements, but careful analysis of what went wrong and why.