Habr AI: a meta-model for diagnosing neural network training detects failures from learning curves

Habr AI described an experimental meta-model that attempts to automatically understand what happens during neural network training. Instead of manually reviewing learning curves, the author proposes a separate classifier capable of recognizing underfitting, overfitting, and data problems based on metrics and curve shape.

Why This Matters

Typically, an engineer looks at train and validation accuracy, compares the gap between them, and tries to visually determine if there is progress or if training should be stopped. This approach works as long as there are few experiments, but quickly becomes routine when running dozens of models and tracking different scenarios. The author starts with a simple idea: if a human can read learning curves and notice typical patterns, then a separate model can be trained to do the same.

Model training → learning curves → features → meta-classifier → stop at the ideal moment.

The idea is that the meta-model analyzes not raw images or texts, but the state of the main model at a specific moment during training.

The potential benefit is clear: stop unpromising runs earlier, catch overfitting faster, and avoid wasting epochs on what no longer yields significant gains. However, the author himself honestly notes that the question of production efficiency and transferability to different tasks remains open: this is precisely a working hypothesis, not a ready-made industrial standard.

How the Dataset Was Built

To train such a diagnostic layer, the author first generated a separate dataset of experiments based on MNIST. Logistic regression, small and large MLPs, and two CNNs of different sizes were used as base models. A total of 270 runs were performed and evaluated not only at the end but also at intermediate stages after 1, 5, 6, 11, 16, 21, and 26 epochs. This is important: the meta-classifier must recognize problems not after the fact, but during training.

Several conditions were varied in each run:

• training set size
• random seed
• presence of artificial class imbalance
• type of data shift on test, including noise and invert

For each point, the author saved train, validation, and test accuracy, the gap between train and validation, the validation curve history, and the epoch number. The data were then labeled with diagnostic tags based on simple rules: underfitting if train accuracy was below 0.7; overfitting if the gap exceeded 0.15; dataset shift if validation accuracy was noticeably higher than test accuracy. These rules simplify the task and do not claim to be universal, but provide a starter set of labels for the experiment.

What the Tests Showed

Particular interest in the work is the set of features. Instead of raw graphs, the author extracted several compact characteristics from the learning curve: starting value, midpoint, endpoint, overall growth, and standard deviation as a measure of stability. These features, along with basic metrics, were fed into a multi-label classifier via MultiOutputClassifier. Among the candidates tested were Random Forest, XGBoost, Logistic Regression, and an ensemble of models to compare which algorithm best captures the dynamics of training. Random Forest showed the best result.

On aggregate classification, the model achieved approximately 0.89 micro F1 and 0.88 macro F1, and was particularly strong at detecting underfitting and data shift. Logistic Regression underperformed as expected, since it struggles to capture nonlinear relationships between curve shape and training state. The ensemble barely improved the result, which is also telling: in this setup, the quality of features and labels matters more than simply complicating the final classifier.

What This Means

The idea of a meta-model for training diagnosis looks practical: even in a simple experiment, it shows that learning curves can be not only viewed visually but also formalized. If the approach holds up under scrutiny on more complex datasets and real ML pipelines, it could become the foundation for intelligent early stopping and automatic monitoring of training quality.