Habr AI: Meta-Model for Diagnosing Neural Network Training Flags Failures from Learning Curves

Habr AI has described an experimental meta-model that tries to work out automatically what is happening while a neural network trains. Instead of inspecting learning curves by hand, the author proposes a separate classifier that uses metrics and curve shape to recognize underfitting, overfitting and data problems.

Why this matters

Engineers usually look at train and validation accuracy, compare the gap between them and try to judge by eye whether there is still progress or whether it is time to stop training. That works while experiments are few, but it quickly turns into a chore when you have to run dozens of models and track different scenarios. The author starts from a simple idea: if a person can read training curves and spot typical patterns, a separate model can probably be taught to do the same.

Model training → learning curves → features → meta-classifier →

stopping at the ideal moment.

The meta-model analyzes not the original images or texts but the state of the main model at a given moment of training. The potential benefit is clear: stop unpromising runs sooner, catch overfitting faster and avoid wasting epochs on training that will no longer deliver a noticeable gain. The author is candid that effectiveness in production and transferability across tasks remain open questions: this is a working hypothesis, not a ready industrial standard.

How the dataset was built

To train this diagnostic layer, the author first generated a separate dataset of experiments based on MNIST. The base models were logistic regression, a small and a large MLP, and two CNNs of different sizes. In total there were 270 runs, evaluated not only at the end but also at intermediate points after 1, 5, 6, 11, 16, 21 and 26 epochs. This matters: the meta-classifier has to recognize problems during training, not after the fact.

Several conditions varied from run to run:

• training set size
• random seed
• presence of artificial class imbalance
• type of data shift on the test set, including noise and invert

For each point the author recorded train, validation and test accuracy, the difference between train and validation, the history of the validation curve and the epoch number. The data then received diagnostic labels by simple rules: underfitting if train accuracy is below 0.7; overfitting if the gap exceeds 0.15; dataset shift if validation accuracy is noticeably higher than test accuracy. These rules simplify the task and do not claim to be universal, but they provide a starting set of labels for the experiment.

What the tests showed

The feature set is of particular interest. Instead of raw plots, the author extracted several compact characteristics from each learning curve: the starting value, the midpoint, the end point, the overall growth and the standard deviation as a measure of stability. These features, together with basic metrics, were fed into a multi-label classifier via MultiOutputClassifier. Random Forest, XGBoost, Logistic Regression and a model ensemble were tested to see which algorithm reads training dynamics best.

Random Forest gave the best result. In the overall classification, the model reached roughly 0.89 micro F1 and 0.88 macro F1, and it was strongest at detecting underfitting and data shift. Logistic regression, as expected, fell behind, since it has more trouble capturing nonlinear relationships between curve shape and training state. The ensemble barely improved the outcome, which is also telling: in this setup, the quality of features and labeling matters more than simply making the final classifier more complex.

What it means

The idea of a meta-model for training diagnostics looks practical: even in a simple experiment, it shows that learning curves can be not only eyeballed but also formalized. If the approach holds up on more complex datasets and real ML pipelines, it could become the basis for smarter early stopping and automatic monitoring of training quality.