Hebbian Learning Without Backprop Collapsed Into k-means and Lost on Every Front

The experiment the author describes on Habr fits in one line: a neural network's hidden layer was trained without backprop, using only local Hebbian rules, and compared with gradient descent on three axes. The score was 0:3. The Hebbian approach did not beat backprop on accuracy, in the few-labeled-examples regime, or on forgetting when switching tasks. The rule itself ultimately collapsed into plain k-means.

The source material is published as an announcement with a short summary, so the specific datasets and metric tables remain in the original article. The logic of the experiment and its conclusions are clear without them, though.

What they tried and why

Hebbian learning rests on an old idea: a connection between neurons strengthens when they are active at the same time. The key property of such a rule is that it is local. A neuron does not need the global error signal that backprop pushes through the whole network. This is attractive for neuromorphic hardware and for biological plausibility. That is why the question keeps coming back: can you get good features without gradients?

The author framed the task pragmatically: replace the hidden layer's training with local rules and see where such a network wins. The bets were on three scenarios where backprop has weak spots: few labels, catastrophic forgetting and overall accuracy.

Where everything broke

According to the author, there were enough problems for a story of their own.

• The network killed its own neurons. This is a classic ailment of competitive learning: a neuron that stops winning receives no updates and stays dead. The layer's capacity shrinks even though it formally has the same number of neurons.
• The evolution of rules fooled itself. When the form of the update rule is found by evolutionary search, the fitness function becomes the goal, and the algorithm finds loopholes rather than useful features. The metric goes up; the quality of the representation does not.
• The author's own mistakes. He writes plainly that he made errors, and this is the most valuable part of the write-up: negative results in this area are rarely published.

Why k-means was inevitable

The ending looks inevitable. If you keep competition between neurons in the Hebbian rule (one wins, as in winner-take-all) along with weight normalization, the update reduces to moving the winner's vector toward the input: w ← w + η(x − w). This is the online k-means formula. Each neuron becomes the centroid of a cluster of inputs.

Such a layer honestly finds structure in unlabeled data, but the cluster structure does not have to match class boundaries. Backprop optimizes features for a specific task; k-means optimizes for the geometry of the input. Hence the loss on accuracy. The expectation that cluster-based features would help when labels are scarce did not pan out either. Nor did this scheme win on forgetting.

Practical takeaways

For anyone experimenting with alternatives to backprop, this story suggests concrete steps.

• First compare the new rule with k-means and PCA. If the results match, there is nothing new to invent.
• Track the share of active neurons during training. Dead units show up immediately if you count them.
• When searching for rules evolutionarily, check what you find on tasks that were not part of the selection.
• Keep a strong backprop baseline on the same data and the same compute budget.

It does not follow that local learning is a dead end. But a simple Hebbian rule on a single hidden layer, judging by this experience, remains a decent implementation of clustering and nothing more.