Researchers Prove Why Adversarial Training Makes Saliency Maps Sparse

Gradient-based saliency maps look noticeably cleaner in adversarially trained networks: instead of a noisy blob, only a few bright pixels remain. Researchers have observed this empirically, but there was no convincing explanation. The preprint arXiv:2610.10666 offers a rigorous proof for two-layer ReLU networks: the sparsity comes from the geometry of ℓ∞ attacks, which makes the model prefer gradients aligned with the coordinate axes.

Where the sparsity comes from

The paper starts from a known result. For a range of loss functions, adversarial training is equivalent to minimizing ordinary empirical risk with two added terms: a weight decay penalty and an adversarial total variation term. The second term essentially penalizes the model for how steep the function is, but it accounts for the norm that bounds the attack.

The key step comes next. Under an ℓ∞ attack, the perturbation lives in a cube, and to first approximation the worst case is determined by the ℓ1 norm of the gradient, which is the norm dual to ℓ∞. This is my interpretation, not a quote from the paper, but it fits well with the authors' conclusion: a penalty of this type behaves like LASSO and prefers gradients with a small number of nonzero components. The authors describe this as anisotropy of the gradient norm: directions along the axes are "cheaper" than diagonal ones.

What exactly is proven

The theorem is asymptotic. The number of data points and the number of neurons grow, while the regularization parameters simultaneously go to zero at a suitable rate. In the limit, the minimizers converge to a Bayes classifier, chosen from among all suitable candidates by minimal gradient norm and minimal Barron norm. The Barron norm measures the complexity of a function that a two-layer network can approximate, so the condition essentially selects the simplest solution in the class.

Key elements of the result:

• architecture: two-layer networks with ReLU;
• training: adversarial, with attacks bounded by the ℓ∞ norm;
• limit: growth of the sample size and network width, with regularization going to zero;
• convergence outcome: a Bayes classifier with minimal gradient and Barron norms;
• consequence: the anisotropy of the gradient norm yields sparse, axis-aligned gradients.

The experimental part is more modest than the theory. The authors compare the ℓ1 norm of the gradient and the sparsity after thresholding in naturally and adversarially trained models. The abstract gives no specific numbers, so you will have to consult the full text to see the scale of the effect.

Where the limits lie

There are several limitations, and they are significant. A two-layer ReLU network is far from modern convolutional networks and vision transformers. The condition on loss functions narrows the range where the equivalence holds. Asymptotics in data and width do not guarantee behavior on finite samples. Finally, this is a preprint without peer review.

There is also a conceptual caveat. A sparse saliency map looks convincing, but it does not follow that it reflects the model's logic more accurately. The network may simply have dropped some features in exchange for robustness, and the map's "cleanliness" may reflect lost information rather than a clearer explanation.

What to check in practice

If you use gradient saliency maps to audit models, take the training method into account when comparing them. Comparing the maps of a naturally trained model and a robust one by visual "neatness" is misleading: part of the effect is set by the attack norm. A simple check is to compute the ℓ1 norm of the gradient and the share of components above a threshold, as the authors do, and compare both models on the same inputs.

For ℓ2 attacks, expect a different geometry; the conclusion about axes cannot be carried over to them. For safe deployment, it helps to supplement gradient maps with methods that do not depend on a single norm, such as permutation tests and feature removal.