Researchers Compare CNN, ViT and Mamba Attention Against Human Gaze

A team of authors measured how closely the attention of three families of vision models matches where humans actually look. The comparison covered convolutional neural networks (CNN), Vision Transformer (ViT) and Vision Mamba (ViM). The models were tested on three classes of tasks: fine-grained classification, medical imaging and emotion recognition. Eye-tracking data served as the reference, and similarity was scored with three metrics. The paper is on arXiv under number 2610.12703.

How the match was measured

Most interpretability research remains qualitative: authors show attractive heatmaps and draw conclusions by eye. This study takes a different route. Post-hoc attribution maps and human gaze heatmaps are brought to a common form through a standardized normalization pipeline. Both end up as spatial probability distributions, so they can be compared directly. The authors applied different XAI methods to each architecture, because what works for convolutions transfers poorly to transformer attention or to Mamba's state-space blocks.

Similarity was computed with three complementary scores:

• Jensen-Shannon similarity: how well the supports match, that is, the regions where both maps have any mass at all;
• Chi-square similarity: how much the distribution of mass across pixels differs;
• Pearson correlation: whether the structure matches, that is, whether values rise and fall in the same places.

Each metric covers the blind spot of the others. A single number is not enough for an honest conclusion, and the most interesting results are built on exactly this point.

Three ways of looking at an image

On average, CNNs show a more global focus. Their saliency maps are spatially blurred, but they correlate well with human perception in general scenes. Humans also take in the whole scene first, and convolutional networks turn out to resemble them.

ViTs work differently. Their attention is fragmented, at the patch level, and often tied to discriminative micro-features: a small texture, a detail of plumage, the edge of an object. For classification this is a sensible strategy. But it does not have to match how a human looks.

The most nontrivial case is Mamba. It is well explainable on medical tasks, but on natural images its intensity distributions diverge from human ones. The authors show that the cause is visible in the metrics themselves: a near-perfect overlap of supports coexists with a vanishingly small linear correlation. Put simply, the model highlights the same regions as a human but distributes "weight" within them differently. Looking at overlap alone, Mamba looks excellent. Looking at correlation alone, it looks poor. The truth is that both metrics describe different sides of the same picture.

Separately, the authors tested how freezing the backbone and ensemble strategies affect interpretability. The abstract gives only a general conclusion: architectural and training choices noticeably change the agreement between a model's saliency map and human attention. It contains no specific numbers; those will have to be found in the full text of the paper.

Practical takeaways

For teams building models in medicine and other high-stakes fields, the takeaway is simple: the choice of architecture affects not only accuracy but also how verifiable the explanations are. A heatmap on its own proves nothing. It needs to be checked against a reference, and preferably against several metrics at once.

A practical checklist that follows from the work:

• do not rely on a single similarity metric: region overlap and correlation can give opposite signals;
• choose the XAI method to fit the architecture rather than applying the same one to all models;
• check explanations separately for each domain: judging by the results, Mamba behaves differently on medical and natural images;
• recompute agreement after freezing the backbone or building an ensemble.

The approach has an obvious limitation: matching human gaze is not the same as being correct. A model can look elsewhere and still be right. But as an audit tool in regulated fields, a quantitative comparison with eye tracking looks considerably more reliable than judging maps by eye.