Утечка в OOD-бенчмарках: как загрязнённый класс обрушил AUROC детектора до 0.326

Researchers in a preprint on arXiv (arXiv:2607.19393, July 2026) showed that a common way of assembling OOD benchmarks can distort results: an anomaly detector scored an AUROC of 0.326 — below the random-guessing level of 0.5 — simply because the "foreign" class was actually present in the model's training set.

Why the detector turned out worse than a coin flip

The AUROC of 0.326 arose from benchmark leakage: the class labeled as OOD ("out-of-distribution") was among those the model had already been trained on. Its examples ended up inside the in-distribution set, and the detector was penalized for correctly recognizing them as familiar. After removing this class and retraining 35 models across two domains, the score rose to 0.911 — from a collapse below chance level to a near-excellent result. The metric itself did not change — only the cleanliness of the benchmark did.

What a leak fingerprint is

A leak fingerprint is a pair of signals by which the authors distinguish a contaminated benchmark from a clean one. Contamination reveals itself when a supervised classifier separates OOD examples almost perfectly (AUROC≈1), while an unsupervised detector collapses below 0.65 at the same time. This divergence is precisely the marker that the "foreign" class is in fact familiar to the model.

Key facts from the study:

• AUROC rose from 0.326 to 0.911 after removing the leaked class and retraining 35 models across two domains
• Leak fingerprint: supervised decodability AUROC≈1 paired with unsupervised detection below 0.65
• Validation across 52 settings (20 with leakage, 32 clean) — ResNet-50 and ViT-B/16 on CIFAR-10/100
• Diagnostic accuracy: sensitivity 18/20 and specificity 31/32 in embedding space
• Controls with the training set excluded worked perfectly — 20/20

How widespread the problem is

Standard cross-dataset benchmarks are mostly clean — that's the main practical takeaway. An audit of 24 standard near/far OOD pairs found leakage in exactly one pair (the notoriously hard CIFAR-100 vs. CIFAR-10) and in none of the far-OOD pairs, confirming the specificity of the diagnostic.

Under the corrected protocol, the perturbation method proved useless: a supervised reader recovers the OOD signal with an AUROC of 0.87–1.00, but no unsupervised detector does the same, and the method itself does not outperform ordinary Mahalanobis distance.

"Perturbation signals are decodable, but not detectable," is how the

preprint's authors phrase the main takeaway.

What this means

The finding is a warning for ML researchers: an anomalously low AUROC (below 0.5) may point not to a bad detector, but to a contaminated benchmark. The authors are not proposing a new OOD detection method, but a corrected protocol and a ready-made leakage test — and, for transparency, they retract an earlier erroneous correlation.

Frequently asked questions

What is OOD detection?

OOD detection (out-of-distribution) is the recognition of data that does not resemble what the model was trained on. It is needed so the model does not give confident answers on unfamiliar or anomalous inputs.

Why was the AUROC below 0.5?

The AUROC of 0.326, below the random-chance level of 0.5, arose from leakage: the "foreign" class was in the training set, so the detector was penalized for making the correct decision. After eliminating the leakage, the metric rose to 0.911.

Does this affect standard benchmarks?

Mostly no. An audit of 24 standard near/far OOD pairs found leakage in only one pair — CIFAR-100 vs. CIFAR-10 — while the usual cross-dataset benchmark construction scheme was found to be clean.