OOD Benchmark Leak: How a Contaminated Class Dragged a Detector's AUROC Down to 0.326

Researchers in an arXiv preprint (arXiv:2607.19393, July 2026) show that a common way of building OOD benchmarks can distort results: an anomaly detector scored an AUROC of 0.326, below the random-guessing level of 0.5, solely because the "foreign" class was actually present in the model's training set.

Why the detector did worse than a coin flip

The AUROC of 0.326 stemmed from benchmark leakage: the class labeled OOD ("out-of-distribution") was among those the model had already been trained on. Its examples ended up inside the in-distribution set, and the detector was penalized for correctly recognizing them as familiar. After the authors removed this class and retrained 35 models across two domains, the score rose to 0.911, from worse than chance to nearly excellent. The metric itself did not change; only the cleanliness of the benchmark did.

What is a leak fingerprint?

A leak fingerprint is a pair of signals the authors use to tell a contaminated benchmark from a clean one. Contamination reveals itself when a supervised classifier separates the OOD examples almost perfectly (AUROC ≈ 1) while an unsupervised detector fails, scoring below 0.65. That gap is the marker that the "foreign" class is actually familiar to the model.

Key facts from the study:

• AUROC rose from 0.326 to 0.911 after the leaked class was removed and 35 models were retrained across two domains
• The leak fingerprint: supervised decodability AUROC ≈ 1 paired with unsupervised detection below 0.65
• Validation across 52 settings (20 with a leak, 32 clean) using ResNet-50 and ViT-B/16 on CIFAR-10/100
• Diagnostic accuracy: sensitivity of 18/20 and specificity of 31/32 in embedding space
• Controls that excluded the training set worked perfectly, at 20/20

How widespread is the problem?

Standard cross-dataset benchmarks are largely clean, which is the main practical takeaway. An audit of 24 standard near/far OOD pairs found a leak in exactly one pair (the notoriously hard CIFAR-100 versus CIFAR-10) and in none of the far-OOD pairs, which supports the specificity of the diagnostic.

Under the corrected protocol, the perturbation method turned out to be useless: a supervised reader recovers the OOD signal with an AUROC of 0.87–1.00, but no unsupervised detector does, and the method itself does not outperform plain Mahalanobis distance.

"Perturbation signals are decodable but not detectable," is how the

preprint's authors sum up their main finding.

What this means

The result is a warning for ML researchers: an anomalously low AUROC (below 0.5) may point not to a bad detector but to a contaminated benchmark. The authors do not propose a new OOD detection method. They offer a corrected protocol and a ready-made leak test, and, for transparency, they retract an earlier erroneous correlation.

Frequently asked questions

What is OOD detection?

OOD (out-of-distribution) detection is the recognition of data that does not resemble what a model was trained on. It exists so that a model does not give confident answers on unfamiliar or anomalous inputs.

Why was the AUROC below 0.5?

The AUROC of 0.326, below the chance level of 0.5, resulted from a leak: the "foreign" class was in the training set, so the detector was penalized for making the correct call. Once the leak was removed, the metric rose to 0.911.

Does this affect standard benchmarks?

Mostly no. An audit of 24 standard near/far OOD pairs found a leak in only one pair, CIFAR-100 versus CIFAR-10, and the usual cross-dataset benchmark construction scheme was judged clean.