KDnuggets: Five outlier-detection methods agreed on just 32 of 816 flagged wines

KDnuggets tested five popular outlier-detection methods on a real wine dataset and got a result that breaks textbook intuition. Of 816 samples flagged by at least one algorithm, only 32 were shared by all the main methods.

Why the methods disagree

The experiment used the open Wine Quality Dataset from UCI: 6497 Portuguese Vinho Verde wines, including 1599 red and 4898 white, with 11 physicochemical features and tasters' scores. This matters because the data turned out not to be "textbook": six of the eleven features had noticeably skewed distributions, so classical assumptions of normality work poorly here.

The first problem appeared even before the algorithms were compared. If you treat a sample as an outlier whenever at least one of its 11 features is extreme, you get too many hits. In that mode, IQR flagged roughly 23% of wines and Z-Score about 26%. The authors attribute this to the multiple-testing effect: even if each individual feature rarely produces a chance extreme, checking 11 columns sharply raises the odds of catching an "anomaly" somewhere. So the analysis used a stricter rule: a sample counts as suspicious only if at least two features are extreme at once.

What the test showed

After that adjustment, the researchers compared five approaches: Robust Z-Score, IQR, Isolation Forest, Local Outlier Factor and Elliptic Envelope. Agreement between them was weak: the Jaccard coefficient for pairs of methods ranged from 0.10 to 0.30. In other words, different tools looked at the same dataset and saw different "oddities".

Of the 816 wines that at least one method labeled an outlier, only 32 samples made it into the shared list of all four main methods. Another 143 wines were flagged by at least three approaches. Everything else fell into a disputed zone: those samples were unusual only from the viewpoint of one or two algorithms.

"The question is not which method is best, but which kind of unusualness you are looking for."

• Robust Z-Score looks for strong deviations in individual features.
• IQR catches extreme values well without assuming a normal distribution.
• Isolation Forest evaluates each item across the whole feature set at once.
• LOF checks how far a point stands out from its local neighborhood.
• Elliptic Envelope relies on multivariate normality and therefore performed worse here.

The authors also warn about a trap in ML methods. Both Isolation Forest and LOF in their test used contamination=0.05, meaning the model was required in advance to flag 5% of objects as outliers. That is not "discovered truth" but a hard-coded quota. So an identical share of detections across algorithms of this class does not mean identical quality.

Which decisions helped

Three engineering decisions strongly influenced the outcome. First, instead of the ordinary Z-Score, they used a robust version based on the median and median absolute deviation: the standard version is too sensitive to the outliers themselves and flagged only 0.8% of rows in this dataset, versus 3.

5% for the more resistant variant. Second, red and white wines were scaled separately, because their baseline levels of chemical indicators differ, and merging them without correction creates false anomalies. Third, Elliptic Envelope was left out of the final "confidence vote".

The method assumes a multivariate normal distribution, and in the Wine Quality Dataset this condition was not met: one feature had a skewness of 5.4, and several others above 1. Excluding the method from the consensus is not cosmetic here but an example of sound analytical discipline: if the assumptions are violated, the tool should not determine the final conclusion.

The authors also cross-checked outliers against tasting scores from 3 to 9. Samples with extreme quality, either very good or very bad, landed on the consensus anomaly list about twice as often. This does not prove that the algorithms "understood taste", but it offers a useful sanity check: chemical deviations do occur more often where a wine also stands out in expert ratings.

What this means

The main takeaway for data science and ML practice is simple: an outlier is not an objective entity but the result of a chosen definition. If you need a workflow without labeled ground truth, it is wiser not to trust a single algorithm, but to build a consensus from several methods and only then decide, together with domain expertise, what to remove and what to keep as a rare but valuable signal.