Outliers

1 · What it is

An outlier is a data point that sits an unusually long way from the rest. How far is too far is left to the analyst to decide. Some outliers are mistakes, such as a value typed in wrongly or an experiment that was not run properly. Google’s course gives the example of a person accidentally typing an extra digit. Others are real, caused by ordinary random variation or by something genuinely interesting. A large dataset will almost certainly contain a few rare but real extremes.

Outliers can even make a model’s weights overflow during training. Spotting them starts with simple checks. A z-score says how many standard deviations a value sits from the mean. In small samples z-scores can mislead, because the largest possible one is (n − 1) ÷ √n. With nine values that ceiling is 8 ÷ 3, about 2.67, so a beyond-3 rule could never fire. Iglewicz and Hoaglin instead recommend a modified z-score built from the median and the median absolute deviation. Values scoring above 3.5 are labelled potential outliers. For data with many features, model-based detectors such as Isolation Forest help. Isolation Forest splits the data at random, and unusual points need noticeably fewer splits to be cut off on their own. It works because anomalies tend to be few and different.

What to do next depends on the cause. A value shown to be wrong should be corrected, or deleted if it cannot be fixed. In Google’s data-centre example, readings of 1,000 degrees are mistakes to delete. Rare hot days between 31 and 45 degrees are real, and clipping them is reasonable. Deleting outliers blindly is risky, because they often carry useful information about the process or about how the data was recorded. Sometimes the outlier is the point, such as an unusual card payment that signals fraud. When the cause is unclear, the NIST handbook advises against simply deleting the value. Resistant tools help instead, such as scaling with the median and IQR. Huber loss gives far-off points a smaller pull without ignoring them. In scikit-learn’s comparison, a ridge regression line was strongly pulled by outliers while the Huber line was not.