Missing data

1 · What it is

Missing data are cells in a dataset that hold no value. Real datasets often have them, stored as blanks, NaN (short for “not a number”) or another placeholder. In pandas, the marker for a missing value, called NA, depends on the column’s data type. A made-up code such as minus one for “no measurement” is worse, because the model treats it as a real number.

How the gaps arose matters as much as how many there are. Rubin sorted missing data into three categories in 1976. Missing completely at random, or MCAR, means every value had the same chance of going missing, so the gaps are unrelated to the data. A weighing scale whose battery ran out is the classic example. Missing at random, or MAR, means the chance differs only between groups you can see in the data. A scale that fails more often on soft floors fits, as long as the floor type was recorded. Missing not at random, or MNAR, means the chance depends on reasons you cannot see, such as people with weaker opinions answering a poll less often. Most modern methods assume at least MAR.

Deleting rows is the bluntest option, and under MCAR it still gives unbiased means and regression weights. In pandas, dropna removes rows or columns that contain gaps. Dropping a whole column makes sense when one feature is mostly empty and adds little. Google’s crash course suggests building one dataset each way and keeping whichever trains the better model.

Imputation means filling the gaps with reasoned guesses. Simple imputation fills each blank with a column’s mean, median or most frequent value, or with a fixed constant. A k-nearest-neighbours imputer averages the value from the most similar complete rows. An iterative imputer, inspired by the MICE method, predicts each incomplete column from the others, round after round. Some models skip filling altogether. scikit-learn’s histogram gradient boosting learns, at each split, which branch rows with a missing value should follow.

Filling is not free. Mean imputation squeezes a column’s spread. In an air-quality example, the ozone standard deviation fell from 33 to 28.7 after the blanks were filled with the mean. Predicting values from other columns looks smarter but makes the guesses too tidy, so correlations come out too strong. Because a guess is rarely as good as the real value, Google recommends a Boolean column that marks which values were imputed. scikit-learn’s imputers can add that flag through an add_indicator option. Statisticians warn that a related shortcut, the indicator method, which fills blanks with zero and adds a flag, can badly bias regression estimates. Imputers learn from data, so they belong after the train and test split. Fitting them on test rows is a form of data leakage.