Why missing values are a problem

Data ends up with gaps for all sorts of reasons. A survey respondent skips the income question. A one-room home has no value for "second bedroom size". Whatever the cause, the gaps show up as NaN, and most models, including scikit-learn's, will refuse to run until you deal with them.

There are three common approaches.

Three ways to handle them

1. Drop the columns with missing values. The simplest option, but usually the worst. You lose every value in that column, even the good ones. Imagine 10,000 rows where one important column is missing a single entry; dropping it throws away 9,999 good values to remove one gap.

2. Impute (fill the gaps). Replace each missing value with a sensible number, usually the column's mean. You keep all the rows and columns. The filled value is a guess, so it is not exactly right, but this almost always beats dropping the column.

3. Impute, and mark what was missing. Do the imputation as above, and also add a new True/False column recording which values were originally missing. Sometimes the fact that a value was missing is itself a useful signal, so this can help. Other times it makes no difference.

Compare them yourself

Run this. It scores all three approaches on our student data with the same random forest, so you can see which wins. Notice we fit the imputer on the training data only, then transform the validation data, never the other way around.

Why imputation usually wins

On most real datasets, imputation beats dropping, because dropping a column to remove a few gaps also throws away all its good values. Approach 3 sometimes edges out Approach 2 and sometimes does not, so it is worth trying but not always worth keeping.

One honest note: our dataset is tiny, so the three scores will be close and the winner can vary. On a large real dataset the gap is clear, and imputation reliably wins. The lesson holds: reach for imputation first, drop columns only when almost all of a column is missing.

Exercise