The bug that fools you

Data leakage is when your model accidentally learns from information it will not have when making real predictions. The result is dangerous: the model looks excellent in testing, then fails in the real world. It is one of the most common and costly mistakes in machine learning.

There are two main kinds: target leakage and train-test contamination.

Target leakage

Target leakage happens when a feature contains information that only becomes known after the target, so it will not be available when you actually predict. The trick is to think about timing: when does each value become known, not just whether it helps the score.

A classic example: predicting who will catch pneumonia, using a feature like "took antibiotics". People take antibiotics after getting pneumonia, so that column is filled in after the outcome. The model looks brilliant in validation (the same pattern repeats there) but is useless in the real world, where a future patient has not taken antibiotics yet at the moment you need the prediction.

Another: predicting whether a student passes, using "received certificate". The certificate is awarded only after passing, so it leaks the answer.

The rule: exclude any feature that is updated or created after the target is known.

See it happen

Here certificate_points is recorded after the exam, so it leaks the result. Watch what it does to the score:

A suspiciously perfect score is itself a warning sign. When a model looks too good to be true, suspect leakage and inspect your features one by one: could this value really be known before the prediction?

Train-test contamination

The second kind: you let information from the validation set sneak into training, usually by preprocessing before splitting. For example, computing the mean for imputation over the whole dataset (including validation rows) before the split. The fix you already have: do preprocessing inside a pipeline, after the split, so each fold only ever sees its own training data. This is a big reason pipelines matter.

How to prevent it

For every feature, ask: would I actually have this value at the moment I make the prediction? If not, drop it. And always do your preprocessing inside a pipeline, so validation data never leaks into training. That matters most with cross-validation, where preprocessing outside the pipeline would quietly contaminate every fold.

Practice

For each case, say whether it is leakage and why: (a) predicting exam score using "hours studied", (b) predicting exam score using "final grade awarded", (c) filling missing values using the average of the entire dataset before splitting into train and validation. Write a one-line answer for each.