Better Workflow · Lesson 5 of 7

Cross-validation

Why a single split can mislead

A single train/test split has a weakness: your score depends on which rows happened to land in the validation set. With a small validation set, one lucky or unlucky split can mislead you. At the extreme, imagine validating on a single row, whichever model happens to nail that one point "wins", purely by luck.

You could reduce that noise by making the validation set bigger, but a bigger validation set means fewer rows left to train on, which gives a worse model. You are stuck choosing between a noisy score and a weaker model.

What cross-validation does

Cross-validation escapes that trap. It breaks the data into several equal parts called folds, say 5. Then it runs the experiment 5 times: each time, one fold is the holdout and the other four are training. Every fold takes a turn as the holdout, so every row is tested exactly once. You then average the scores.

Because the measure now uses all the data (each row contributes to the score once), it is far more reliable than any single split.

Python

import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score

students = pd.DataFrame({
    "study_hours":    [2, 5, 1, 6, 3, 8, 4, 7, 1, 6, 5, 3, 7, 4, 8],
    "attendance_pct": [60, 85, 50, 95, 70, 98, 75, 90, 45, 88, 80, 65, 92, 78, 99],
    "prev_score":     [50, 72, np.nan, 85, 58, 92, np.nan, 88, 35, 80, 75, 55, 84, 62, 95],
    "region":         ["Rural","Urban","Rural","Urban","Rural","Urban","Rural","Urban","Rural","Urban","Rural","Rural","Urban","Rural","Urban"],
    "exam_score":     [52, 78, 41, 90, 61, 96, 68, 89, 38, 82, 77, 59, 87, 66, 98],
})
X = students.drop(columns=["exam_score"])
y = students["exam_score"]

pre = ColumnTransformer([
    ("num", SimpleImputer(strategy="mean"), ["study_hours", "attendance_pct", "prev_score"]),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), ["region"]),
])
model = Pipeline([("pre", pre), ("rf", RandomForestRegressor(random_state=1))])

# 3 folds. We negate because sklearn reports the score as a negative number.
scores = -1 * cross_val_score(model, X, y, cv=3, scoring="neg_mean_absolute_error")
print("MAE for each fold:", scores.round(1))
print("Average MAE:", round(scores.mean(), 1))

Now your score is an average across folds, not a single lucky draw. Notice this only works cleanly because the pipeline carries the preprocessing into every fold automatically, one of the reasons pipelines matter.

The negative MAE quirk

scikit-learn follows a convention where a higher score is always better. Mean absolute error is the opposite (lower is better), so sklearn reports it as a negative number. That is why the code multiplies by -1, to turn it back into the normal, positive MAE you are used to.

When to use cross-validation

There is a tradeoff: cross-validation is more reliable, but slower, because it trains the model once per fold.

Small datasets: use cross-validation. The extra computing cost is tiny and the more trustworthy score is well worth it.
Large datasets: a single validation set is usually enough, and faster, since you already have plenty of data to test on.

A rough rule: if your model trains in a couple of minutes or less, cross-validation is worth it. Or just run it once and look at the fold scores, if they are all close, a single split would have been fine.

Exercise

Exercise — check your understanding

Cross-validation gave one MAE per fold in fold_maes. Compute the average and store it in avg_mae.

You’re reading for free. Sign in to keep your progress and earn a certificate when you finish.Sign in to keep my progress →

← PreviousPipelines Next →Gradient boosting and XGBoost