How good is your model?

Once you build a model, you need to measure how good it is. Measuring quality is what lets you improve: you change something, then check whether the error went down.

The measure we start with is mean absolute error (MAE). Build it up from the last word:

Error for one student is actual - predicted. If a student scored 80 and you predicted 70, the error is 10.
Absolute means we drop the minus sign, so an error of -10 counts as 10. We care how far off we are, not in which direction.
Mean means we average those absolute errors across all the students.

In plain words: on average, our predictions are off by about this many marks. Lower is better.

Why you cannot test on the training data

It is tempting to predict on the same students you trained on and measure the error there. That is called an in-sample score, and it is misleading.

Here is why. Suppose that, purely by chance, in your small sample the students who studied the least also happened to have the highest previous scores. The model will seize on that coincidence and predict high marks for low-study students. On the training data it looks brilliant, because that is exactly where the coincidence lives. But on new students, where studying little usually means lower marks, it falls apart.

A model is only useful on new data. So you must measure it on data it did not learn from. The simple way: set aside some rows before training, then test on those held-out rows. That held-out set is your validation data, and train_test_split creates it for you.

Python

import pandas as pd
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error

students = pd.DataFrame({
    "study_hours":    [2, 5, 1, 6, 3, 8, 4, 7, 1, 6, 5, 3],
    "attendance_pct": [60, 85, 50, 95, 70, 98, 75, 90, 45, 88, 80, 65],
    "prev_score":     [50, 72, 40, 85, 58, 92, 65, 88, 35, 80, 75, 55],
    "exam_score":     [55, 78, 42, 90, 62, 95, 68, 89, 38, 84, 77, 60],
})
X = students[["study_hours", "attendance_pct", "prev_score"]]
y = students["exam_score"]

# Hold back some rows for validation. random_state makes the split repeatable.
train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)

model = DecisionTreeRegressor(random_state=1)
model.fit(train_X, train_y)

# In-sample: predict on the SAME rows we trained on (looks too good)
train_mae = mean_absolute_error(train_y, model.predict(train_X))
# Validation: predict on rows the model has NOT seen (the honest measure)
val_mae = mean_absolute_error(val_y, model.predict(val_X))

print("Training MAE (in-sample):", round(train_mae, 1))
print("Validation MAE (honest): ", round(val_mae, 1))

Look at the gap. The training error is tiny, but the validation error is larger. The validation number is the honest one, the measure of how the model will do in the real world. Improving that number is the whole game, and the next lesson is your first tool for it.

Exercise

Exercise — check your understanding

The validation truth val_y and the model's predictions are already set up. Compute the mean absolute error and store it in mae.

You’re reading for free. Sign in to keep your progress and earn a certificate when you finish.Sign in to keep my progress →

← PreviousYour first model Next →Underfitting and overfitting

Model validation

How good is your model?

Why you cannot test on the training data

Exercise