Trying different models

Now that you can measure accuracy honestly with validation, you can compare different models and keep the best one. For a decision tree, the most important choice is how deep it grows.

Depth, leaves, and the two ways to fail

A tree makes its prediction by asking a series of yes or no questions. Each question is a split, and the depth is how many splits happen before a prediction. Every split doubles the number of groups: one split makes 2 groups, two splits make 4, and ten splits make over a thousand. The final groups are called leaves, and the tree predicts using the students who land in each leaf.

This sets up a balancing act:

• A very deep tree overfits. With many splits, each leaf holds only a handful of students. Its predictions match the training students almost exactly, but they rest on so few examples that they are unreliable on new data. Great on training, poor on validation.
• A very shallow tree underfits. With only a few splits, each leaf holds a wide mix of students, so predictions are rough for everyone, poor even on the training data.

You want the sweet spot in between: deep enough to capture real patterns, shallow enough to stay reliable on new data. You find it by watching the validation error.

Controlling depth with max_leaf_nodes

max_leaf_nodes is a simple way to control this. More leaves moves you toward overfitting, fewer toward underfitting. To compare values cleanly, write a small reusable function that returns the validation MAE for a given setting, then loop over a few settings:

Run it and read the errors. Whichever number of leaves gives the lowest validation MAE is the best of the options. (On a dataset this small the differences can be noisy; on a large real dataset you see a clear low point.)

The takeaway

Every model can fail in one of two ways:

• Overfitting: it learns coincidences in the training data that do not hold later, so new predictions are off.
• Underfitting: it misses real patterns, so even training predictions are off.

Validation data, which the model never trained on, is how you tell candidates apart and keep the best one. Try several, measure on held-out data, pick the lowest error: that loop is the core habit of building good models.

Exercise