From one tree to a forest

A single decision tree leaves you stuck between two bad options: too deep and it overfits, too shallow and it underfits. Even advanced models live with this tension, but many use clever ideas to handle it better. The random forest is one of the most useful.

A random forest builds many decision trees, each slightly different, and makes its prediction by averaging the predictions of all of them. One tree might err in one direction and another in a different direction, so averaging cancels out much of the error. The result is usually more accurate than any single tree, and much less likely to overfit.

Building one

You build a random forest exactly like a decision tree, you just swap in the RandomForestRegressor class. Here we train both on the same data and compare their validation errors:

Python

import pandas as pd
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error

students = pd.DataFrame({
    "study_hours":    [2, 5, 1, 6, 3, 8, 4, 7, 1, 6, 5, 3],
    "attendance_pct": [60, 85, 50, 95, 70, 98, 75, 90, 45, 88, 80, 65],
    "prev_score":     [50, 72, 40, 85, 58, 92, 65, 88, 35, 80, 75, 55],
    "exam_score":     [55, 78, 42, 90, 62, 95, 68, 89, 38, 84, 77, 60],
})
X = students[["study_hours", "attendance_pct", "prev_score"]]
y = students["exam_score"]
train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)

tree = DecisionTreeRegressor(random_state=1)
tree.fit(train_X, train_y)
print("Single tree MAE: ", round(mean_absolute_error(val_y, tree.predict(val_X)), 1))

forest = RandomForestRegressor(random_state=1)
forest.fit(train_X, train_y)
print("Random forest MAE:", round(mean_absolute_error(val_y, forest.predict(val_X)), 1))

Compare the two errors. With almost no extra effort, the forest usually does better.

Why it is a great default

The best part: a random forest works well straight out of the box. A single tree needs you to tune max_leaf_nodes carefully to find its sweet spot. A random forest gives strong results with its default settings, so it is an excellent model to reach for first, before you spend any time tuning. You can tune it later for a little more, but you rarely have to. That mix of high accuracy and low fuss is why it is so widely used.

Exercise

Exercise — check your understanding

Define a RandomForestRegressor called forest with random_state=1. The line that fits it on train_X and train_y is already written for you.

You’re reading for free. Sign in to keep your progress and earn a certificate when you finish.Sign in to keep my progress →

← PreviousUnderfitting and overfitting Next →Where to go next

Random forests

From one tree to a forest

Building one

Why it is a great default

Exercise