Ensemble methods

A random forest is an ensemble: it combines many models (many trees) into one stronger prediction. Gradient boosting is another ensemble method, and for table data it is often the most accurate of all.

How gradient boosting works

Where a random forest builds its trees independently and averages them, gradient boosting builds them one after another, each one fixing the combined mistakes of the trees before it:

1. Start with a single, naive model (its predictions can be rough).
2. Use the current ensemble to predict, and measure how wrong it is (the loss).
3. Fit a new model aimed at reducing that error, and add it to the ensemble.
4. Repeat, each new model correcting what is still wrong.

The "gradient" in the name refers to using gradient descent on the loss to fit each new model. Done well, this gives excellent accuracy.

scikit-learn has a built-in version you can run right here:

Tuning the key settings

Gradient boosting has a few settings that strongly affect accuracy. The two most important:

• n_estimators: how many models (cycles) to add. Too few underfits (inaccurate everywhere); too many overfits (great on training, poor on new data). Typical values run from 100 to 1000.
• learning_rate: how much each new model contributes. A smaller rate means each tree helps a little less, so you can safely use more trees. A small learning_rate with a large n_estimators usually gives the best accuracy, at the cost of longer training.

See the effect for yourself:

What about XGBoost?

You will hear a lot about XGBoost (extreme gradient boosting). It is a famous, very fast library that implements exactly this method, and it wins many competitions. The idea is identical to what you just ran; XGBoost is a highly optimised version of it. In a full Python setup you would write:

from xgboost import XGBRegressor
model = XGBRegressor(n_estimators=500, learning_rate=0.05, random_state=1)
model.fit(train_X, train_y)

Two more XGBoost settings are worth knowing:

• early_stopping_rounds: finds a good n_estimators for you automatically. You set n_estimators high, and training stops once the validation score stops improving (say, after 5 rounds with no gain). You pass validation data through eval_set so it can check.
• n_jobs: runs the work across your processor's cores in parallel. It does not change accuracy, it just speeds up training on large datasets.

The method underneath all of this is exactly the runnable example above.

Exercise