Decision trees, random forests and gradient boosting

For tables of data (rows and columns, like a spreadsheet or database), the most successful classic models are built from decision trees. Random forests and gradient boosting are the usual winners in Kaggle competitions on tabular data and the first things many companies try on customer, sales and risk data. This lesson explains how they work in plain terms and how to use them in scikit-learn.

A decision tree is a flowchart of yes/no questions about the features. To predict, you start at the top and follow the answers down to a leaf, which gives the prediction. Training means finding the best questions.

At each step the algorithm tries many possible questions ("is worst radius at most 16.8?") and picks the one that separates the classes best, using a measure of impurity such as Gini. A group is pure if it contains one class only. It then repeats on each half until it stops.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42, stratify=data.target
)

tree = DecisionTreeClassifier(max_depth=2, random_state=42)
tree.fit(X_train, y_train)
print(export_text(tree, feature_names=list(data.feature_names)))
print("Test accuracy:", round(tree.score(X_test, y_test), 3))

export_text prints the learned rules. Output shown is from scikit-learn on the breast cancer dataset.

You can read this like a rulebook: a tumour with a small radius and a low concave-points value is predicted benign (class 1). This explainability is the main strength of a single tree. Other strengths: no feature scaling needed, handles numbers and (after encoding) categories, and captures curved and interacting patterns that a straight-line model cannot.

The big weakness: left alone, a tree keeps asking questions until every training row is in its own pure leaf. That memorises the data (overfitting, see *Train, validation, test and overfitting*). Small changes in the data can also produce a very different tree, which is why single trees are unstable.

One tree is unstable, but the average of many different trees is stable and accurate. A random forest builds hundreds of trees and combines them (majority vote for classification, average for regression). To make the trees different from each other, each one:

- is trained on a bootstrap sample: rows drawn at random with replacement, so each tree sees a slightly different version of the data; - considers only a random subset of features at each split.

Think of asking 200 people who each saw part of the evidence, instead of one person who saw everything. Their individual mistakes are different, so they cancel out. This is called bagging, and it mainly reduces variance (overfitting).

Boosting builds trees one after another. The first tree makes a rough prediction. The second tree is trained to correct the errors of the first. The third corrects what is still wrong, and so on. Each tree is small and weak on its own (often just a few levels deep), and the learning rate controls how much each new tree adds. Many small corrections add up to a strong model.

In scikit-learn, HistGradientBoostingClassifier is the fast, modern choice and it accepts missing values directly. Popular libraries with the same idea are XGBoost, LightGBM and CatBoost, which you can install with pip.

from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.model_selection import cross_val_score

models = {
    "Single tree, no limit": DecisionTreeClassifier(random_state=42),
    "Single tree, depth 3": DecisionTreeClassifier(max_depth=3, random_state=42),
    "Random forest": RandomForestClassifier(n_estimators=200, random_state=42),
    "Gradient boosting": HistGradientBoostingClassifier(random_state=42),
}

for name, model in models.items():
    scores = cross_val_score(model, X_train, y_train, cv=5)
    print(f"{name:24s} {scores.mean():.3f} (+/- {scores.std():.3f})")

Run it after the first cell. Uses X_train and y_train from there.

The pattern is typical: a limited tree beats an unlimited tree, a forest beats both, and boosting is slightly ahead. With differences this small (the +/- shows the spread across folds), do not claim one is certainly better on a dataset of 455 rows. On this small, clean dataset even logistic regression does well. The gap between linear models and tree ensembles is usually larger on messy business data with interactions and mixed columns.

A forest can tell you which features it used most. feature_importances_ measures how much each feature reduced impurity across all trees.

import pandas as pd

forest = RandomForestClassifier(n_estimators=200, random_state=42)
forest.fit(X_train, y_train)

importance = pd.Series(forest.feature_importances_, index=data.feature_names)
print(importance.sort_values(ascending=False).head(5).round(3))
print("Test accuracy:", round(forest.score(X_test, y_test), 3))

Prints the five most important features and the test accuracy.

Boosting can keep adding trees and start to overfit. Early stopping holds back part of the training data, watches the score on it, and stops when it has not improved for several rounds.

gb = HistGradientBoostingClassifier(
    learning_rate=0.1,        # how much each new tree contributes
    max_iter=300,             # maximum number of trees
    early_stopping=True,      # stop when a held-out part of the training data stops improving
    validation_fraction=0.15,
    n_iter_no_change=10,
    random_state=42,
)
gb.fit(X_train, y_train)
print("Trees actually used:", gb.n_iter_)
print("Test accuracy:", round(gb.score(X_test, y_test), 3))

n_iter_ shows how many trees were actually built. Results can vary slightly between versions.

Common questions: "How does a decision tree decide where to split?" (the split that most reduces impurity), "Why do trees overfit and how do you stop it?", "Bagging versus boosting", "Why is a random forest better than one tree?" (averaging many different trees reduces variance) and "Do trees need feature scaling?" (no). If asked about XGBoost, say it is an optimised implementation of gradient boosting.

Here is the output of export_text for my decision tree, which predicts [target]:

[paste the export_text output]

Explain in plain English what rules the tree learned, which features it relies on most, and whether any rule looks suspicious or like it could be caused by leakage or noise. I am a beginner.

What to expect: A good answer rewrites the branches as sentences and flags splits that seem to use information not available at prediction time.