Train, validation, test and overfitting

Imagine a student who memorises last year's question paper word for word. They score 100% on that paper and fail the real exam, because the questions are different. Machine learning models can do the same thing. A model that only memorises its training data is useless on new data, and the whole point of ML is to work on data it has not seen.

This lesson explains how to measure honestly whether a model will work on new data, and what to do when it will not.

You split your data into parts, each with one job:

Why not just two sets? Every time you look at a score and then change something (a different model, a different setting), you slightly tune your choices to that data. After many rounds, the score on it is too optimistic. The test set stays untouched so that its score is an honest estimate. Use it once, after all decisions are made.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

# A noisy dataset: 15% of the labels are deliberately wrong, like real-life data.
X, y = make_classification(
    n_samples=400, n_features=10, n_informative=3, n_redundant=0,
    flip_y=0.15, class_sep=1.0, random_state=1,
)
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, random_state=1)
print(len(X_train), len(X_val), len(X_test))   # 240 80 80

for depth in [1, 2, 3, 5, 8, 12, None]:
    model = DecisionTreeClassifier(max_depth=depth, random_state=0)
    model.fit(X_train, y_train)
    print(depth, "train:", round(model.score(X_train, y_train), 3),
          "validation:", round(model.score(X_val, y_val), 3))

Splits 400 rows into 240 train, 80 validation and 80 test. Then trains decision trees of increasing depth and prints train and validation accuracy.

A decision tree with no depth limit memorises the training set: 100% accuracy there, but only about 82% on validation. The deeper the tree, the higher the training score climbs, while the validation score stops improving. That growing gap is overfitting. (Your numbers may differ a little with other library versions.)

Think of fitting a curve through dots on a graph. A straight line through dots that clearly curve underfits. A wiggly line that touches every dot, including the odd ones caused by measurement noise, overfits. The best line follows the trend and ignores the noise.

The tool is simple: always compare the training score with the validation score. A big gap means overfitting. Two low scores mean underfitting.

With a small dataset, one validation split is a gamble: a lucky or unlucky split changes the score a lot (with 80 rows, one extra mistake moves accuracy by more than one percentage point). K-fold cross-validation solves this. Split the training data into k parts (folds). Train k times, each time holding out a different fold for validation, then average the k scores.

In 5-fold cross-validation each row is used for validation exactly once and for training four times.

from sklearn.model_selection import cross_val_score

for depth in [2, 3, 5, 8, None]:
    model = DecisionTreeClassifier(max_depth=depth, random_state=0)
    scores = cross_val_score(model, X_temp, y_temp, cv=5)
    print(depth, scores.round(2), "mean:", round(scores.mean(), 3))

final = DecisionTreeClassifier(max_depth=3, random_state=0)
final.fit(X_temp, y_temp)                        # refit on train + validation
print("Test accuracy:", round(final.score(X_test, y_test), 3))   # look at this once

Run this after the previous code cell: it reuses X_temp, y_temp (train plus validation, 320 rows), X_test, y_test and DecisionTreeClassifier.

Cross-validation shows the trend clearly: shallow trees score about 0.82 and the unlimited tree scores about 0.76. The five scores per setting also show the spread. A model that scores 0.82 plus or minus 0.04 is not clearly better than one at 0.81.

The workflow: use cross-validation on the training data to choose a model and settings, then refit on all training data, then score once on the test set. The test score (0.825 here) is the number you report.

Data leakage means information from outside the training data sneaks into training, so your validation score looks great but the model fails in real life. It is the most common reason a project works in the notebook and fails in production.

Typical causes:

- Fitting a scaler, imputer or feature selector on all the data before splitting. - A feature that contains the answer or is only known after the event (for example "cancellation_date" when predicting churn, or "loan_defaulted_flag" when predicting default). - The same customer, patient or image appearing in both train and test sets. - Shuffling time-ordered data so the model sees the future.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(0)
X = rng.normal(size=(100, 5000))   # 5000 columns of pure noise
y = rng.integers(0, 2, size=100)   # random labels: there is nothing to learn

# Wrong: choose the 20 "best" columns using ALL rows, then cross-validate.
X_leaky = SelectKBest(f_classif, k=20).fit_transform(X, y)
print("Leaky  :", round(cross_val_score(LogisticRegression(max_iter=1000), X_leaky, y, cv=5).mean(), 2))

# Right: the selection happens inside the pipeline, so it only sees training folds.
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=1000))
print("Correct:", round(cross_val_score(pipe, X, y, cv=5).mean(), 2))

The labels here are random, so no model can honestly do better than 50% accuracy.

Both versions have random labels, so the honest answer is about 50%. The leaky version picked the 20 columns that happened to match the labels using all rows, so cross-validation reported 85%. The correct version does the selection inside each training fold and reports about 48%, which is the truth. The fix is always the same: put every step that learns from data inside a Pipeline, and cross-validate the pipeline.

Very common questions: "What is overfitting and how do you detect it?" (large gap between training and validation scores), "Why do we need a test set as well as a validation set?", "Explain k-fold cross-validation", "What is data leakage? Give an example" and "Your model has 99% accuracy; what do you check first?". Answer with a concrete example like the ones above.

I am building a model to predict [target] from [describe your table and columns]. Here is my code:

[paste your code]

Review it for data leakage and for mistakes in how I split and evaluate. Check specifically: preprocessing fitted before splitting, features that would not be known at prediction time, duplicate or related rows across the split, and time order. List each problem and the fix.

What to expect: A good answer points to exact lines, explains why each is leakage, and suggests moving preprocessing into a Pipeline.