Classification: predicting categories

What you will build: a model that looks at 30 measurements of a tumour sample and predicts whether it is malignant or benign. You will use scikit-learn's built-in breast cancer dataset, so there is nothing to download. Along the way you learn the steps every classification project follows.

Prerequisites: you can run Python in Google Colab or Kaggle (see *Free notebooks: Google Colab and Kaggle*), you know what a pandas DataFrame is, and you have read *How machine learning learns* and *Regression: predicting numbers*. Colab already has scikit-learn installed.

Classification is supervised learning where the answer is a category, not a number. Is this email spam or not spam? Will this customer churn, yes or no? Which of ten digits is in this image? You give the model many examples with the right answer attached (the label), and it learns a rule that works on new examples.

When there are two categories it is binary classification. With three or more it is multiclass classification. The code is almost the same for both.

Regression and classification use the same workflow. What changes is the target and how you judge the result: a classifier is scored on how often it is right, and on which kinds of mistakes it makes (see *Metrics: accuracy, precision, recall, F1 and ROC*).

scikit-learn's load_breast_cancer gives us 569 samples. Each has 30 numeric measurements (radius, texture, smoothness and so on) and a label: 0 for malignant, 1 for benign. Note the order: 0 is malignant. We will keep that in mind when reading results.

import pandas as pd
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer(as_frame=True)
X = data.data      # 30 numeric measurements per tumour
y = data.target    # 0 = malignant, 1 = benign

print(X.shape)
print(data.target_names)
print(y.value_counts())
print(X.iloc[:3, :4])

Prints the shape, the class names, how many samples are in each class and the first rows.

Always check the class balance first. Here 357 of 569 samples (about 63%) are benign. The classes are not equal, which matters for the next step.

We hold back 20% of the data as a test set that the model never sees during training. stratify=y keeps the share of each class the same in both parts, which is important when one class is rarer. random_state=42 makes the split repeatable. The full reasoning is in *Train, validation, test and overfitting*.

Then we build a baseline: a dummy model that always predicts the most common class. Any real model must beat it, otherwise it has learned nothing.

from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
print("Baseline accuracy:", round(accuracy_score(y_test, baseline.predict(X_test)), 3))

The dummy model gets roughly 63% by always saying "benign".

Logistic regression is the standard first classifier. Despite the name it is used for classification. It computes a weighted sum of the features, like linear regression, and then squeezes the result into a probability between 0 and 1 using an S-shaped curve. If the probability is above 0.5 it predicts class 1, otherwise class 0.

Logistic regression works best when features are on a similar scale, so we put a StandardScaler (subtract the mean, divide by the standard deviation) in a pipeline with the model. A pipeline runs the steps in order and, importantly, learns the scaling from the training data only.

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import confusion_matrix, classification_report

logreg = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
logreg.fit(X_train, y_train)

pred = logreg.predict(X_test)
print("Accuracy:", round(accuracy_score(y_test, pred), 3))
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, target_names=data.target_names))

fit() learns from the training data; predict() gives labels for the test data.

Accuracy is 98.2%, far above the 63.2% baseline. The confusion matrix shows where the errors are. Rows are the true classes and columns are the predicted ones, in the order 0 then 1:

- 41 malignant tumours were correctly called malignant. - 1 malignant tumour was wrongly called benign (the dangerous mistake). - 1 benign tumour was wrongly called malignant. - 71 benign tumours were correctly called benign.

Exact numbers can differ slightly with other library versions.

Most classifiers can say how confident they are. predict_proba returns one column per class, in the order of model.classes_ (here 0 then 1). predict simply picks the class with the higher probability.

Probabilities are useful because you can choose your own cut-off instead of 0.5. If missing a malignant tumour is much worse than a false alarm, you would flag anything above, say, 0.2 probability of malignant. This is explained in the metrics lesson.

K-nearest neighbours (KNN) predicts by looking at the k most similar training examples and taking a vote. It needs no training in the usual sense: it just remembers the data. Because it measures distances, features with big numbers (like area, in the thousands) would drown out features with small numbers (like smoothness, below 1). Scaling fixes that. Run both versions to see the effect.

from sklearn.neighbors import KNeighborsClassifier

knn_raw = KNeighborsClassifier(n_neighbors=5)
knn_raw.fit(X_train, y_train)
print("KNN without scaling:", round(knn_raw.score(X_test, y_test), 3))

knn_scaled = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
knn_scaled.fit(X_train, y_train)
print("KNN with scaling:   ", round(knn_scaled.score(X_test, y_test), 3))

Uses X_train, y_train, X_test, y_test, make_pipeline and StandardScaler from the earlier cells, so run this in the same notebook.

I trained a binary classifier in scikit-learn. Here is my confusion matrix (rows = true class, columns = predicted class, order 0 then 1):

[confusion matrix]

Class 1 means [what class 1 means, e.g. customer churned]. Explain in plain English what each of the four numbers means, which mistake is worse for [your use case], and what I could try to reduce it. Assume I am a beginner.

What to expect: A good answer names true/false positives and negatives for your real use case, and suggests options such as changing the threshold, class weights or a different metric.

Interviewers often ask: "Why is accuracy not enough?", "What does logistic regression do?" (it outputs a probability using a sigmoid; the decision threshold is a choice), "Why scale features?" and "What is a baseline?". Be ready to explain the confusion matrix in words and to say which mistake costs more in a given problem.