Metrics: accuracy, precision, recall, F1 and ROC

Suppose a bank builds a fraud detector. It checks 1,000 transactions, only 50 of which are fraud, and the model says "not fraud" every time. Its accuracy is 95%. It also catches zero fraud. The number looks good and the model is worthless.

This is why classification needs more than accuracy. In this lesson you learn the metrics that tell the real story, when to use each, and how to pick a threshold.

For a yes/no problem, call the thing you are looking for the positive class (fraud, disease, churn). Every prediction falls into one of four boxes:

scikit-learn prints the matrix with rows as actual classes and columns as predicted classes, in the order 0 then 1, so for a binary problem it reads [[TN, FP], [FN, TP]]. Always check that order before reading it.

An easy way to remember: precision is about the quality of your alarms. Recall is about how few real cases slip past you. F1 is the harmonic mean of the two, which means it is low if either one is low. You cannot get a good F1 by being great at only one.

Here we create a fraud-like dataset where only about 5% of rows are positive, train logistic regression and compute everything. The data is synthetic, made with make_classification.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (accuracy_score, confusion_matrix, f1_score,
                             precision_score, recall_score, roc_auc_score)
from sklearn.model_selection import train_test_split

# 5000 customers, about 5% of them are fraudulent (class 1)
X, y = make_classification(n_samples=5000, n_features=10, n_informative=5,
                           weights=[0.95, 0.05], random_state=42)
print(np.bincount(y))

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
pred = model.predict(X_test)
proba = model.predict_proba(X_test)[:, 1]

print(confusion_matrix(y_test, pred))
print("accuracy :", round(accuracy_score(y_test, pred), 3))
print("precision:", round(precision_score(y_test, pred), 3))
print("recall   :", round(recall_score(y_test, pred), 3))
print("f1       :", round(f1_score(y_test, pred), 3))
print("ROC AUC  :", round(roc_auc_score(y_test, proba), 3))

Prints the confusion matrix and five metrics.

Read the matrix: TN = 1178, FP = 3, FN = 63, TP = 6. Now compute by hand:

- Accuracy = (6 + 1178) / 1250 = 0.947. Looks great. - Precision = 6 / (6 + 3) = 0.667. When the model flags fraud, it is right two times out of three. - Recall = 6 / (6 + 63) = 0.087. It found fewer than 9 of every 100 real frauds.

The 94.7% accuracy is almost entirely the easy, common class. This model is nearly useless for its real job.

print("Always predict 'not fraud':", round(accuracy_score(y_test, np.zeros_like(y_test)), 3))

for threshold in [0.5, 0.3, 0.2, 0.1]:
    p = (proba >= threshold).astype(int)
    print(f"threshold {threshold}: precision {precision_score(y_test, p):.2f}  "
          f"recall {recall_score(y_test, p):.2f}  f1 {f1_score(y_test, p):.2f}")

Uses the variables from the previous cell. Also tries different thresholds.

A model that predicts "not fraud" for everyone has 94.5% accuracy, only 0.2 points below ours. Accuracy simply cannot see the difference.

Most classifiers output a probability. The default rule "positive if probability is at least 0.5" is just a convention, not a law. Lowering the threshold flags more cases: recall rises (fewer misses) and precision usually falls (more false alarms). Raising it does the opposite. The table above shows this: at 0.1 the model catches 65% of fraud but only 22% of its alerts are real.

So the right threshold depends on the cost of each mistake:

The ROC curve shows what happens at every possible threshold. The vertical axis is the true positive rate (recall). The horizontal axis is the false positive rate: FP / (FP + TN), the share of real negatives wrongly flagged. A perfect model hugs the top-left corner. Random guessing is the diagonal line.

ROC AUC (area under the curve) squeezes it into one number between 0 and 1. A useful way to read it: pick one random positive and one random negative; the AUC is the chance the model gives the positive a higher score. 0.5 means no skill, 1.0 means perfect ranking. Its big advantage is that it does not depend on any single threshold.

from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y_test, proba)
# Pick the first threshold that catches at least 60% of fraud, and see its false positive rate
i = (tpr >= 0.60).argmax()
print(f"threshold {thresholds[i]:.3f}: recall (TPR) {tpr[i]:.2f}, false positive rate {fpr[i]:.2f}")

roc_curve returns the false positive rate, true positive rate and the threshold at each point.

To catch 61% of fraud, we must accept flagging 11% of the genuine transactions. Whether that is acceptable is a business question the curve lets you ask clearly.

Beyond choosing a good metric and threshold, you can tell the model to care more about the rare class. Most scikit-learn classifiers accept class_weight="balanced", which weights errors on the rare class more heavily.

balanced = LogisticRegression(max_iter=1000, class_weight="balanced").fit(X_train, y_train)
p = balanced.predict(X_test)
print("balanced weights: precision", round(precision_score(y_test, p), 2),
      "recall", round(recall_score(y_test, p), 2))

Uses the same split as before.

Recall jumped from 0.09 to 0.84, but precision fell to 0.16. This is the same trade-off as moving the threshold, not a free improvement. Other options are collecting more positive examples or oversampling the rare class; always apply those to the training data only, never the test set.

Expect: "Explain precision and recall with an example", "When is accuracy misleading?", "What is the ROC curve?", "How would you tune a model for fraud detection?" A strong answer names the cost of each type of mistake, mentions the threshold, and says what you would do with imbalance. Be careful to say which class is positive; many people mix up precision and recall under pressure.

Act as an interviewer for a junior data scientist role. Ask me one question at a time about classification metrics (confusion matrix, precision, recall, F1, ROC AUC, thresholds, class imbalance). After each of my answers, tell me what was correct, what was missing, and then ask a harder follow-up. Start with an easy question.

What to expect: A good session gives you feedback after each answer and moves from definitions to trade-offs and real scenarios.