Statistics you actually need

You do not need a statistics degree to do data science, but you do need a small set of ideas used correctly. This lesson covers the ones that come up every week: summarising data, spread, the normal curve, samples and uncertainty, hypothesis tests and p-values, and the traps that make good-looking results wrong.

School-level maths is enough. Code uses NumPy and SciPy, which are preinstalled in Colab and Kaggle.

The mean is the average. The median is the middle value when sorted. The standard deviation (std) measures the typical distance from the mean: small std means values are tightly packed, large means spread out. Use the median when a few extreme values would drag the mean.

import numpy as np

# Bill amounts (Rs) at a table-service cafe; the last one is a bulk catering order
bills = np.array([180, 220, 150, 190, 210, 160, 200, 230, 170, 195, 4000])

print("mean  :", round(np.mean(bills), 1))
print("median:", np.median(bills))
print("std   :", round(np.std(bills, ddof=1), 1))

typical = bills[:-1]   # without the catering order
print("mean without it  :", np.mean(typical))
print("median without it:", np.median(typical))

The mean with the catering order is about 537, far above anything a normal table pays, while the median is 195. Without the order, mean 190.5 and median 192.5 agree. ddof=1 gives the sample standard deviation, which is the one to use when your data is a sample.

A mean of 537 would give a very wrong idea of a typical bill. This is why the first thing to check, always, is whether the mean and median tell the same story.

A distribution shows how often each value occurs. Many things (heights, measurement errors, exam marks in a large class) form a bell shape called the normal distribution. For a normal distribution, about 68% of values lie within 1 std of the mean and about 95% within 2 std.

rng = np.random.default_rng(7)
x = rng.normal(loc=100, scale=15, size=100_000)   # mean 100, std 15

print(((x > 85) & (x < 115)).mean())    # within 1 std: about 0.68
print(((x > 70) & (x < 130)).mean())    # within 2 std: about 0.95

The two printed numbers are close to 0.68 and 0.95. Not everything is normal: incomes, waiting times and sales of bestsellers are skewed with a long tail. Look at a histogram before assuming.

You almost never see all the data. You see a sample and want to say something about the population. If you survey 12 days of sales, the average of those 12 days is a guess for the true average. A different 12 days would give a different guess.

A confidence interval states the uncertainty: a range that, if you repeated the study many times, would contain the true value about 95% of the time. It does not mean there is a 95% chance this one interval holds the truth, a common misunderstanding. More data makes the interval narrower.

from scipy import stats

daily_cups = np.array([120, 135, 128, 142, 110, 150, 138, 125, 131, 145, 122, 139])
mean = daily_cups.mean()
se = stats.sem(daily_cups)          # standard error of the mean
low, high = stats.t.interval(0.95, df=len(daily_cups) - 1, loc=mean, scale=se)
print(round(mean, 1), "95% interval:", round(low, 1), "to", round(high, 1))

The mean is about 132.1 and the interval about 124.7 to 139.5. The honest report is "around 132 cups a day, probably between 125 and 139", not just "132".

The standard error is the std of your estimate of the mean. It shrinks as the sample grows (roughly with the square root of the sample size). That is why 12 days gives a wide range and 365 days a narrow one.

Suppose Brew & Bean tries a new menu board for 30 days and compares the daily cups to 30 days with the old board. The new average is higher, but days vary a lot. Is the board working, or is this luck?

A hypothesis test answers: if the board made no difference (the null hypothesis), how surprising would a gap this big be? The p-value is that probability. A small p-value (the usual cutoff is 0.05) says the data would be unusual under no effect, so we reject the null.

rng = np.random.default_rng(1)
old_board = rng.normal(100, 20, 30).round()   # 30 days, true average 100
new_board = rng.normal(112, 20, 30).round()   # 30 days, true average 112

print(old_board.mean(), new_board.mean())
t_stat, p_value = stats.ttest_ind(new_board, old_board, equal_var=False)
print("t =", round(t_stat, 2), " p =", round(p_value, 4))

Here the two means are about 98.7 and 111.4, and p is about 0.007. That is below 0.05, so a gap this large would be unusual by luck alone. (The data is simulated, so we know the board really helps.)

# How often do 20 tests of PURE NOISE give a "significant" result?
rng = np.random.default_rng(3)
false_alarms = 0
for _ in range(20):
    group_a = rng.normal(0, 1, 30)
    group_b = rng.normal(0, 1, 30)     # same distribution: there is no real difference
    if stats.ttest_ind(group_a, group_b).pvalue < 0.05:
        false_alarms += 1
print("significant results from noise:", false_alarms, "out of 20")

Both groups come from the same distribution, yet on average about 1 in 20 tests will still look significant at 0.05. With seed 3 this run prints 1. Run enough tests and you will always find something.

If you do not trust a formula, resample your own data. Draw many new samples of the same size from your data (with replacement), compute the mean each time, and look at the middle 95% of those means.

rng = np.random.default_rng(1)
boot_means = [rng.choice(daily_cups, size=len(daily_cups), replace=True).mean()
              for _ in range(5000)]
print(np.percentile(boot_means, [2.5, 97.5]).round(1))

The result is close to the t-interval above (about 125.8 to 138.2 in a test run; yours will differ slightly). It works for medians and other statistics where no easy formula exists.

Typical questions: "Explain a p-value to a non-technical person", "Mean vs median: when do you use which?", "What is the difference between correlation and causation?", "How would you test whether a new feature improved sales?". For the last one, describe randomly splitting users or days into two groups, choosing the metric and sample size beforehand, then comparing with a test and an interval. Answer in plain words with an example, as this lesson does.

Explain [statistical concept, e.g. p-value] to me. I am a [your background, e.g. final-year BCA student] who knows basic Python but not much statistics.

Use one everyday example from India (a cricket match, a shop, an exam). Then show a 10-line Python example using NumPy or SciPy with comments. Finish with the two most common mistakes people make with it and a question to test my understanding.

What to expect: Check that the example uses the same words the lesson does (null hypothesis, sample, population). If the explanation says "probability the hypothesis is true", it is wrong for a p-value.