A/B testing and experiments

Brew & Bean wants more people who open the pickup-ordering page to actually place an order. The designer suggests changing the button from "Place order" to "Order now, pick up in 10 minutes". Will it help?

You could change it and compare this month with last month, but many things differ between months: weather, exams, festivals, offers. An A/B test removes that problem by running both versions at the same time and choosing randomly who sees which.

Because assignment is random, the two groups are alike in everything except the change. So any lasting difference in the metric can be credited to the change. This is what lets an experiment prove cause, which a simple before-and-after comparison cannot.

Suppose 1,000 people see A and 100 order (10%). Another 1,000 see B and 125 order (12.5%). B looks better. But if you showed the same page to two random groups of 1,000, the conversion rates would differ a little by luck alone. The question is: is this gap bigger than luck would usually produce?

The p-value answers a narrow question: if there were truly no difference between A and B, how often would we see a gap this large or larger just by chance? A small p-value (the common cut-off is 0.05) means the gap would be unusual under pure chance, so we call the result statistically significant.

The p-value is not the probability that B is better, and a result with p below 0.05 does not mean the effect is large or important. Always look at the size of the difference too. A confidence interval gives a range of plausible values for the real difference and is more informative than the p-value alone.

from math import sqrt, erf


def normal_cdf(z):
    return 0.5 * (1 + erf(z / sqrt(2)))


def two_proportion_test(conv_a, n_a, conv_b, n_b):
    """Two-sided z-test for the difference between two conversion rates."""
    p_a = conv_a / n_a
    p_b = conv_b / n_b
    pooled = (conv_a + conv_b) / (n_a + n_b)
    se = sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
    z = (p_b - p_a) / se
    p_value = 2 * (1 - normal_cdf(abs(z)))
    return p_a, p_b, z, p_value


def sample_size_per_group(baseline, min_lift):
    """Rule of thumb for 5% significance and 80% power."""
    return round(16 * baseline * (1 - baseline) / min_lift ** 2)


# 1,000 visitors each: 100 orders vs 125 orders
p_a, p_b, z, p = two_proportion_test(100, 1000, 125, 1000)
print(f"A: {p_a:.1%}  B: {p_b:.1%}  z = {z:.2f}  p-value = {p:.3f}")

# Same rates with 4,000 visitors each
p_a, p_b, z, p = two_proportion_test(400, 4000, 500, 4000)
print(f"A: {p_a:.1%}  B: {p_b:.1%}  z = {z:.2f}  p-value = {p:.4f}")

# Visitors needed per group to detect a 2 percentage point lift from 10%
print(sample_size_per_group(0.10, 0.02))

Uses only the Python standard library. Save as ab_test.py and run with python ab_test.py.

Look at the first two lines. The rates are identical (10% and 12.5%), but with 1,000 visitors per group the result is not significant (p = 0.077) and with 4,000 per group it is clearly significant. More data makes real differences easier to separate from luck. With too little data, you cannot tell.

The last line says: to detect a lift from 10% to 12% with decent reliability you need about 3,600 visitors per group. That formula is a rough rule of thumb; real tools use more exact calculations, but it shows the point. Small improvements need a lot of traffic.

Power is the chance your test will detect a real effect of the size you care about. A common target is 80%. A test with low power often ends with "not significant", which wrongly feels like "no effect". It really means "we couldn't tell". That is why you calculate the sample size before the test and not after.

A test can also tell you B is no better than A, and that is a useful answer: you avoided shipping a change that did nothing. Small businesses with little traffic often cannot run a precise A/B test; they can still run simple trials, such as one outlet or one week with a change, but should be honest that those are weaker evidence.

Product and data analyst interviews often ask: "How would you test this change?", "What metric would you choose and why?", "The test shows p = 0.04 but the lift is tiny; do you ship?", or "How long would you run it?". Strong answers state the hypothesis, one primary metric plus guardrails, randomisation unit, sample size and duration, and a decision rule. They also mention peeking and multiple metrics as risks.

I want to run an A/B test for [business or product]. The change is [describe the change]. Our current baseline for [metric] is about [number] and we get about [number] visitors per week.

Help me plan: (1) a hypothesis, (2) the primary metric and two guardrail metrics, (3) the unit of randomisation, (4) roughly how many visitors per group I need and how many weeks that takes, showing the formula, (5) a decision rule, (6) risks that could make the result misleading.

What to expect: Ask the AI to show its sample size calculation and check it against the rule of thumb in this lesson. If the weeks needed are too long, reduce ambition: test a bigger change or accept a weaker trial.