Power and Sample Size

22.1 Errors, power and the minimum detectable effect (40 min)

Concept. A test can be wrong in two ways:

Power is the probability of detecting an effect of a given size, if it's really there. The convention is 80%: one real effect in five goes unnoticed. The minimum detectable effect (MDE) is the smallest true effect a test detects with that power. Power depends on four things:

• effect size δ: bigger effects are easier to see;
• noise σ (the standard deviation of the metric): noisier metrics need more data;
• sample size n: precision grows with √n, so halving the MDE needs 4× the customers;
• α: a stricter threshold costs power.

22.2 The sample-size formula (50 min)

For two equal groups and a difference in means, the number of customers per group is:

n = 2 × (z₁₋α/₂ + z_power)² × σ² / δ²
  = 2 × (1.96 + 0.84)² × σ² / δ²      for α = 0.05, power = 80%

Create day22/power_formula.py:

# day22/power_formula.py   (python -m day22.power_formula)
"""How many customers do we need? The textbook formula, then statsmodels."""
import numpy as np
from scipy.stats import norm
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

ALPHA, POWER = 0.05, 0.80
z = norm.ppf(1 - ALPHA / 2) + norm.ppf(POWER)          # 1.96 + 0.84 = 2.80

# Baseline (Day 21, last month): means and standard deviations per customer over 4 weeks
baseline = {"revenue": (70.22, 99.03), "orders": (1.147, 1.351)}
lift = 0.05                                             # we want to detect a +5% change

print(f"z_(1-alpha/2) + z_(power) = {z:.3f}\n")
for metric, (mean, sd) in baseline.items():
    delta = lift * mean
    n_formula = 2 * z**2 * sd**2 / delta**2
    n_sm = NormalIndPower().solve_power(effect_size=delta / sd, alpha=ALPHA, power=POWER)
    print(f"{metric:<9} detect {delta:+.3f}: n per group = {n_formula:,.0f} (formula)  {n_sm:,.0f} (statsmodels)")

# A proportion: conversion from 60.6% to 60.6% x 1.025 (+2.5% relative)
p0 = 0.606
p1 = p0 * 1.025
h = proportion_effectsize(p1, p0)                       # Cohen's h for proportions
n_conv = NormalIndPower().solve_power(effect_size=h, alpha=ALPHA, power=POWER)
print(f"converted detect {p0:.3f} -> {p1:.3f}: n per group = {n_conv:,.0f}")

python -m day22.power_formula

z_(1-alpha/2) + z_(power) = 2.802

revenue   detect +3.511: n per group = 12,488 (formula)  12,488 (statsmodels)
orders    detect +0.057: n per group = 8,711 (formula)  8,711 (statsmodels)
converted detect 0.606 -> 0.621: n per group = 16,214

Detecting a 5% lift in revenue needs about 25,000 customers; in orders, 17,500. Orders need fewer because their spread relative to the mean is smaller (1.35 / 1.15 vs 99 / 70). For proportions, statsmodels uses Cohen's h, a variance-stabilising scale; detecting a 2.5% relative change in conversion needs 32,000 customers.

The formula assumes a normal approximation, which holds well for means of thousands of customers even when individual revenue is very skewed.

22.3 Power by simulation (60 min)

Concept. Formulas need assumptions. Simulation needs none: run the experiment many times at a given sample size, with the true effect built in, and count how often the test detects it. With a lab, that's easy.

Create day22/power_by_simulation.py:

# day22/power_by_simulation.py   (python -m day22.power_by_simulation)
"""
Power by simulation: run the experiment many times at each sample size and count how often
the test detects the (true, known) effect. No formula assumptions needed.
"""
import numpy as np
import polars as pl
from scipy.stats import ttest_ind

from causalkit.lab import population, run

pop = population()
rng = np.random.default_rng(22)
REPS = 200

rows = []
for n_per_group in [2_500, 5_000, 10_000, 20_000]:
    hits = {"revenue": 0, "orders": 0}
    for r in range(REPS):
        idx = rng.choice(pop.height, size=2 * n_per_group, replace=False)
        sub = pop[np.sort(idx)]
        z = rng.permutation(np.repeat([0, 1], n_per_group))
        out = run(sub, z, "email_v2", seed=r)
        for m in hits:
            y = out[m].to_numpy()
            hits[m] += ttest_ind(y[z == 1], y[z == 0], equal_var=False).pvalue < 0.05
    rows.append({"n_per_group": n_per_group,
                 "power_revenue": hits["revenue"] / REPS, "power_orders": hits["orders"] / REPS})

print(pl.DataFrame(rows))
print("Formula said: 80% power needs 12,488 (revenue) and 8,711 (orders) per group.")

python -m day22.power_by_simulation

│ n_per_group ┆ power_revenue ┆ power_orders │
│ 2500        ┆ 0.24          ┆ 0.31         │
│ 5000        ┆ 0.435         ┆ 0.595        │
│ 10000       ┆ 0.695         ┆ 0.82         │
│ 20000       ┆ 0.905         ┆ 0.975        │
Formula said: 80% power needs 12,488 (revenue) and 8,711 (orders) per group.

The simulation agrees with the formula: orders reach 80% power near 9,000 per group, revenue somewhere between 10,000 and 20,000. Simulation shines when the formula doesn't apply: clustered designs (Day 28), ratio metrics, sequential stopping (Day 27), or any analysis too complex for a closed form.

22.4 Underpowered tests and what they cost (35 min)

At 2,500 customers per group, email_v2 has 24% power on revenue: three times out of four, a real 5% lift goes unnoticed. That's the obvious cost. The hidden one is worse: when an underpowered test does come out significant, it overstates the effect, because only the lucky draws cross the threshold. This is called a Type M (magnitude) error, and it's why small tests that "found a big effect" so often fail to replicate. You'll see it directly on Day 26 (the winner's curse).

Practical rules:

• Decide the MDE from the business: the smallest effect that would change the decision (it must at least pay for the change). Then compute n.
• If n is larger than you have, don't run a weak test and hope. Change the design: a less noisy metric, CUPED (Day 25), a longer window, or a bigger effect worth testing.
• Never compute "observed power" after a test. It's a restatement of the p-value and tells you nothing new.

22.5 Day 22 mini-project: MDE curves and the guardrail problem (70 min)

Goal. Compute every metric's MDE at the sample we can afford, plot MDE curves, and check whether the free-shipping margin guardrail (Day 30) is testable at all.

Create day22/mde_plan.py:

# day22/mde_plan.py   (python -m day22.mde_plan)
"""
Mini-project: the minimum detectable effect (MDE) of every metric at every affordable sample size,
and what that means for the planned tests.
"""
import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
import polars as pl
from scipy.stats import norm

from causalkit.lab import population, run

pop = population()
base = run(pop, np.zeros(pop.height), "aa_test", seed=99)        # last month
Z = norm.ppf(0.975) + norm.ppf(0.80)

metrics = ["revenue", "orders", "converted", "margin"]
stats = {m: (base[m].mean(), base[m].std()) for m in metrics}


def mde(sd: float, n_per_group: float) -> float:
    return Z * sd * np.sqrt(2 / n_per_group)


rows = []
for m, (mean, sd) in stats.items():
    rows.append({"metric": m, "baseline": round(mean, 3), "sd": round(sd, 3),
                 "MDE at 20k/group": round(mde(sd, 20_000), 3),
                 "MDE relative": f"{mde(sd, 20_000) / mean:.1%}"})
print(pl.DataFrame(rows))

# The free-shipping margin guardrail: could we even detect a loss of $0.50 per customer?
sd_margin = stats["margin"][1]
n_needed = 2 * Z**2 * sd_margin**2 / 0.50**2
print(f"\nDetecting a $0.50 margin loss needs {n_needed:,.0f} customers per group "
      f"({2 * n_needed:,.0f} total). We have {pop.height:,}.")

n = np.linspace(1_000, 20_000, 60)
fig, ax = plt.subplots(figsize=(7, 4))
for m, (mean, sd) in stats.items():
    ax.plot(n, 100 * mde(sd, n) / mean, label=m)
ax.set_xlabel("Customers per group"); ax.set_ylabel("Minimum detectable effect (% of baseline)")
ax.set_ylim(0, 25); ax.legend(); ax.set_title("MDE shrinks with the square root of sample size")
fig.tight_layout(); fig.savefig("figures/day22_mde_curves.png", dpi=150)
print("Saved figures/day22_mde_curves.png")

python -m day22.mde_plan

│ metric    ┆ baseline ┆ sd     ┆ MDE at 20k/group ┆ MDE relative │
│ revenue   ┆ 70.217   ┆ 99.03  ┆ 2.774            ┆ 4.0%         │
│ orders    ┆ 1.147    ┆ 1.351  ┆ 0.038            ┆ 3.3%         │
│ converted ┆ 0.606    ┆ 0.489  ┆ 0.014            ┆ 2.3%         │
│ margin    ┆ 21.065   ┆ 29.709 ┆ 0.832            ┆ 4.0%         │

Detecting a $0.50 margin loss needs 55,421 customers per group (110,841 total). We have 40,000.
Saved figures/day22_mde_curves.png

With all 40,000 customers, email_v2 can detect a 4% revenue lift: the planned +5% is within reach. Complete the sample-size line of docs/EXPERIMENT_PLAN_email_v2.md.

The margin guardrail is a different story: seeing a $0.50 loss per customer (2.4% of margin) would take 110,000 customers. Day 25 (CUPED) and Day 30 (non-inferiority) are the way out.

22.6 Save your work with Git (15 min)

git add .
git commit -m "Day 22: power by formula and simulation, MDE planning"
git push

Day 22 checklist

• I can explain α, power and the MDE, and what each depends on
• I can compute sample sizes for means and proportions, by formula and with statsmodels
• I can estimate power by simulation
• I can explain why an underpowered significant result tends to overstate the effect
• I can decide whether a planned test is feasible, and what to change if it isn't