Sequential Testing

27.1 Why stop early, and how to do it honestly (40 min)

Concept. There are good reasons to look before the planned end: stop a harmful change quickly, or ship a clear winner sooner. Day 26 showed that looking with the usual 1.96 threshold every time breaks the error rate. The fix, developed for clinical trials, is a group-sequential design:

1. Fix the number of looks in advance (here K = 4, once a week).
2. At each look, compare the z-statistic with a boundary higher than 1.96.
3. Choose the boundaries so the probability of ever crossing one, if there's no effect, is exactly 5%.

The boundaries "spend" the 5% error budget across the looks. Where you spend it is a design choice.

27.2 Calibrating boundaries by simulation (60 min)

If there's no effect, the z-statistic at successive looks behaves like a random walk divided by √k: each look adds a new week of data to the old. We simulate 400,000 such walks and search for the boundary scale that gives a 5% crossing rate.

Create day27/boundaries.py:

# day27/boundaries.py   (python -m day27.boundaries)
"""
Group-sequential boundaries for K = 4 weekly looks, found by simulation.
Under 'no effect', the z-statistic at look k behaves like a random walk scaled by sqrt(k).
  Pocock:          same critical value at every look
  O'Brien-Fleming: very strict early, close to 1.96 at the end:  c * sqrt(K / k)
Each is calibrated so the chance of EVER crossing is 5%.
"""
import numpy as np

K, SIMS, ALPHA = 4, 400_000, 0.05
rng = np.random.default_rng(27)
paths = rng.standard_normal((SIMS, K)).cumsum(axis=1) / np.sqrt(np.arange(1, K + 1))   # z at each look
looks = np.arange(1, K + 1)


def crossing_rate(bounds: np.ndarray) -> float:
    return (np.abs(paths) > bounds).any(axis=1).mean()


def calibrate(shape: np.ndarray) -> np.ndarray:
    lo, hi = 1.0, 5.0
    for _ in range(40):                                  # bisection on the scale c
        c = (lo + hi) / 2
        lo, hi = (c, hi) if crossing_rate(c * shape) > ALPHA else (lo, c)
    return hi * shape


naive = np.full(K, 1.96)
pocock = calibrate(np.ones(K))
obf = calibrate(np.sqrt(K / looks))

if __name__ == "__main__":
    for name, b in [("naive 1.96 every week", naive), ("Pocock", pocock), ("O'Brien-Fleming", obf)]:
        print(f"{name:<22} boundaries {np.round(b, 3)}   overall false positive rate {crossing_rate(b):.3f}")

python -m day27.boundaries

naive 1.96 every week  boundaries [1.96 1.96 1.96 1.96]   overall false positive rate 0.126
Pocock                 boundaries [2.361 2.361 2.361 2.361]   overall false positive rate 0.050
O'Brien-Fleming        boundaries [4.049 2.863 2.338 2.024]   overall false positive rate 0.050

These match the published tables (Pocock 1977: 2.361; O'Brien and Fleming 1979: 2.024 at the final of four looks). Simulation lets you calibrate any design, including uneven looks or a different number of them, without tables.

Even four looks at 1.96 give a 12.6% false positive rate: less than daily peeking's 29%, but still 2.5 times the promise.

27.3 Pocock versus O'Brien–Fleming (45 min)

O'Brien–Fleming is the more common choice in practice: it costs almost nothing if you end up running the full test, and it protects against stopping on an early fluke.

Going further: "always-valid" methods (mixture sequential probability ratio tests, confidence sequences) allow looking at any time, with no fixed schedule. Several commercial platforms use them. They're built on the same idea: pay for flexibility with wider early boundaries.

27.4 Weekly outcomes from the lab (35 min)

Sequential tests need results as they accumulate. The lab's run_weekly() (at the end of causalkit/lab.py) returns one row per customer per week, so we can compute cumulative revenue after weeks 1, 2, 3 and 4 for the same customers. Read its docstring, then look at how the mini-project uses it.

27.5 Day 27 mini-project: four stopping rules, 400 experiments (75 min)

Goal. Run email_v2 (real effect) and an A/A test 200 times each with weekly looks. For four stopping rules, record how often each declares significance and how many weeks the test runs on average.

Create day27/sequential_in_lab.py:

# day27/sequential_in_lab.py   (python -m day27.sequential_in_lab)
"""
Mini-project: run email_v2 (real effect) and an A/A test 200 times each, looking at cumulative
revenue every week. Compare three stopping rules on false positives, power and duration.
"""
import numpy as np
import polars as pl

from causalkit.lab import population, run_weekly
from day27.boundaries import naive, obf, pocock

pop = population()
rng = np.random.default_rng(270)
RULES = {"fixed: look once at week 4": np.array([np.inf, np.inf, np.inf, 1.96]),
         "naive: 1.96 every week": naive, "Pocock": pocock, "O'Brien-Fleming": obf}


def weekly_z(experiment: str, seed: int) -> np.ndarray:
    z = rng.permutation(np.repeat([0, 1], pop.height // 2))
    w = run_weekly(pop, z, experiment, seed=seed).sort("customer_id", "week")
    rev = w["revenue"].to_numpy().reshape(pop.height, 4).cumsum(axis=1)   # cumulative by customer
    t = z.astype(bool)
    diff = rev[t].mean(0) - rev[~t].mean(0)
    se = np.sqrt(rev[t].var(0, ddof=1) / t.sum() + rev[~t].var(0, ddof=1) / (~t).sum())
    return diff / se


results = []
for experiment in ["aa_test", "email_v2"]:
    zs = np.array([weekly_z(experiment, 5000 + r) for r in range(200)])
    for name, bounds in RULES.items():
        crossed = np.abs(zs) > bounds
        stop_week = np.where(crossed.any(1), crossed.argmax(1) + 1, 4)
        results.append({"experiment": experiment, "rule": name,
                        "share significant": round(crossed.any(1).mean(), 3),
                        "average weeks run": round(stop_week.mean(), 2)})

with pl.Config(fmt_str_lengths=40, tbl_width_chars=110, tbl_rows=-1):
    print(pl.DataFrame(results))

python -m day27.sequential_in_lab

│ experiment ┆ rule                       ┆ share significant ┆ average weeks run │
│ aa_test    ┆ fixed: look once at week 4 ┆ 0.04              ┆ 4.0               │
│ aa_test    ┆ naive: 1.96 every week     ┆ 0.13              ┆ 3.72              │
│ aa_test    ┆ Pocock                     ┆ 0.045             ┆ 3.9               │
│ aa_test    ┆ O'Brien-Fleming            ┆ 0.035             ┆ 3.99              │
│ email_v2   ┆ fixed: look once at week 4 ┆ 0.945             ┆ 4.0               │
│ email_v2   ┆ naive: 1.96 every week     ┆ 0.96              ┆ 1.69              │
│ email_v2   ┆ Pocock                     ┆ 0.915             ┆ 2.06              │
│ email_v2   ┆ O'Brien-Fleming            ┆ 0.94              ┆ 2.64              │

Read the two halves together:

• Naive peeking finds email_v2 fastest (1.7 weeks), but would have "found" an effect in 13% of A/A tests: the speed is bought with false discoveries.
• O'Brien–Fleming keeps the A/A error rate at the planned level, keeps 94% power (the fixed test has 94.5%), and still stops email_v2 after 2.6 weeks on average: a third of the test time saved, for free.
• Pocock stops a little earlier but loses a few points of power by being stricter at the end.

(Rates from 200 replays have a simulation error of about ±1.5 points for the A/A rows.)

Add a Monitoring section to docs/EXPERIMENT_PLAN_email_v2.md: four weekly looks, O'Brien–Fleming boundaries, and a rule that only the primary metric can trigger an early stop.

27.6 Save your work with Git (15 min)

git add .
git commit -m "Day 27: group-sequential boundaries, early stopping in the lab"
git push

Coming next (Days 28–30). Everything so far assumed one customer's treatment doesn't affect another, and that we can randomise customers. Days 28–29 drop both assumptions. Day 30 is the capstone: the free-shipping experiment, from plan to decision.

Day 27 checklist

• I can explain why repeated looks need higher thresholds, and how boundaries spend α
• I can calibrate group-sequential boundaries by simulation
• I can choose between Pocock and O'Brien–Fleming for a business test
• I can show that a sequential design saves time without inflating false positives