Designing an A/B Test

21.1 Anatomy of an experiment plan (40 min)

Concept. Every decision about an experiment should be written down before any data arrive. Otherwise it's too easy to choose, after the fact, the metric, segment or stopping day that gives the answer you hoped for. A plan answers:

This is the estimand card of Day 12 extended into a full plan. In regulated fields it's called pre-registration; in tech companies, an experiment brief. The name doesn't matter. Writing it first does.

21.2 The lab and last month's baseline (40 min)

Create day21/the_lab.py:

# day21/the_lab.py   (python -m day21.the_lab)
"""Meet the experimentation lab: 40,000 customers, and last month's metrics as a baseline."""
import numpy as np
import polars as pl

from causalkit.lab import population, run

pop = population()
print(pop.drop("_rate", "_basket").head(5))
print(f"{pop.height:,} customers in {pop['household_id'].n_unique():,} households")

# "Last month": nobody treated. This is the baseline any new experiment is planned against.
last_month = run(pop, np.zeros(pop.height), "aa_test", seed=99)
summary = last_month.select(
    pl.col("orders", "revenue", "converted", "margin").mean().name.suffix("_mean"),
    pl.col("orders", "revenue", "converted", "margin").std().name.suffix("_sd"),
).unpivot().with_columns(pl.col("value").round(3))
print(summary)

python -m day21.the_lab

│ customer_id ┆ household_id ┆ region ┆ device  ┆ loyalty ┆ pre_orders ┆ pre_revenue │
│ L00001      ┆ H17668       ┆ ON     ┆ mobile  ┆ 0       ┆ 2          ┆ 44.58       │
...
40,000 customers in 16,168 households
│ orders_mean    ┆ 1.147  │
│ revenue_mean   ┆ 70.217 │
│ converted_mean ┆ 0.606  │
│ margin_mean    ┆ 21.065 │
│ orders_sd      ┆ 1.351  │
│ revenue_sd     ┆ 99.03  │
│ converted_sd   ┆ 0.489  │
│ margin_sd      ┆ 29.709 │

"Last month" is the lab run with nobody treated: the historical baseline any plan starts from. Look at the standard deviations: revenue's is 99 against a mean of 70. A few customers spend a lot, most spend little. That spread, not the mean, determines how many customers a test needs (Day 22).

21.3 Assignment by hashing (45 min)

Concept. Large companies don't flip coins and store the results. They hash each customer's ID together with an experiment-specific salt and use the result as the coin. This gives three properties for free:

1. Deterministic: the same customer always lands in the same group, on every device and every visit.
2. Independent across experiments: different salts give unrelated assignments, so customers can be in many tests at once.
3. Stateless: nothing to store or sync; any server can compute the group.

Create day21/hash_assignment.py:

# day21/hash_assignment.py   (python -m day21.hash_assignment)
"""
Assignment the way experimentation platforms do it: hash the customer id with an
experiment-specific salt. Deterministic (same customer, same group, every time),
independent across experiments (different salts), and needs no stored table.
"""
import hashlib

import numpy as np
import polars as pl
from scipy.stats import chisquare

from causalkit.lab import population


def bucket(customer_id: str, salt: str) -> int:
    """A number from 0 to 9,999, effectively random but fixed for (salt, customer)."""
    digest = hashlib.sha256(f"{salt}:{customer_id}".encode()).hexdigest()
    return int(digest[:8], 16) % 10_000


def assign(ids: pl.Series, salt: str, treat_share: float = 0.5) -> np.ndarray:
    return np.array([bucket(c, salt) < treat_share * 10_000 for c in ids], dtype=np.int8)


if __name__ == "__main__":
    pop = population()
    ids = pop["customer_id"]

    z = assign(ids, "email_v2")
    print("Treated share:", z.mean().round(4))
    print("Same answer the second time:", bool((assign(ids, "email_v2") == z).all()))

    z_other = assign(ids, "free_shipping")
    print("Overlap with another experiment's treated group:",
          round(((z == 1) & (z_other == 1)).mean(), 4), "(independent would be 0.25)")

    # Sample ratio mismatch (SRM) check: are the group sizes consistent with a 50/50 design?
    counts = np.bincount(z, minlength=2)
    stat, p = chisquare(counts, f_exp=[pop.height / 2] * 2)
    print(f"Group sizes {counts.tolist()}  SRM p-value {p:.3f}")

python -m day21.hash_assignment

Treated share: 0.5003
Same answer the second time: True
Overlap with another experiment's treated group: 0.2479 (independent would be 0.25)
Group sizes [19987, 20013]  SRM p-value 0.897

The last line is a sample ratio mismatch (SRM) check: a chi-square test of whether the group sizes are consistent with the planned 50/50 split. An SRM is one of the most common signs that an experiment is broken: a bug in assignment, logging, or eligibility. Industry rule: if SRM p < 0.001, don't read the results; find the bug.

21.4 A/A tests: test the testing machine (50 min)

Concept. In an A/A test, both groups get the same experience. The true effect is zero, so a correctly working pipeline should declare "significant" about 5% of the time at α = 0.05, and its p-values should be spread evenly between 0 and 1. If not, something in assignment, logging or analysis is broken, and every real test using that pipeline is suspect.

Create day21/aa_tests.py:

# day21/aa_tests.py   (python -m day21.aa_tests)
"""
A/A tests: both groups get the SAME experience. Any 'significant' difference is a false positive.
Run 500 of them: about 5% should be significant at alpha = 0.05, and p-values should be uniform.
If not, the pipeline (assignment, logging, analysis) is broken, before any real test runs.
"""
import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import ttest_ind

from causalkit.lab import population, run

pop = population()
rng = np.random.default_rng(21)
pvals = []
for r in range(500):
    z = rng.permutation(np.repeat([0, 1], pop.height // 2))
    out = run(pop, z, "aa_test", seed=r)
    y = out["revenue"].to_numpy()
    pvals.append(ttest_ind(y[z == 1], y[z == 0], equal_var=False).pvalue)
pvals = np.array(pvals)

print(f"Share of A/A tests with p < 0.05: {np.mean(pvals < 0.05):.3f}  (should be about 0.05)")
print(f"Share with p < 0.01:              {np.mean(pvals < 0.01):.3f}  (should be about 0.01)")

fig, ax = plt.subplots(figsize=(6.5, 3.8))
ax.hist(pvals, bins=20, range=(0, 1), color="#3987e5", edgecolor="white")
ax.axhline(len(pvals) / 20, color="black", linestyle="--", label="expected if uniform")
ax.set_xlabel("p-value"); ax.set_ylabel("A/A tests"); ax.legend()
ax.set_title("500 A/A tests: p-values should be flat")
fig.tight_layout(); fig.savefig("figures/day21_aa_pvalues.png", dpi=150)
print("Saved figures/day21_aa_pvalues.png")

python -m day21.aa_tests

Share of A/A tests with p < 0.05: 0.048  (should be about 0.05)
Share with p < 0.01:              0.008  (should be about 0.01)
Saved figures/day21_aa_pvalues.png

Open the figure: a flat histogram. That's what a healthy pipeline looks like. Many experimentation teams run A/A tests continuously, alongside real tests, as a smoke alarm.

21.5 Day 21 mini-project: the plan, and a sample-ratio bug (75 min)

Part 1 — write the plan. Create docs/EXPERIMENT_PLAN_email_v2.md with every element of the table in 21.1, filled in for email_v2. Leave the sample-size line as "to be computed on Day 22".

Part 2 — a bug hunt. The new email's tracking pixel fails for 10% of treated customers who never place an order, so those customers silently disappear from the treated group's data. Create day21/srm_bug.py:

# day21/srm_bug.py   (python -m day21.srm_bug)
"""
Mini-project, part 2: a logging bug. The new email's tracking pixel fails for 10% of treated
customers who never ordered, so they vanish from the treated group's data.
The SRM check catches it, and the estimate shows why it matters.
"""
import numpy as np
import polars as pl
from scipy.stats import chisquare

from causalkit.lab import population, run, truth
from day21.hash_assignment import assign

pop = population()
z = assign(pop["customer_id"], "email_v2")
out = run(pop, z, "email_v2", seed=7)

rng = np.random.default_rng(3)
lost = (out["treated"] == 1) & (out["orders"] == 0) & pl.Series(rng.random(out.height) < 0.10)
logged = out.filter(~lost)


def report(df: pl.DataFrame, label: str) -> None:
    counts = df.group_by("treated").len().sort("treated")["len"].to_numpy()
    p_srm = chisquare(counts, f_exp=[counts.sum() / 2] * 2).pvalue
    m = df.group_by("treated").agg(pl.col("revenue").mean()).sort("treated")["revenue"]
    print(f"{label:<22} groups {counts.tolist()}  SRM p = {p_srm:.4f}  revenue lift = {m[1] - m[0]:+.2f}")


report(out, "complete data")
report(logged, "with the logging bug")
print(f"True revenue effect: {truth('email_v2')['revenue']:+.2f} per customer")

python -m day21.srm_bug

complete data          groups [19987, 20013]  SRM p = 0.8966  revenue lift = +4.32
with the logging bug   groups [19987, 19278]  SRM p = 0.0003  revenue lift = +7.13
True revenue effect: +3.48 per customer

The bug removes non-buyers only from the treated group, so the treated group looks better than it is: the lift jumps from +4.32 to +7.13. That's collider bias from Day 16, created by a logging failure. The SRM check catches it (p = 0.0003) before anyone reads the lift.

Try it. Change the failure rate from 10% to 3% (< 0.10 → < 0.03). Does the SRM check still catch it? Is the estimate still biased?

complete data          groups [19987, 20013]  SRM p = 0.8966  revenue lift = +4.32
with the logging bug   groups [19987, 19804]  SRM p = 0.3589  revenue lift = +5.10
True revenue effect: +3.48 per customer

21.6 Save your work with Git (20 min)

git add .
git commit -m "Day 21: experiment plan, lab, hash assignment, A/A tests, SRM"
git push

Day 21 checklist

• I can write an experiment plan with hypothesis, unit, metrics, guardrails, MDE and decision rule
• I can assign treatment by hashing IDs with a salt, and explain why companies do it
• I can run a sample-ratio-mismatch check and know what to do when it fails
• I can use A/A tests to check that a pipeline's false positive rate is about α