Day 30: Capstone: The Free-Shipping Experiment · Applied Causal Inference & Decision Science

Written lesson

Time	Block
0:00 – 0:30	30.1 The brief and how it is assessed
0:30 – 1:15	30.2 The plan, before any data
1:15 – 1:30	Break
1:30 – 2:00	30.3 Assign, audit, run
2:00 – 3:00	30.4 The readout: primary metric and guardrail
3:00 – 3:15	Break
3:15 – 3:50	30.5 Pre-registered segments
3:50 – 4:40	30.6 The decision memo
4:40 – 5:00	30.7 Answer key, final commit and Module 3 retrospective

30.1 The brief and how it is assessed (30 min)

The brief. NorthMart's e-commerce director writes:

"Competitors offer free shipping from $35. We charge $7 below $50. Marketing is sure lowering our threshold to $35 will raise sales; Finance is worried we'll give away margin on every order between $35 and $50. I want a proper test before deciding. I need to know three things: does it raise revenue, does it hurt margin by more than we can accept (Finance says a loss of up to $1 per customer per month is fine if revenue grows), and should we launch it for everyone or only for some customers? Plan it before you run it."

Deliverables.

#	Deliverable	Where
D1	Pre-registered plan, written before running	`docs/FREE_SHIPPING_PLAN.md`
D2	Power and MDE for revenue and the margin guardrail, overall and by segment	`day30/plan.py`
D3	Assignment audit (SRM, balance)	`day30/run_test.py`
D4	Readout: CUPED-adjusted effects, non-inferiority guardrail, corrected segment tests	`day30/readout.py`
D5	One-page decision memo	`docs/FREE_SHIPPING_DECISION.md`

Assessment rubric.

Criterion	Weight	Excellent
Pre-registration	20%	Metrics, guardrail margin, segments, analysis and decision rule fixed before running; committed to Git first
Design	20%	Power justified; stratified assignment; audit clean
Analysis	25%	CUPED, robust SEs, non-inferiority test done correctly, segments corrected
Decision	20%	Clear recommendation that follows from the pre-registered rule and states its uncertainty
Communication	15%	A director can read the memo in five minutes and act

30.2 The plan, before any data (45 min)

Write docs/FREE_SHIPPING_PLAN.md using the Day 21 template. The brief fixes most of it:

Element	Choice
Hypothesis	Lowering the free-shipping threshold to $35 raises revenue; margin loss is at most $1/customer
Unit, population	customer; all 40,000
Duration	4 weeks
Primary metric	revenue per customer (two-sided test, α = 0.05)
Secondary	orders, conversion (Holm-corrected)
Guardrail	margin per customer: non-inferiority with margin −$1.00 (one-sided, α = 0.05): pass if the lower bound of the one-sided 95% interval is above −$1.00
Segments	loyalty members vs non-members, pre-registered, Holm correction across segment tests (the director asked "for whom?")
Analysis	regression with CUPED covariates (pre_revenue, pre_orders) and stratum dummies; robust SEs
Assignment	stratified hash assignment, salt `free_shipping_2026`
Decision rule	launch for a population if its revenue effect is positive and significant and it passes the margin guardrail

Non-inferiority, in one sentence: instead of asking "did margin fall?" (which needs 81,000 customers to answer, Day 25), we ask "can we rule out a fall worse than $1?", which is the question Finance actually asked.

Now check feasibility. Create day30/plan.py:

# day30/plan.py   (python -m day30.plan)
"""
Capstone step 1 (before running anything): can 40,000 customers answer the questions?
Primary metric: revenue (superiority).  Guardrail: margin (non-inferiority: no loss worse than $1.00).
Pre-registered segments: loyalty members and non-members.
"""
import numpy as np
import polars as pl
import statsmodels.formula.api as smf
from scipy.stats import norm

from causalkit.lab import population, run

NI_MARGIN = -1.00                       # the largest margin loss per customer we would accept
pop = population()
last_month = run(pop, np.zeros(pop.height), "aa_test", seed=99).join(
    pop.select("customer_id", "pre_revenue", "pre_orders", "loyalty"), on="customer_id").to_pandas()

Z_TWO = norm.ppf(0.975) + norm.ppf(0.80)     # two-sided superiority test, 80% power
Z_ONE = norm.ppf(0.95) + norm.ppf(0.80)      # one-sided non-inferiority test, 80% power

rows = []
for label, sel in [("everyone", np.ones(len(last_month), bool)),
                   ("non-members", last_month.loyalty.to_numpy() == 0),
                   ("loyalty members", last_month.loyalty.to_numpy() == 1)]:
    d = last_month[sel]
    n_group = sel.sum() / 2
    for metric in ["revenue", "margin"]:
        r2 = smf.ols(f"{metric} ~ pre_revenue + pre_orders", d).fit().rsquared
        sd = d[metric].std() * np.sqrt(1 - r2)                          # CUPED-adjusted SD
        mde = Z_TWO * sd * np.sqrt(2 / n_group)
        row = {"population": label, "metric": metric, "per group": int(n_group),
               "CUPED SD": round(sd, 2), "MDE (two-sided)": round(mde, 2)}
        if metric == "margin":   # power to show 'no worse than -$1' if the true effect is 0
            se = sd * np.sqrt(2 / n_group)
            row["NI power if truly 0"] = f"{norm.cdf((0 - NI_MARGIN) / se - norm.ppf(0.95)):.0%}"
        rows.append(row)
with pl.Config(tbl_width_chars=120):
    print(pl.DataFrame(rows))

python -m day30.plan

│ population      ┆ metric  ┆ per group ┆ CUPED SD ┆ MDE (two-sided) ┆ NI power if truly 0 │
│ everyone        ┆ revenue ┆ 20000     ┆ 85.22    ┆ 2.39            ┆ null                │
│ everyone        ┆ margin  ┆ 20000     ┆ 25.57    ┆ 0.72            ┆ 99%                 │
│ non-members     ┆ revenue ┆ 14003     ┆ 76.94    ┆ 2.58            ┆ null                │
│ non-members     ┆ margin  ┆ 14003     ┆ 23.08    ┆ 0.77            ┆ 98%                 │
│ loyalty members ┆ revenue ┆ 5996      ┆ 100.46   ┆ 5.14            ┆ null                │
│ loyalty members ┆ margin  ┆ 5996      ┆ 30.14    ┆ 1.54            ┆ 57%                 │

Record in the plan: overall and for non-members, the test can detect a revenue effect of about $2.40–$2.60 and has near-certain power to pass the guardrail if margin is truly unchanged. For loyalty members (only 30% of customers), the revenue MDE is $5 and guardrail power is 57%: a "fail" there will be hard to distinguish from "uncertain". Write that down now, so nobody can reinterpret it later.

Commit the plan before running anything:

git add docs/FREE_SHIPPING_PLAN.md day30/plan.py
git commit -m "Day 30: pre-registered free-shipping plan"

The commit timestamp is your proof of pre-registration.

30.3 Assign, audit, run (30 min)

Create day30/run_test.py:

# day30/run_test.py   (python -m day30.run_test)
"""Capstone step 2: assign (stratified), audit, run the 4-week test, store the results."""
from pathlib import Path

from causalkit.assignment import add_strata, assign_stratified, balance_table, srm_pvalue
from causalkit.lab import population, run

pop = population()
z = assign_stratified(pop, "free_shipping_2026")
print(f"Treated {z.sum():,} of {len(z):,}   SRM p = {srm_pvalue(z):.3f}")
print(balance_table(pop, z))

out = run(pop, z, "free_shipping", seed=30)          # the four weeks happen here
results = out.join(add_strata(pop).select("customer_id", "loyalty", "pre_orders", "pre_revenue", "stratum"),
                   on="customer_id")
Path("data/experiments").mkdir(parents=True, exist_ok=True)
results.write_parquet("data/experiments/free_shipping.parquet")
print(f"Saved {results.height:,} rows to data/experiments/free_shipping.parquet")

python -m day30.run_test

Treated 19,997 of 40,000   SRM p = 0.976
│ covariate   ┆ treated ┆ control ┆ smd     ┆ flag  │
│ pre_orders  ┆ 2.263   ┆ 2.274   ┆ -0.0051 ┆ false │
│ pre_revenue ┆ 137.758 ┆ 138.956 ┆ -0.0072 ┆ false │
│ loyalty     ┆ 0.3     ┆ 0.3     ┆ -0.0001 ┆ false │
│ mobile      ┆ 0.62    ┆ 0.621   ┆ -0.0027 ┆ false │
Saved 40,000 rows to data/experiments/free_shipping.parquet

Clean audit: no SRM, every |SMD| below 0.01. Only now may you look at outcomes.

30.4 The readout: primary metric and guardrail (60 min)

Create day30/readout.py:

# day30/readout.py   (python -m day30.readout)
"""Capstone step 3: the pre-registered readout. CUPED throughout; margin as a non-inferiority guardrail."""
import polars as pl
import statsmodels.formula.api as smf
from scipy.stats import norm
from statsmodels.stats.multitest import multipletests

from causalkit.analysis import readout

NI_MARGIN = -1.00
df = pl.read_parquet("data/experiments/free_shipping.parquet")
COVS = ["pre_revenue", "pre_orders", "C(stratum)"]

print("1. All customers (CUPED-adjusted)")
with pl.Config(tbl_width_chars=120):
    print(readout(df, ["revenue", "orders", "converted", "margin"], covariates=COVS))


def guardrail(d: pl.DataFrame, label: str) -> dict:
    fit = smf.ols("margin ~ treated + " + " + ".join(COVS), d.to_pandas()).fit(cov_type="HC2")
    eff, se = fit.params["treated"], fit.bse["treated"]
    lower = eff - norm.ppf(0.95) * se                       # one-sided 95% lower bound
    p_ni = norm.sf((eff - NI_MARGIN) / se)                  # H0: effect <= -1
    return {"population": label, "margin_effect": round(eff, 3), "lower_bound_95": round(lower, 3),
            "p_noninferiority": fmt(p_ni), "passes": bool(lower > NI_MARGIN)}


def fmt(p: float) -> str:
    return f"{p:.4f}" if p >= 0.0001 else "<0.0001"


print(f"\n2. Margin guardrail: non-inferiority, margin {NI_MARGIN:+.2f} per customer")
print(pl.DataFrame([guardrail(df, "everyone")]))

print("\n3. Pre-registered segments (Holm correction across the 4 segment tests)")
rows = []
for flag, label in [(0, "non-members"), (1, "loyalty members")]:
    seg = df.filter(pl.col("loyalty") == flag)
    for metric in ["revenue", "margin"]:
        fit = smf.ols(f"{metric} ~ treated + pre_revenue + pre_orders + C(stratum)",
                      seg.to_pandas()).fit(cov_type="HC2")
        rows.append({"segment": label, "metric": metric, "effect": round(fit.params["treated"], 3),
                     "se": round(fit.bse["treated"], 3), "p": fit.pvalues["treated"]})
seg_table = pl.DataFrame(rows)
reject, p_holm, _, _ = multipletests(seg_table["p"].to_numpy(), method="holm")
seg_table = seg_table.with_columns(p=pl.Series([fmt(v) for v in seg_table["p"]]),
                                   p_holm=pl.Series([fmt(v) for v in p_holm]), significant=pl.Series(reject))
print(seg_table)
print(pl.DataFrame([guardrail(df.filter(pl.col("loyalty") == f), lab)
                    for f, lab in [(0, "non-members"), (1, "loyalty members")]]))

python -m day30.readout

Part 1 and 2:

1. All customers (CUPED-adjusted)
│ metric    ┆ control_mean ┆ effect ┆ se    ┆ ci_low ┆ ci_high ┆ lift   ┆ p_value │
│ revenue   ┆ 69.967       ┆ 5.365  ┆ 0.867 ┆ 3.665  ┆ 7.065   ┆ +7.67% ┆ <0.0001 │
│ orders    ┆ 1.151        ┆ 0.079  ┆ 0.012 ┆ 0.055  ┆ 0.103   ┆ +6.90% ┆ <0.0001 │
│ converted ┆ 0.603        ┆ 0.028  ┆ 0.005 ┆ 0.019  ┆ 0.037   ┆ +4.63% ┆ <0.0001 │
│ margin    ┆ 20.99        ┆ -0.395 ┆ 0.256 ┆ -0.897 ┆ 0.107   ┆ -1.88% ┆ 0.1228  │

2. Margin guardrail: non-inferiority, margin -1.00 per customer
│ population ┆ margin_effect ┆ lower_bound_95 ┆ p_noninferiority ┆ passes │
│ everyone   ┆ -0.395        ┆ -0.816         ┆ 0.0091           ┆ true   │

For all customers: revenue up $5.37 (+7.7%), clearly significant. Margin down $0.40, not significantly different from zero, and the guardrail passes: we can rule out a loss worse than $0.82, inside Finance's $1 limit. Following the overall rule, we'd launch for everyone.

But the director asked a third question.

30.5 Pre-registered segments (35 min)

Part 3:

3. Pre-registered segments (Holm correction across the 4 segment tests)
│ segment         ┆ metric  ┆ effect ┆ se    ┆ p       ┆ p_holm  ┆ significant │
│ non-members     ┆ revenue ┆ 6.975  ┆ 0.95  ┆ <0.0001 ┆ <0.0001 ┆ true        │
│ non-members     ┆ margin  ┆ 0.316  ┆ 0.28  ┆ 0.2605  ┆ 0.5210  ┆ false       │
│ loyalty members ┆ revenue ┆ 1.615  ┆ 1.856 ┆ 0.3840  ┆ 0.5210  ┆ false       │
│ loyalty members ┆ margin  ┆ -2.052 ┆ 0.547 ┆ 0.0002  ┆ 0.0005  ┆ true        │
│ population      ┆ margin_effect ┆ lower_bound_95 ┆ p_noninferiority ┆ passes │
│ non-members     ┆ 0.316         ┆ -0.146         ┆ <0.0001          ┆ true   │
│ loyalty members ┆ -2.052        ┆ -2.952         ┆ 0.9728           ┆ false  │

The overall result averages two very different stories:

Non-members: revenue +$6.98, significant; margin unchanged (+$0.32, guardrail passes easily). The lower threshold brings in orders that wouldn't have happened, enough to pay for the shipping.
Loyalty members: revenue +$1.62, not significant; margin −$2.05, significantly negative even after correction, and the guardrail fails. Members already buy regularly; free shipping mostly subsidises orders they'd have placed anyway.

This is a legitimate segment finding, unlike Day 26's "best region": the segments were pre-registered, there are only two, the tests are Holm-corrected, and the planned power analysis said members would be the hard case. The margin loss for members is large enough to be detected even so.

30.6 The decision memo (50 min)

Write docs/FREE_SHIPPING_DECISION.md: one page, for the director. Headings:

Recommendation (two sentences).
What we tested (one paragraph: design, size, duration, pre-registration).
Results (a small table: revenue and margin, overall and by segment, with intervals).
Why not launch for everyone? The overall guardrail passed. Explain why the segment result should still change the decision, and why it isn't fishing.
Risks and next steps (for example: the effect after four weeks may fade or grow; members excluded from free shipping may notice; re-test members with a different perk).

Model answer for sections 1 and 4

Recommendation. Lower the free-shipping threshold to $35 for customers who are not loyalty members: it raised their revenue by about $7 per customer per month with no detectable loss of margin. Keep the $50 threshold for loyalty members, for whom it cost about $2 of margin per customer with no significant revenue gain.

Why not launch for everyone? Across all customers, the margin loss ($0.40) was within Finance's $1 limit, so the overall rule would say "launch". But that average hides a clear split we planned to check: members lose $2 of margin each, twice the acceptable limit. Launching for non-members only keeps nearly all of the revenue gain ($6.98 × 70% of customers ≈ $4.88 per customer overall, against $5.37 for everyone) and turns a margin loss into a small gain. Because this split was written into the plan before the test, tested with a correction for multiple comparisons, and is large relative to its uncertainty, it's a finding, not a fishing expedition.

30.7 Answer key, final commit and Module 3 retrospective (20 min)

After the memo is committed, compare with the truth. Create day30/answer_key.py:

# day30/answer_key.py   (python -m day30.answer_key)
"""Instructor check, after the memo is written: the true effects, overall and by segment."""
import numpy as np
import polars as pl
from scipy.stats import norm

from causalkit import lab

pop = lab._population(2030)
rate, basket = pop["_rate"].to_numpy(), pop["_basket"].to_numpy()
loyal = pop["loyalty"].to_numpy().astype(bool)
lift = np.where(loyal, *lab.EXPERIMENTS["free_shipping"]["lift"][::-1])

mu0 = rate * lab.TEST_WEEKS
mu1 = mu0 * (1 + lift)
share_35_50 = (norm.cdf((np.log(lab.OLD_THRESHOLD / basket) + 0.045) / 0.3)
               - norm.cdf((np.log(lab.NEW_THRESHOLD / basket) + 0.045) / 0.3))
d_revenue = (mu1 - mu0) * basket
d_margin = lab.GROSS_MARGIN * d_revenue - lab.SHIPPING_FEE * share_35_50 * mu1

rows = []
for label, sel in [("everyone", np.ones_like(loyal)), ("non-members", ~loyal), ("loyalty members", loyal)]:
    rows.append({"population": label, "true revenue effect": round(d_revenue[sel].mean(), 2),
                 "true margin effect": round(d_margin[sel].mean(), 2),
                 "true order lift": f"{lift[sel].mean():.0%}"})
print(pl.DataFrame(rows))

python -m day30.answer_key

│ population      ┆ true revenue effect ┆ true margin effect ┆ true order lift │
│ everyone        ┆ 4.98                ┆ -0.52              ┆ 8%              │
│ non-members     ┆ 5.9                 ┆ 0.02               ┆ 10%             │
│ loyalty members ┆ 2.83                ┆ -1.79              ┆ 3%              │

Every estimate's confidence interval contains its true value, and the segment decision is the right one: free shipping is break-even on margin for non-members (+$0.02) and loses $1.79 per member.

git add .
git commit -m "Day 30: free-shipping capstone: readout and decision"
git push

What you learned in Module 3.

Day	Idea	The lab lesson
21	Plans, hashing, A/A tests, SRM	A 10% logging bug inflated the lift by 65%; SRM caught it
22	Power	5% revenue lift needs ~25,000 customers; a $0.50 margin loss needs 110,000
23	Stratification	11% less noise for free
24	Regression readout	Covariates cut noise 13%; the estimate's target doesn't move
25	CUPED, capping	CUPED is worth 32% more customers; capping changes the estimand
26	Multiple tests, peeking	Daily peeking: 29% false positives; the "best region" overstates by 74%
27	Sequential tests	O'Brien–Fleming saves a third of test time at 5% error
28	Interference	Household spillover hid 35% of the effect; cluster randomisation fixes it
29	Switchbacks, geo tests	Before/after was 3× the truth; pre-period growth cut geo noise 17-fold
30	Capstone	Overall "launch" would have cost $2 per loyalty member; the pre-registered segments caught it

Coming next: Module 4, Selection on Observables & Instrumental Variables. Experiments are the benchmark, but most business decisions were never randomised. Module 4 returns to observational data with propensity scores, matching, weighting, doubly robust estimators and instrumental variables, and grades each one against the truth.

Day 30 checklist

I pre-registered a complete plan and committed it before running the test
I justified the design with power analysis, overall and by segment
I tested a guardrail with non-inferiority instead of a significance test
I analysed pre-registered segments with a multiple-testing correction
I wrote a decision memo whose recommendation follows from the plan

Capstone: The Free-Shipping Experiment

What you'll learn today

Today's plan