Switchback and Geo Experiments

29.1 When you can't randomise customers (30 min)

Some changes reach everyone at once, or can only be bought for whole areas:

Both designs have few units (days, cities), so the principles from Days 21–25 matter even more: stratify or pair, use the pre-period, and quantify uncertainty with methods that don't rely on large samples.

29.2 Switchback experiments: randomising time (70 min)

Concept. In a switchback, the whole marketplace switches between treatment and control on a random schedule (by day, hour, or block of days). Each period is one unit. Analysis compares treated periods with control periods, controlling for time patterns (day of week, trend).

Create day29/switchback.py. It's a toy marketplace (not the customer lab): 112 days of orders with weekly seasonality, an upward trend, autocorrelated noise and a true effect of +40 orders/day. Half of the effect lingers into the day after a switch-off.

# day29/switchback.py   (python -m day29.switchback)
"""
A change that hits EVERY customer at once: a new delivery-promise algorithm on the website.
Customer-level randomisation is impossible, so randomise TIME instead (switchback).
Toy marketplace: 112 days of daily orders with weekly seasonality, a slow upward trend,
autocorrelated noise, and a true effect of +40 orders/day. Part of the effect carries over
into the day after the algorithm is switched off (customers' expectations adjust slowly).
"""
import numpy as np
import polars as pl
import statsmodels.formula.api as smf

DAYS, EFFECT, CARRYOVER = 112, 40.0, 0.5
dow = np.arange(DAYS) % 7
season = np.array([0, -20, -10, 0, 30, 80, 60])[dow]


def simulate(treated: np.ndarray, rng) -> np.ndarray:
    noise = np.zeros(DAYS)
    for d in range(1, DAYS):
        noise[d] = 0.6 * noise[d - 1] + rng.normal(0, 25)
    carry = np.r_[0, treated[:-1]] * (1 - treated) * CARRYOVER * EFFECT    # lingering effect after switch-off
    return 1000 + 1.5 * np.arange(DAYS) + season + EFFECT * treated + carry + noise


def estimate(y: np.ndarray, t: np.ndarray, drop_after_switch: bool = False) -> float:
    df = pl.DataFrame({"y": y, "t": t, "dow": dow, "day": np.arange(DAYS)})
    if drop_after_switch:                                  # wash-out: ignore the first day of each period
        df = df.filter(pl.Series(np.r_[False, t[1:] != t[:-1]]).not_())
    return smf.ols("y ~ t + C(dow) + day", df.to_pandas()).fit().params["t"]


rng = np.random.default_rng(29)
designs = {"before/after (56 days each)": [], "switchback, daily": [],
           "switchback, weekly blocks": [], "switchback, weekly + wash-out": []}
for r in range(1000):
    before_after = (np.arange(DAYS) >= 56).astype(float)
    daily = rng.integers(0, 2, DAYS).astype(float)
    weekly = np.repeat(rng.integers(0, 2, DAYS // 7), 7).astype(float)
    ba_y = simulate(before_after, rng)
    designs["before/after (56 days each)"].append(ba_y[56:].mean() - ba_y[:56].mean())
    designs["switchback, daily"].append(estimate(simulate(daily, rng), daily))
    y_w = simulate(weekly, rng)
    designs["switchback, weekly blocks"].append(estimate(y_w, weekly))
    designs["switchback, weekly + wash-out"].append(estimate(y_w, weekly, drop_after_switch=True))

with pl.Config(fmt_str_lengths=40, tbl_width_chars=100):
    print(pl.DataFrame([{"design": k, "mean estimate": round(np.mean(v), 1), "SD": round(np.std(v), 1)}
                        for k, v in designs.items()]))
print(f"True effect: +{EFFECT:.0f} orders per day")

python -m day29.switchback

│ design                        ┆ mean estimate ┆ SD   │
│ before/after (56 days each)   ┆ 124.1         ┆ 11.4 │
│ switchback, daily             ┆ 29.7          ┆ 6.3  │
│ switchback, weekly blocks     ┆ 38.7          ┆ 10.9 │
│ switchback, weekly + wash-out ┆ 40.3          ┆ 11.5 │
True effect: +40 orders per day

• Before/after ("launch it and compare with last month") reports +124: three times the truth, because the trend adds 1.5 orders a day. No standard error fixes that.
• Daily switchback is the most precise (SD 6.3) but biased low (29.7): every switch-off day still carries half the effect, contaminating the control days.

29.3 Carryover and wash-out periods (45 min)

Carryover is interference across time: today's treatment affects tomorrow's outcome (customers' expectations, inventory positions, drivers' locations). It biases switchbacks towards zero, more so the more often you switch.

Two fixes, used together:

1. Longer periods (weekly blocks): fewer switches, so less contamination. Bias almost disappears (38.7), but precision drops (fewer, longer periods).
2. Wash-out: drop the first day after each switch from the analysis. With weekly blocks, the estimate is unbiased (40.3).

Choosing the period length is a trade-off between bias (carryover) and variance (number of periods). Run an A/A switchback first to learn the noise, and estimate carryover from past launches.

Try it. Set CARRYOVER = 0 and rerun. Which design is best now?

│ before/after (56 days each)   ┆ 124.6         ┆ 11.5 │
│ switchback, daily             ┆ 39.8          ┆ 6.0  │
│ switchback, weekly blocks     ┆ 41.0          ┆ 10.6 │
│ switchback, weekly + wash-out ┆ 41.3          ┆ 11.4 │

29.4 Geo experiments: randomising places (40 min)

Concept. In a geo experiment, regions or cities are the units. Typical numbers are 10–200 units of very different sizes, with strong common shocks. Three habits make them work:

1. Use the pre-period. Compare each region's growth with its own history (the CUPED idea, applied to regions), not raw levels.
2. Pair or stratify similar regions before randomising (Day 23), when the pre-period doesn't already absorb the differences.
3. Randomisation inference (Day 13): with 20 units, normal-approximation p-values are shaky. Re-draw the assignment thousands of times under "no effect" and see where the real result falls.

29.5 Day 29 mini-project: a 20-city radio campaign (70 min)

Goal. NorthMart buys local radio in 10 of 20 cities. Cities differ in size by a factor of 20. True effect: +4% of a city's sales. Compare two analyses over 2,000 simulated campaigns, then analyse one campaign with randomisation inference.

Create day29/geo_experiment.py:

# day29/geo_experiment.py   (python -m day29.geo_experiment)
"""
Mini-project: a geo experiment. A local radio campaign can only be bought by CITY.
20 cities of very different sizes; 10 get the campaign. With only 20 units, the analysis matters.
Toy data: weekly sales for 8 pre-period weeks and 4 test weeks; true effect +4% of a city's sales.
"""
import numpy as np
import polars as pl

rng = np.random.default_rng(29)
N, PRE, TEST, LIFT = 20, 8, 4, 0.04
size = rng.lognormal(np.log(50_000), 0.8, N)                    # typical weekly sales per city ($)


def sales(treated: np.ndarray, seed: int) -> tuple[np.ndarray, np.ndarray]:
    r = np.random.default_rng(seed)
    drift = r.normal(0, 0.02, N)                                 # cities drift a little
    pre = size[:, None] * (1 + r.normal(0, 0.03, (N, PRE)))
    test = size[:, None] * (1 + drift[:, None] + LIFT * treated[:, None] + r.normal(0, 0.03, (N, TEST)))
    return pre.sum(1), test.sum(1)


def raw_lift(pre, test, t) -> float:
    """Difference in average test-period sales, as a share of average sales (cities differ hugely in size)."""
    return (test[t == 1].mean() - test[t == 0].mean()) / test.mean()


def growth_lift(pre, test, t) -> float:
    """Each city compared with ITSELF first: growth vs its own pre-period, then treated minus control."""
    growth = test / (pre / PRE * TEST) - 1
    return growth[t == 1].mean() - growth[t == 0].mean()


est = {"raw sales": [], "growth vs own pre-period": []}
for r in range(2000):
    t = np.zeros(N); t[rng.choice(N, 10, replace=False)] = 1
    pre, test = sales(t, r)
    est["raw sales"].append(raw_lift(pre, test, t))
    est["growth vs own pre-period"].append(growth_lift(pre, test, t))
print(pl.DataFrame([{"analysis": k, "mean": f"{np.mean(v):+.1%}", "SD": f"{np.std(v):.1%}"} for k, v in est.items()]))
print(f"True effect: +{LIFT:.0%}")

# One real geo test, analysed with randomisation inference: re-draw the assignment 20,000 times
t = np.zeros(N); t[rng.choice(N, 10, replace=False)] = 1
pre, test = sales(t, 424_242)
observed = growth_lift(pre, test, t)
null = []
for _ in range(20_000):
    fake = np.zeros(N); fake[rng.choice(N, 10, replace=False)] = 1
    null.append(growth_lift(pre, test, fake))                    # sharp null: the campaign did nothing anywhere
p = np.mean(np.abs(null) >= abs(observed))
print(f"\nOne geo test: estimated lift {observed:+.1%}, randomisation-inference p-value {p:.4f}")

python -m day29.geo_experiment

│ analysis                 ┆ mean  ┆ SD    │
│ raw sales                ┆ +4.0% ┆ 20.4% │
│ growth vs own pre-period ┆ +4.0% ┆ 1.2%  │
True effect: +4%

One geo test: estimated lift +4.9%, randomisation-inference p-value 0.0022

• Comparing raw sales is unbiased but useless: an SD of 20 points against a 4-point effect. Which cities happen to be treated matters far more than the campaign.
• Comparing growth against each city's own pre-period cuts the noise 17-fold. Same data, same 20 cities.
• In the single test, randomisation inference re-draws 20,000 assignments of 10 cities: only 0.22% give a lift as large as the one observed. No distributional assumptions needed.

Write a one-paragraph geo test plan in docs/ for a real campaign: number of cities, how you'd pair them, pre-period length, the analysis, and the MDE implied by the 1.2% standard deviation.

29.6 Save your work with Git (15 min)

git add .
git commit -m "Day 29: switchback designs, carryover, geo experiment with randomisation inference"
git push

Day 29 checklist

• I can tell when customer-level randomisation is impossible, and choose a switchback or geo design
• I can explain why before/after comparisons fail, even with long periods
• I can handle carryover with block length and wash-out periods
• I can analyse a geo test with pre-period growth and randomisation inference