Interference and Network Effects

28.1 When treatment leaks (40 min)

Concept. Interference means one unit's treatment changes another unit's outcome. Then "treated vs control" compares two groups that are both partly treated, and the difference is not the effect the business cares about: treat everyone vs treat no one.

Two kinds, with opposite consequences:

The standard fix is to randomise clusters that contain the interference: households, cities, store catchments, whole markets.

28.2 Spillover in NorthMart households (60 min)

The lab's run() has a spillover argument: the share of a treated customer's lift that reaches the other members of their household. Suppose it's 50%.

Create day28/household_spillover.py:

# day28/household_spillover.py   (python -m day28.household_spillover)
"""
Interference: household members share inboxes and shopping trips. If half of the email's lift
spills over to the other members of a treated customer's household, what does each design estimate?
Target: the effect of emailing EVERYONE vs NO ONE (the decision marketing actually faces).
"""
import numpy as np
import polars as pl

from causalkit.lab import population, run, truth

SPILL = 0.5
pop = population()
hh = pop["household_id"].to_numpy()
households, inverse = np.unique(hh, return_inverse=True)
rng = np.random.default_rng(28)

est = {"individual randomisation": [], "household randomisation": []}
for r in range(300):
    z_ind = rng.permutation(np.repeat([0, 1], pop.height // 2))
    z_hh = (rng.random(len(households)) < 0.5)[inverse].astype(np.int8)
    for name, z in [("individual randomisation", z_ind), ("household randomisation", z_hh)]:
        y = run(pop, z, "email_v2", seed=6000 + r, spillover=SPILL)["orders"].to_numpy()
        est[name].append(y[z == 1].mean() - y[z == 0].mean())

target = truth("email_v2")["orders"]
print(pl.DataFrame([{"design": k, "mean estimate": round(np.mean(v), 4), "SD": round(np.std(v), 4),
                     "bias vs truth": f"{np.mean(v) / target - 1:+.0%}"} for k, v in est.items()]))
print(f"True effect of emailing everyone vs no one: {target:+.4f} orders per customer")

python -m day28.household_spillover

│ design                   ┆ mean estimate ┆ SD     ┆ bias vs truth │
│ individual randomisation ┆ 0.0368        ┆ 0.0147 ┆ -35%          │
│ household randomisation  ┆ 0.0564        ┆ 0.0145 ┆ -1%           │
True effect of emailing everyone vs no one: +0.0570 orders per customer

The customer-level test reports an effect 35% too small: two thirds of control customers (67%) share a household with a treated customer and get half the lift. Randomising whole households restores the comparison: no control customer lives with a treated one.

Try it. Repeat the individual-randomisation test with spillover 0 and 1.0. How does the bias change?

# day28/full_spillover.py   (python -m day28.full_spillover)
import numpy as np

from causalkit.lab import population, run, truth

pop = population()
rng = np.random.default_rng(283)
for spill in [0.0, 0.5, 1.0]:
    est = []
    for r in range(60):
        z = rng.permutation(np.repeat([0, 1], pop.height // 2))
        y = run(pop, z, "email_v2", seed=9000 + r, spillover=spill)["orders"].to_numpy()
        est.append(y[z == 1].mean() - y[z == 0].mean())
    print(f"spillover {spill:.1f}: average estimate {np.mean(est):.4f}  (truth {truth('email_v2')['orders']:.4f})")

spillover 0.0: average estimate 0.0566  (truth 0.0570)
spillover 0.5: average estimate 0.0373  (truth 0.0570)
spillover 1.0: average estimate 0.0185  (truth 0.0570)

28.3 Cluster randomisation and its price (45 min)

Concept. When households are randomised, the household is the unit: standard errors must be clustered by household. How much precision does that cost? It depends on the intra-cluster correlation (ICC), how alike members of a cluster are:

design effect = 1 + (m − 1) × ICC        m = average cluster size

The design effect is the factor by which the sample must grow to match an individually randomised test.

Create day28/cluster_se.py:

# day28/cluster_se.py   (python -m day28.cluster_se)
"""
Randomising households means the unit of randomisation is the household: standard errors
must be clustered by household. How much does clustering cost here? It depends on the
intra-cluster correlation (ICC): how alike members of the same household are.
"""
import numpy as np
import polars as pl
import statsmodels.formula.api as smf

from causalkit.lab import population, run

pop = population()
hh = pop["household_id"].to_numpy()
households, inverse = np.unique(hh, return_inverse=True)
rng = np.random.default_rng(281)
z = (rng.random(len(households)) < 0.5)[inverse].astype(np.int8)
df = run(pop, z, "email_v2", seed=281).with_columns(household_id=pl.Series(hh)).to_pandas()

naive = smf.ols("orders ~ treated", df).fit(cov_type="HC2")
clustered = smf.ols("orders ~ treated", df).fit(cov_type="cluster", cov_kwds={"groups": inverse})
print(f"Effect {naive.params['treated']:+.4f}   naive SE {naive.bse['treated']:.4f}   "
      f"household-clustered SE {clustered.bse['treated']:.4f}")

# ICC: share of variance that is between households (one-way ANOVA estimator)
m = df.groupby("household_id")["orders"].agg(["mean", "size"])
k = m["size"].mean()
between = np.average((m["mean"] - df["orders"].mean()) ** 2, weights=m["size"]) * len(m) / (len(m) - 1)
within = df.groupby("household_id")["orders"].var(ddof=1).mean()
icc = max(0.0, (between - within / k) / (between - within / k + within))
print(f"Average household size {k:.2f}   ICC {icc:.3f}   design effect 1 + (m - 1) x ICC = {1 + (k - 1) * icc:.3f}")
for rho in [0.05, 0.1, 0.3]:
    print(f"  if ICC were {rho:.2f}: design effect {1 + (k - 1) * rho:.2f} -> need {1 + (k - 1) * rho:.0%} of the sample")

Effect +0.0380   naive SE 0.0139   household-clustered SE 0.0138
Average household size 2.47   ICC 0.000   design effect 1 + (m - 1) x ICC = 1.000
  if ICC were 0.05: design effect 1.07 -> need 107% of the sample
  if ICC were 0.10: design effect 1.15 -> need 115% of the sample
  if ICC were 0.30: design effect 1.44 -> need 144% of the sample

In the lab, household members shop independently (ICC ≈ 0), so clustering costs nothing. In real data, household members share income, location and habits; ICCs of 0.1–0.3 are common. For cities (Day 29), ICCs are smaller but clusters are huge, so the design effect is enormous.

Always cluster standard errors at the level of randomisation, even when you expect the ICC to be small. If it is small, you lose nothing.

28.4 Where interference hides (40 min)

The question to ask in every plan: if I treat this customer, whose outcome could change besides theirs? If the answer isn't "nobody", interference needs a design response.

28.5 Day 28 mini-project: measure the spillover (70 min)

Goal. Cluster randomisation avoids spillover. A two-stage (saturation) design measures it, which tells you whether the cluster design was needed and how big the network effect is.

• Stage 1: randomise households (with 2+ members) into NONE, ONE, or ALL.
• Stage 2: in ONE households, randomly choose which member is emailed.

Comparing each group with NONE households gives three effects.

Create day28/saturation_design.py:

# day28/saturation_design.py   (python -m day28.saturation_design)
"""
Mini-project: MEASURE the spillover instead of just avoiding it. Two-stage (saturation) design,
households with 2+ members only:
  stage 1: each household is randomised to NONE, ONE (one random member emailed) or ALL
  stage 2: in ONE households, which member is emailed is random
Comparisons against NONE households give the direct, spillover and total effects.
"""
import numpy as np
import polars as pl

from causalkit.lab import population, run

SPILL = 0.5
pop = population()
hh = pop["household_id"].to_numpy()
_, inverse, sizes = np.unique(hh, return_inverse=True, return_counts=True)
multi = sizes[inverse] >= 2
rng = np.random.default_rng(282)

res = {"direct (emailed, in ONE households)": [], "spillover (not emailed, in ONE households)": [],
       "total (everyone, in ALL households)": []}
for r in range(200):
    arm = rng.integers(0, 3, inverse.max() + 1)[inverse]           # 0 = NONE, 1 = ONE, 2 = ALL
    # stage 2: give every customer a random number; in each household, the smallest one is "chosen"
    draw = pl.DataFrame({"hh": inverse, "u": rng.random(len(hh))})
    first = (draw["u"] == draw.select(pl.col("u").min().over("hh")).to_series()).to_numpy()
    z = ((arm == 2) | ((arm == 1) & first)) & multi
    y = run(pop, z.astype(np.int8), "email_v2", seed=7000 + r, spillover=SPILL)["orders"].to_numpy()
    base = y[multi & (arm == 0)].mean()
    res["direct (emailed, in ONE households)"].append(y[multi & (arm == 1) & first].mean() - base)
    res["spillover (not emailed, in ONE households)"].append(y[multi & (arm == 1) & ~first].mean() - base)
    res["total (everyone, in ALL households)"].append(y[multi & (arm == 2)].mean() - base)

rate4 = pop["_rate"].to_numpy()[multi].mean() * 4                  # hidden truth, for checking only
truths = [0.05 * rate4, SPILL * 0.05 * rate4, 0.05 * rate4]          # approximate: lift of 5% for everyone
with pl.Config(fmt_str_lengths=50, tbl_width_chars=110):
    print(pl.DataFrame([{"effect": k, "average estimate": round(np.mean(v), 4), "SD": round(np.std(v), 4),
                         "approximate truth": round(t, 4)} for (k, v), t in zip(res.items(), truths)]))

python -m day28.saturation_design

│ effect                                     ┆ average estimate ┆ SD     ┆ approximate truth │
│ direct (emailed, in ONE households)        ┆ 0.0562           ┆ 0.0262 ┆ 0.057             │
│ spillover (not emailed, in ONE households) ┆ 0.0279           ┆ 0.0195 ┆ 0.0285            │
│ total (everyone, in ALL households)        ┆ 0.0553           ┆ 0.0177 ┆ 0.057             │

All three effects are recovered: the spillover is half the direct effect, as built into the lab. The price is precision: each comparison uses only a third of the households, so standard errors are large. In a real test you'd run it once, to learn the spillover, then design future tests accordingly.

28.6 Save your work with Git (15 min)

git add .
git commit -m "Day 28: interference, cluster randomisation, saturation design"
git push

Day 28 checklist

• I can explain how positive spillover and competition bias customer-level tests
• I can randomise clusters and cluster standard errors at the randomisation level
• I can compute a design effect from the ICC and cluster size
• I can design a two-stage experiment that measures direct, spillover and total effects