Staggered Adoption: What Breaks TWFE

43.1 Rollouts happen in waves (40 min)

Day 41's campaign started everywhere at once. Most real changes don't: a new store format, a pricing system or an app feature is rolled out to some stores in spring, more in summer, the rest in autumn, and some never. That's staggered adoption.

Until about 2018, everyone estimated it with Day 41's TWFE regression. Then several papers (de Chaisemartin & D'Haultfœuille 2020; Goodman-Bacon 2021; Sun & Abraham 2021; Callaway & Sant'Anna 2021; Borusyak, Jaravel & Spiess 2024) showed that TWFE can be badly wrong, even with the wrong sign, whenever the effect differs across waves or changes over time. It nearly always does.

43.2 Look at the design (50 min)

rollout() simulates NorthMart's new store format: a quarter of stores adopt at week 15, a quarter at 30, a quarter at 45, a quarter never. The effect grows by $0.4k a week after adoption, for up to 20 weeks, and it's larger for early adopters (they volunteered because they suited the format).

Create day43/rollout_overview.py:

# day43/rollout_overview.py   (python -m day43.rollout_overview)
"""
NorthMart's new store format was rolled out in three waves (weeks 15, 30, 45); a quarter of stores never
got it. Look at the design and at average sales by cohort before estimating anything.
"""
import matplotlib.pyplot as plt
import polars as pl

from causalkit.stores import public, rollout, truth

full = rollout()
df = public(full)
print(df.group_by("cohort").agg(stores=pl.col("store_id").n_unique()).sort("cohort"))
print("True effects:", truth(full))

avg = df.group_by("cohort", "week").agg(pl.col("sales").mean()).sort("week")
fig, ax = plt.subplots(figsize=(9, 4))
for c, colour in zip([0, 15, 30, 45], ["#64748b", "#2563eb", "#16a34a", "#d97706"]):
    s = avg.filter(pl.col("cohort") == c)
    ax.plot(s["week"], s["sales"], color=colour, label="never" if c == 0 else f"adopted week {c}")
    if c:
        ax.axvline(c, color=colour, ls=":", lw=1)
ax.set_xlabel("week")
ax.set_ylabel("average sales ($k)")
ax.legend()
fig.tight_layout()
fig.savefig("figures/day43_rollout.png", dpi=120)
print("Saved figures/day43_rollout.png")

python -m day43.rollout_overview

│ cohort ┆ stores │
│ 0      ┆ 60     │
│ 15     ┆ 60     │
│ 30     ┆ 60     │
│ 45     ┆ 60     │
True effects: {'ATT': 6.91, 'ATT_cohort_15': 9.522, 'ATT_cohort_30': 5.548, 'ATT_cohort_45': 2.04}
Saved figures/day43_rollout.png

The ATT here is the average effect over all treated store-weeks: $6.91k. Early adopters have been treated longest and suit the format best, so their average effect is largest.

43.3 The TWFE coefficient goes wrong (55 min)

Create day43/twfe_bias.py:

# day43/twfe_bias.py   (python -m day43.twfe_bias)
"""
The standard regression for a staggered rollout, sales ~ d | store + week, run on two versions of the
same rollout: effects that are constant over time, and effects that grow with time since adoption.
"""
import pyfixest as pf

from causalkit.stores import public, rollout, truth

for title, dynamic in [("effects constant over time", False), ("effects grow after adoption", True)]:
    full = rollout(dynamic=dynamic)
    fit = pf.feols("sales ~ d | store_id + week", public(full).to_pandas(), vcov={"CRV1": "store_id"})
    t = truth(full)
    print(f"{title:<32} TWFE {fit.coef()['d']:5.2f} (SE {fit.se()['d']:.2f})   true ATT {t['ATT']:5.2f}   "
          f"by cohort: {t['ATT_cohort_15']:.2f} / {t['ATT_cohort_30']:.2f} / {t['ATT_cohort_45']:.2f}")

python -m day43.twfe_bias

effects constant over time       TWFE  4.13 (SE 0.23)   true ATT  4.74   by cohort: 6.03 / 4.02 / 2.41
effects grow after adoption      TWFE  3.67 (SE 0.29)   true ATT  6.91   by cohort: 9.52 / 5.55 / 2.04

With effects that grow over time, TWFE reports $3.67k against a true $6.91k: less than 55% of the truth, with a standard error that makes it look precise. Even with effects constant over time (but different across waves), it's off by 13%.

43.4 Why: forbidden comparisons (40 min)

Concept. Goodman-Bacon (2021) proved that the TWFE coefficient is a weighted average of every 2×2 DiD in the data:

The last one is the "forbidden comparison". The early adopters' sales are still rising because their effect is still growing. Using them as a control subtracts that growth from the late adopters' effect. If the early adopters' effect grows fast enough, the 2×2 can be negative.

Try it. Drop the never-treated stores and rerun TWFE on adopters only. Create day43/no_never_treated.py. What happens, and why?

# day43/no_never_treated.py   (python -m day43.no_never_treated)
import polars as pl
import pyfixest as pf

from causalkit.stores import public, rollout, truth

full = rollout()
for name, d in [("all stores", full), ("adopters only", full.filter(pl.col("cohort") > 0))]:
    fit = pf.feols("sales ~ d | store_id + week", public(d).to_pandas(), vcov={"CRV1": "store_id"})
    print(f"{name:<14} TWFE {fit.coef()['d']:.2f}   true ATT {truth(d)['ATT']:.2f}")

all stores     TWFE 3.67   true ATT 6.91
adopters only  TWFE 1.54   true ATT 6.91

43.5 Day 43 mini-project: four 2×2s by hand (65 min)

Create causalkit/did.py with the _wide helper and two_by_two (the rest of the file is for Day 44):

"""Staggered difference-in-differences by hand (Days 43-44)."""
import numpy as np
import polars as pl


def _wide(df: pl.DataFrame, y: str = "sales"):
    w = df.pivot(on="week", index="store_id", values=y).sort("store_id")
    cohorts = df.group_by("store_id").agg(pl.col("cohort").first()).sort("store_id")["cohort"].to_numpy()
    weeks = np.array([int(c) for c in w.columns[1:]])
    return w.drop("store_id").to_numpy(), cohorts, weeks


def two_by_two(df: pl.DataFrame, treated_cohort: int, control_cohort: int, pre: tuple, post: tuple) -> float:
    """A single 2x2 DiD between two cohorts over two windows of weeks (inclusive ranges)."""
    Y, c, weeks = _wide(df)
    def m(cohort, win):
        cols = (weeks >= win[0]) & (weeks <= win[1])
        return Y[c == cohort][:, cols].mean()
    return (m(treated_cohort, post) - m(treated_cohort, pre)) - (m(control_cohort, post) - m(control_cohort, pre))


def att_gt(df: pl.DataFrame, control: str = "never", y: str = "sales") -> pl.DataFrame:
    """Callaway & Sant'Anna (2021) group-time effects, no covariates.
    ATT(g, t) = change in cohort g from week g-1 to week t, minus the same change for the controls:
    never-treated stores (control="never") or stores not yet treated by week t (control="notyet")."""
    Y, c, weeks = _wide(df, y)
    rows = []
    for g in sorted(set(c[c > 0])):
        base = np.flatnonzero(weeks == g - 1)[0]
        for k, t in enumerate(weeks):
            if t == g - 1:
                continue
            ctrl = (c == 0) if control == "never" else ((c == 0) | (c > max(t, g - 1)))
            if ctrl.sum() == 0:
                continue
            diff_g = Y[c == g, k] - Y[c == g, base]
            diff_c = Y[ctrl, k] - Y[ctrl, base]
            rows.append({"cohort": int(g), "week": int(t), "rel": int(t - g),
                         "att": diff_g.mean() - diff_c.mean(), "n_treated": int((c == g).sum())})
    return pl.DataFrame(rows)


def aggregate(gt: pl.DataFrame, how: str = "simple") -> pl.DataFrame | float:
    """simple: average of post-treatment ATT(g,t), weighted by cohort size (= the ATT over treated store-weeks).
    event:  average by weeks since adoption (an event study).   group: average per cohort."""
    post = gt.filter(pl.col("rel") >= 0)
    if how == "simple":
        return float((post["att"] * post["n_treated"]).sum() / post["n_treated"].sum())
    by = "rel" if how == "event" else "cohort"
    src = gt if how == "event" else post
    return (src.group_by(by).agg((pl.col("att") * pl.col("n_treated")).sum() / pl.col("n_treated").sum())
            .rename({"att": "att"}).sort(by))


def bootstrap_se(df: pl.DataFrame, estimator, reps: int = 200, seed: int = 0, **kw) -> float:
    """Store-level (cluster) bootstrap: resample whole stores with replacement."""
    rng = np.random.default_rng(seed)
    ids = df["store_id"].unique().sort().to_list()
    out = []
    for _ in range(reps):
        pick = rng.choice(ids, len(ids))
        sample = pl.concat([df.filter(pl.col("store_id") == s).with_columns(store_id=pl.lit(f"{s}_{i}"))
                            for i, s in enumerate(pick)])
        out.append(estimator(sample, **kw))
    return float(np.std(out, ddof=1))

Create day43/forbidden_comparisons.py:

# day43/forbidden_comparisons.py   (python -m day43.forbidden_comparisons)
"""
Mini-project: Goodman-Bacon (2021) showed that TWFE is a weighted average of every possible 2x2 DiD
in the data. Compute four of them by hand. The last two use stores that were ALREADY treated as the
control group: their own growing effect is subtracted from the late adopters' effect.
"""
from causalkit.did import two_by_two
from causalkit.stores import public, rollout

df = public(rollout())
comparisons = [
    ("wave 1 (wk 15) vs never, weeks 1-14 vs 15-60", 15, 0, (1, 14), (15, 60)),
    ("wave 3 (wk 45) vs never, weeks 1-44 vs 45-60", 45, 0, (1, 44), (45, 60)),
    ("wave 2 (wk 30) vs wave 1, weeks 15-29 vs 30-44", 30, 15, (15, 29), (30, 44)),
    ("wave 3 (wk 45) vs wave 1, weeks 30-44 vs 45-60", 45, 15, (30, 44), (45, 60)),
]
for name, g, c, pre, post in comparisons:
    print(f"{name:<50} {two_by_two(df, g, c, pre, post):+6.2f}")
print("\nTrue average effects: wave 1 9.52 (weeks 15-60); wave 2 3.20 (weeks 30-44); wave 3 2.04 (weeks 45-60)")

python -m day43.forbidden_comparisons

wave 1 (wk 15) vs never, weeks 1-14 vs 15-60       +10.04
wave 3 (wk 45) vs never, weeks 1-44 vs 45-60        +1.41
wave 2 (wk 30) vs wave 1, weeks 15-29 vs 30-44      -3.28
wave 3 (wk 45) vs wave 1, weeks 30-44 vs 45-60      +1.19

True average effects: wave 1 9.52 (weeks 15-60); wave 2 3.20 (weeks 30-44); wave 3 2.04 (weeks 45-60)

The third line is the headline: wave 2's true effect over weeks 30–44 is +3.20, but compared with wave 1 (whose effect was still growing by $0.6k a week), it comes out −3.28. The sign flips. The fourth is less wrong because wave 1's effect had stopped growing (capped at 20 weeks) by week 35.

The clean comparisons (lines 1–2) are close to the truth. Tomorrow's estimators use only clean comparisons.

43.6 Save your work with Git (20 min)

git add .
git commit -m "Day 43: staggered rollout, TWFE bias, forbidden comparisons"
git push

Day 43 checklist

• I can describe a staggered rollout and its cohorts
• I can show that TWFE is biased when effects differ across cohorts or grow over time
• I can explain the Goodman-Bacon decomposition and what makes a comparison "forbidden"
• I know TWFE is worst when every unit eventually adopts