Modern DiD Estimators

44.1 The fix: only clean comparisons (30 min)

All the modern estimators share one idea: never use an already-treated unit as a control. They differ in how they build the comparison:

44.2 Callaway & Sant'Anna by hand (60 min)

Concept. For cohort g (stores that adopted in week g) and any week t, compare their change from week g−1 (the last week before adoption) to week t with the same change for never-treated stores:

ATT(g, t) = [mean sales of cohort g in week t − in week g−1] − [the same for never-treated stores]

Each ATT(g,t) is a clean 2×2. Then aggregate them for the question at hand: one overall ATT, one per cohort, or one per week since adoption.

Type att_gt, aggregate and bootstrap_se into causalkit/did.py. Create day44/callaway_santanna.py:

# day44/callaway_santanna.py   (python -m day44.callaway_santanna)
"""
Callaway & Sant'Anna (2021): never use an already-treated store as a control.
Estimate one clean 2x2 for every (cohort, week), then average them in whichever way the question needs.
"""
import polars as pl

from causalkit.did import aggregate, att_gt, bootstrap_se
from causalkit.stores import public, rollout, truth

full = rollout()
df = public(full)
t = truth(full)

gt = att_gt(df, control="never")
print(f"{gt.height} group-time effects, e.g.:")
print(gt.filter(pl.col("rel").is_in([-5, 0, 5, 10])).sort("cohort", "rel").with_columns(pl.col("att").round(2)))

simple = aggregate(gt, "simple")
se = bootstrap_se(df, lambda d: aggregate(att_gt(d), "simple"), reps=100)
print(f"\nOverall ATT: {simple:.2f} (bootstrap SE {se:.2f})   truth {t['ATT']:.2f}")
print("By cohort:")
by_cohort = aggregate(gt, "group").with_columns(
    pl.col("att").round(2),
    truth=pl.col("cohort").map_elements(lambda c: t[f"ATT_cohort_{c}"], return_dtype=pl.Float64).round(2))
print(by_cohort)

python -m day44.callaway_santanna

177 group-time effects, e.g.:
│ cohort ┆ week ┆ rel ┆ att   ┆ n_treated │
│ 15     ┆ 10   ┆ -5  ┆ 0.33  ┆ 60        │
│ 15     ┆ 15   ┆ 0   ┆ 0.72  ┆ 60        │
│ 15     ┆ 20   ┆ 5   ┆ 4.28  ┆ 60        │
│ 15     ┆ 25   ┆ 10  ┆ 6.06  ┆ 60        │
...
Overall ATT: 6.97 (bootstrap SE 0.42)   truth 6.91
By cohort:
│ cohort ┆ att  ┆ truth │
│ 15     ┆ 9.63 ┆ 9.52  │
│ 30     ┆ 5.59 ┆ 5.55  │
│ 45     ┆ 2.01 ┆ 2.04  │

$6.97k against a true $6.91k, where TWFE said $3.67k. Every cohort is right too. The negative rel rows are pre-period placebo comparisons: a built-in pre-trend check.

The bootstrap resamples whole stores (Day 41: cluster at the level of treatment assignment). It takes a minute.

44.3 The imputation estimator (50 min)

Concept. Fit the TWFE model without the treatment, using only untreated store-weeks (never-treated stores, and adopters before they adopted). That model has never seen an effect, so it predicts each treated store-week's sales without the new format. The effect is actual minus predicted, averaged.

Create day44/imputation.py:

# day44/imputation.py   (python -m day44.imputation)
"""
The imputation idea (Borusyak, Jaravel & Spiess 2024; Gardner's two-stage DiD is the same estimate):
1. fit store and week effects on UNTREATED store-weeks only (never-treated, and adopters before adoption);
2. predict every treated store-week's sales without the new format;
3. the effect is actual minus predicted, averaged over treated store-weeks.
By hand, then with pyfixest's did2s (which also gives a correct standard error).
"""
import pyfixest as pf

from causalkit.stores import public, rollout, truth

full = rollout()
df = public(full).to_pandas()

untreated = df[df.d == 0]
stage1 = pf.feols("sales ~ 1 | store_id + week", untreated)
treated = df[df.d == 1].copy()
treated["predicted_y0"] = stage1.predict(newdata=treated)
by_hand = (treated.sales - treated.predicted_y0).mean()

gardner = pf.did2s(df.assign(d=df.d.astype(float)), yname="sales", first_stage="~ 0 | store_id + week",
                   second_stage="~ 0 + d", treatment="d", cluster="store_id")
print(f"Imputation by hand:   {by_hand:.2f}")
print(f"pyfixest did2s:       {gardner.coef()['d']:.2f}  (SE {gardner.se()['d']:.2f})")
print(f"Truth:                {truth(full)['ATT']:.2f}")

python -m day44.imputation

Imputation by hand:   7.02
pyfixest did2s:       7.02  (SE 0.33)
Truth:                6.91

Same number by hand and from the library; the library adds a standard error that accounts for the first stage having been estimated. Imputation uses more of the pre-period than Callaway & Sant'Anna (which relies on week g−1 alone as the baseline), so it is usually more precise: SE 0.33 vs 0.42 here.

44.4 Never-treated or not-yet-treated controls? (45 min)

Callaway & Sant'Anna can use either never-treated stores, or every store not yet treated by week t. Not-yet-treated gives more controls (more precision), but assumes that stores' timing of adoption is unrelated to their trends, and that nobody changed behaviour in anticipation.

Try it. Compare att_gt(df, control="never") and control="notyet". Create day44/not_yet_treated.py.

# day44/not_yet_treated.py   (python -m day44.not_yet_treated)
from causalkit.did import aggregate, att_gt
from causalkit.stores import public, rollout, truth

full = rollout()
df = public(full)
for control in ["never", "notyet"]:
    gt = att_gt(df, control=control)
    print(f"controls = {control:<7} ATT {aggregate(gt):.2f}   by cohort "
          f"{[round(x, 2) for x in aggregate(gt, 'group')['att']]}")
print(f"truth ATT {truth(full)['ATT']:.2f}")

controls = never   ATT 6.97   by cohort [9.63, 5.59, 2.01]
controls = notyet  ATT 6.77   by cohort [9.27, 5.52, 2.01]
truth ATT 6.91

44.5 Day 44 mini-project: event studies, done right (65 min)

How does the effect evolve with weeks since adoption? Compare three event studies with the truth. Create day44/event_study_modern.py:

# day44/event_study_modern.py   (python -m day44.event_study_modern)
"""
Mini-project: how does the effect evolve with weeks since adoption? Three event studies against the truth:
  TWFE with relative-week dummies (the common, contaminated version),
  Callaway & Sant'Anna aggregated by event time,
  pyfixest's saturated (Sun & Abraham-style) event study, which interacts event time with cohort.
"""
import matplotlib.pyplot as plt
import numpy as np
import polars as pl
import pyfixest as pf

from causalkit.did import aggregate, att_gt
from causalkit.stores import public, rollout

full = rollout()
df = public(full)
pdf = df.to_pandas()

true_es = (full.filter(pl.col("cohort") > 0).with_columns(rel=pl.col("week") - pl.col("cohort"))
           .group_by("rel").agg(truth=pl.col("_tau").mean()))

pdf["rel"] = np.where(pdf.cohort > 0, pdf.week - pdf.cohort, -1)       # never-treated sit at the reference
twfe = pf.feols("sales ~ i(rel, ref=-1) | store_id + week", pdf, vcov={"CRV1": "store_id"}).tidy()
twfe_es = pl.DataFrame({"rel": [int(float(n.split("::")[1])) for n in twfe.index],
                        "TWFE dummies": twfe.Estimate.to_numpy()})

cs_es = aggregate(att_gt(df), "event").rename({"att": "Callaway-Sant'Anna"})

sat = pf.event_study(pdf, "sales", "store_id", "week", "cohort", estimator="saturated", cluster="store_id")
agg = sat.aggregate()
sat_es = pl.DataFrame({"rel": agg.index.astype(float).astype(int).to_numpy(), "saturated": agg.Estimate.astype(float).to_numpy()})

es = (true_es.join(twfe_es, on="rel", how="left").join(cs_es, on="rel", how="left")
      .join(sat_es, on="rel", how="left").sort("rel"))
show = es.filter(pl.col("rel").is_in([-10, -5, 0, 5, 10, 20, 30, 40])).with_columns(pl.exclude("rel").round(2))
with pl.Config(tbl_width_chars=120):
    print(show)

fig, ax = plt.subplots(figsize=(9, 4))
for col, style in [("truth", "k-"), ("TWFE dummies", "o"), ("Callaway-Sant'Anna", "s"), ("saturated", "^")]:
    s = es.filter(pl.col("rel").is_between(-14, 44))
    ax.plot(s["rel"], s[col], style, ms=3, label=col)
ax.axvline(-0.5, color="grey", ls="--", lw=0.8)
ax.set_xlabel("weeks since adoption")
ax.set_ylabel("effect on weekly sales ($k)")
ax.legend()
fig.tight_layout()
fig.savefig("figures/day44_event_studies.png", dpi=120)
print("Saved figures/day44_event_studies.png")

python -m day44.event_study_modern

│ rel ┆ truth ┆ TWFE dummies ┆ Callaway-Sant'Anna ┆ saturated │
│ -10 ┆ 0.0   ┆ -0.16        ┆ -0.41              ┆ -0.41     │
│ -5  ┆ 0.0   ┆ 0.48         ┆ 0.42               ┆ 0.42      │
│ 0   ┆ 0.41  ┆ 0.72         ┆ 0.71               ┆ 0.71      │
│ 5   ┆ 2.48  ┆ 3.06         ┆ 2.96               ┆ 2.96      │
│ 10  ┆ 4.55  ┆ 4.55         ┆ 4.2                ┆ 4.2       │
│ 20  ┆ 10.0  ┆ 10.64        ┆ 10.69              ┆ 10.69     │
│ 30  ┆ 10.0  ┆ 10.49        ┆ 9.96               ┆ 9.96      │
│ 40  ┆ 12.0  ┆ 12.07        ┆ 11.67              ┆ 11.67     │
Saved figures/day44_event_studies.png

Three observations:

1. Callaway & Sant'Anna and the saturated estimator give identical event studies here: with never-treated controls and no covariates, they are the same estimator written two ways.
2. The TWFE event study with a full set of relative-week dummies is also close in this simulation. Most of TWFE's damage came from collapsing everything into one coefficient (Day 43). But Sun & Abraham proved that the dummy version can still be contaminated when cohorts' effect paths differ, and nothing in the output tells you whether that has happened. Use the clean estimators by default.
3. The effect builds up over 20 weeks. A study that stopped 5 weeks after adoption would report a third of the eventual effect.

Write in day44/NOTES.md: for a rollout you know at work (or a plausible one), which estimator would you use, which controls, and what's the estimand?

44.6 Save your work with Git (20 min)

git add .
git commit -m "Day 44: Callaway-Sant'Anna, imputation (did2s), modern event studies"
git push

Coming next (Days 45–47). DiD needs many treated units, or at least a control group on a parallel path. What if only one region ran the campaign, and it's unlike the average region? Synthetic control builds a better comparison from a weighted mix of regions.

Day 44 checklist

• I can estimate group-time ATTs by hand and aggregate them overall, by cohort and by event time
• I can explain and run the imputation estimator (did2s)
• I can choose between never-treated and not-yet-treated controls
• I use clean event studies, and know why the TWFE-dummy version is risky