Placebo and Falsification Tests

49.1 Try to break your own design (30 min)

Every quasi-experiment rests on an assumption the data can't prove. What the data can do is fail tests that the assumption implies. A placebo test applies the design where the true effect is known to be zero:

A design that "finds" effects where there are none can't be trusted where there might be one. Run placebos before you look at the real estimate, and report them all, not just the ones that pass.

49.2 Placebo dates for DiD (45 min)

Create day49/did_placebos.py:

# day49/did_placebos.py   (python -m day49.did_placebos)
"""
Placebo-in-time for DiD: keep only the weeks BEFORE the real campaign and pretend it started at week 25.
Nothing happened at week 25, so the estimate should be about 0. A 'placebo effect' means the design
would also have found an effect where there was none.
"""
import pyfixest as pf

from causalkit.stores import campaign, public

for title, gap in [("parallel trends", 0.0), ("treated stores growing faster", 0.15)]:
    df = public(campaign(trend_gap=gap)).to_pandas()
    pre = df[df.week < 40].copy()
    pre["fake_d"] = pre.treated_group * (pre.week >= 25)
    fake = pf.feols("sales ~ fake_d | store_id + week", pre, vcov={"CRV1": "store_id"})
    real = pf.feols("sales ~ d | store_id + week", df, vcov={"CRV1": "store_id"})
    print(f"{title:<30} placebo (week 25): {fake.coef()['fake_d']:+5.2f} (p = {fake.pvalue()['fake_d']:.3f})   "
          f"real estimate: {real.coef()['d']:.2f}")

python -m day49.did_placebos

parallel trends                placebo (week 25): -0.37 (p = 0.280)   real estimate: 2.74
treated stores growing faster  placebo (week 25): +2.55 (p = 0.000)   real estimate: 7.24

With parallel trends, the fake campaign at week 25 finds nothing. With the trend gap, it finds a highly significant "effect" of $2.55k, from a campaign that didn't exist. The placebo uses only pre-campaign data, so it can be run before the real campaign is analysed, or even before it ends.

49.3 Placebo regions for synthetic control (70 min)

Concept. Synthetic control has no standard errors. Abadie, Diamond & Hainmueller (2010) proposed permutation inference: pretend each of the 29 untreated regions ran the campaign, build its synthetic control, and compare. The statistic is the ratio of post-period to pre-period RMSPE, which rewards a big post-period gap but discounts regions that were never fitted well. If R07's ratio is the largest of 30, the probability of that by chance is 1/30.

Type placebo_in_space into synth.py. Create day49/sc_placebos.py:

# day49/sc_placebos.py   (python -m day49.sc_placebos)
"""
Synthetic control has no standard errors; inference comes from placebos (Abadie et al., 2010).
In space: pretend each of the other 29 regions ran the campaign. If R07's post/pre misfit ratio is the
largest of 30, the chance of that by luck is 1/30.
In time: pretend R07's campaign started at week 60, using only data before week 80.
"""
import matplotlib.pyplot as plt
import polars as pl

from causalkit.regions import public, regions
from causalkit.synth import placebo_in_space, synthetic_control

df = public(regions())
ranks = placebo_in_space(df, "R07", 80)
print(ranks.head(6).with_columns(pl.col("effect", "rmspe_pre", "ratio").round(2)))
rank = ranks["region"].to_list().index("R07") + 1
print(f"R07 ranks {rank} of {ranks.height}: permutation p-value = {rank}/{ranks.height} = {rank / ranks.height:.3f}")

pre_only = df.filter(pl.col("week") < 80)
fake = synthetic_control(pre_only, "R07", 60)
print(f"\nPlacebo in time (fake start week 60): effect {fake['effect']:+.1f} $k per week, "
      f"post/pre ratio {fake['ratio']:.2f}")

fig, ax = plt.subplots(figsize=(9, 4))
for r in ranks["region"]:
    sub = df if r == "R07" else df.filter(pl.col("region") != "R07")
    sc = synthetic_control(sub, r, 80)
    if r == "R07":
        ax.plot(sc["weeks"], sc["gap"], color="#dc2626", lw=2, label="R07", zorder=3)
    elif sc["rmspe_pre"] < 3 * ranks.filter(pl.col("region") == "R07")["rmspe_pre"][0]:
        ax.plot(sc["weeks"], sc["gap"], color="#94a3b8", lw=0.7)
ax.axvline(80, color="grey", ls="--", lw=0.8)
ax.set_xlabel("week")
ax.set_ylabel("actual minus synthetic ($k)")
ax.set_title("Placebo gaps (regions with a pre-period fit at most 3x worse than R07's)")
ax.legend()
fig.tight_layout()
fig.savefig("figures/day49_sc_placebos.png", dpi=120)
print("Saved figures/day49_sc_placebos.png")

python -m day49.sc_placebos

│ region ┆ is_treated ┆ effect ┆ rmspe_pre ┆ ratio │
│ R07    ┆ true       ┆ 17.33  ┆ 3.86      ┆ 5.41  │
│ R18    ┆ false      ┆ 21.6   ┆ 6.62      ┆ 3.36  │
│ R14    ┆ false      ┆ 17.6   ┆ 6.21      ┆ 3.11  │
│ R04    ┆ false      ┆ -7.83  ┆ 4.54      ┆ 1.9   │
│ R17    ┆ false      ┆ -6.78  ┆ 4.36      ┆ 1.84  │
│ R01    ┆ false      ┆ -29.99 ┆ 17.71     ┆ 1.74  │
R07 ranks 1 of 30: permutation p-value = 1/30 = 0.033

Placebo in time (fake start week 60): effect +5.5 $k per week, post/pre ratio 1.90
Saved figures/day49_sc_placebos.png

Read it carefully:

• R07 ranks first: p = 0.033. With 30 regions, that's the smallest p-value possible. Permutation inference needs many units.
• R18 and R14 show placebo "effects" of $17–22k, as big as R07's, from no campaign at all. They are among the most urban regions, so they are hard to match from the remaining donors (R07 is excluded from every placebo's donor pool): their pre-period RMSPE is 60–70% worse than R07's and their post-period drifts. That's why the ratio, not the raw effect, is the test statistic: it discounts gaps that come from a poor fit.
• The placebo in time finds $5.5k: not zero, but a third of the real estimate, with a ratio of 1.9 against R07's 5.4.

Open the figure: R07's gap rises far above all placebo gaps at week 80 (regions with poor pre-fit are hidden, as Abadie et al. recommend).

49.4 Manipulation and the donut (40 min)

Concept. RD's key assumption is that customers can't precisely control which side of the cutoff they land on. If customers know the Gold rule, some at $950 will buy a little more in December to reach $1,000. Those who do are the keen ones, so the customers just above the cutoff are no longer comparable with those just below.

customers(manipulation=0.3) simulates it: 30% of customers with organic spending of $850–$1,000 top up to just over $1,000.

Try it. In the manipulated data, estimate the RD normally, then with a donut: drop customers within $20, then $50, of the cutoff. Create day49/donut.py.

# day49/donut.py   (python -m day49.donut)
from rdrobust import rdrobust

from causalkit.tiers import CUTOFF, customers, public

df = public(customers(manipulation=0.3)).to_pandas()
for donut in [0, 20, 50]:
    d = df[(df.pre_spend - CUTOFF).abs() >= donut]
    rd = rdrobust(d.spend, d.pre_spend, c=CUTOFF)
    print(f"donut ${donut:>2}: estimate {rd.coef.iloc[0, 0]:6.1f}, robust CI [{rd.ci.iloc[2, 0]:.1f}, {rd.ci.iloc[2, 1]:.1f}]")

donut $ 0: estimate  230.8, robust CI [211.2, 262.1]
donut $20: estimate  282.6, robust CI [244.7, 347.5]
donut $50: estimate  159.5, robust CI [110.0, 221.9]

49.5 Day 49 mini-project: the RD falsification battery (65 min)

Three checks every RD needs, on clean and manipulated data. Create day49/rd_falsification.py:

# day49/rd_falsification.py   (python -m day49.rd_falsification)
"""
Mini-project: three checks every RD needs, run on two datasets:
  clean, and one where customers just below $1,000 who knew the rule bought their way over it.
  1. Density: is there a suspicious pile-up just above the cutoff? (Cattaneo, Jansson & Ma: rddensity)
  2. Covariates: do pre-determined traits jump at the cutoff? They shouldn't.
  3. Placebo cutoffs: is there a 'jump' where no rule exists?
"""
import numpy as np
import polars as pl
from rddensity import rddensity
from rdrobust import rdrobust

from causalkit.tiers import CUTOFF, customers, public, truth

for title, m in [("clean", 0.0), ("customers top up to reach Gold", 0.3)]:
    full = customers(manipulation=m)
    df = public(full).to_pandas()
    rd = rdrobust(df.spend, df.pre_spend, c=CUTOFF)
    dens = rddensity(X=df.pre_spend.to_numpy(), c=CUTOFF)
    print(f"=== {title}")
    print(f"RD estimate {rd.coef.iloc[0, 0]:.1f}, robust CI [{rd.ci.iloc[2, 0]:.1f}, {rd.ci.iloc[2, 1]:.1f}]"
          f"   (truth {truth(full)['effect_at_cutoff']:.1f})")
    print(f"1. density test p-value: {dens.test['p_jk']:.3f}")
    rows = []
    for cov in ["age", "tenure_years", "engaged"]:
        r = rdrobust(df[cov], df.pre_spend, c=CUTOFF)
        rows.append({"check": f"2. jump in {cov}", "estimate": round(r.coef.iloc[0, 0], 3),
                     "robust p": round(r.pv.iloc[2, 0], 3)})
    for c in [700, 850, 1150, 1300]:
        side = df[df.pre_spend < CUTOFF] if c < CUTOFF else df[df.pre_spend >= CUTOFF]
        r = rdrobust(side.spend, side.pre_spend, c=c)
        rows.append({"check": f"3. placebo cutoff ${c}", "estimate": round(r.coef.iloc[0, 0], 1),
                     "robust p": round(r.pv.iloc[2, 0], 3)})
    with pl.Config(tbl_width_chars=100, fmt_str_lengths=40):
        print(pl.DataFrame(rows))

python -m day49.rd_falsification

=== clean
RD estimate 123.9, robust CI [101.9, 145.0]   (truth 120.0)
1. density test p-value: 0.502
│ 2. jump in age          ┆ 0.725    ┆ 0.227    │
│ 2. jump in tenure_years ┆ 0.061    ┆ 0.493    │
│ 2. jump in engaged      ┆ 0.039    ┆ 0.092    │
│ 3. placebo cutoff $700  ┆ -14.5    ┆ 0.283    │
│ 3. placebo cutoff $850  ┆ -10.9    ┆ 0.87     │
│ 3. placebo cutoff $1150 ┆ 4.6      ┆ 0.798    │
│ 3. placebo cutoff $1300 ┆ -6.2     ┆ 0.744    │
=== customers top up to reach Gold
RD estimate 230.8, robust CI [211.2, 262.1]   (truth 120.0)
1. density test p-value: 0.000
│ 2. jump in age          ┆ 0.921    ┆ 0.099    │
│ 2. jump in tenure_years ┆ 0.025    ┆ 0.739    │
│ 2. jump in engaged      ┆ 0.008    ┆ 0.717    │
│ 3. placebo cutoff $700  ┆ -27.1    ┆ 0.063    │
│ 3. placebo cutoff $850  ┆ 4.3      ┆ 0.766    │
│ 3. placebo cutoff $1150 ┆ 19.2     ┆ 0.407    │
│ 3. placebo cutoff $1300 ┆ 13.8     ┆ 0.536    │

1. Density (Cattaneo, Jansson & Ma, 2020): in clean data, the number of customers changes smoothly across $1,000 (p = 0.50). With manipulation, the pile-up just above the cutoff is unmistakable (p < 0.001). This is the check that catches it.
2. Covariates: no jumps in either dataset. The keen customers who moved differ in intent, which isn't in the data. Covariate checks only catch manipulation that shows up in measured traits; passing them doesn't clear the design.
3. Placebo cutoffs: no jumps where no rule exists, in either dataset. Each placebo uses customers from one side of the real cutoff only, so the real jump can't leak in.

The clean design passes everything; the manipulated one fails only the density test, and that's enough. Write a short "Falsification" section in day49/NOTES.md with the three results for each dataset and what you'd conclude.

49.6 Save your work with Git (20 min)

git add .
git commit -m "Day 49: placebo dates, placebo regions, density test, donut RD, covariate checks"
git push

Day 49 checklist

• I run placebo tests before looking at the real estimate, and report all of them
• I can run a placebo-in-time test for DiD
• I can run permutation inference for synthetic control with the RMSPE ratio, and know its minimum p-value
• I can run a density test, covariate continuity checks and placebo cutoffs for an RD
• I know that a donut RD is a diagnostic, not a cure for manipulation