Comparing Estimators

39.1 Bias, variance, and the assumption you trust (30 min)

Every estimator in Module 4 is a bet on an assumption:

Notice the last column of the "bets on" list: five estimators share one assumption. Their agreement is not evidence that it holds.

39.2 The horse race (70 min)

Every estimator, 30 datasets, both worlds, each graded against the truth for its own estimand. Create day39/horse_race.py (it takes a few minutes):

# day39/horse_race.py   (python -m day39.horse_race)
"""
Every Module 4 estimator, 30 simulated datasets, two worlds:
  A: selection on observables holds          B: hidden brand affinity drives joining and spending
Each estimate is compared with the truth for ITS estimand.
"""
import numpy as np
import polars as pl
import statsmodels.formula.api as smf

from causalkit.estimators import OUTCOME_FORMULA, aipw, g_computation, ipw, match_att, two_sls
from causalkit.loyalty import public, truth, world

ESTIMATORS = {
    # name: (function returning (estimate, se or None), estimand)
    "naive difference": (lambda df: (df.spend[df.joined == 1].mean() - df.spend[df.joined == 0].mean(), None), "ATE"),
    "OLS, flexible controls": (lambda df: (smf.ols(OUTCOME_FORMULA.replace("~", "~ joined +"), df).fit()
                                           .params["joined"], None), "ATE"),
    "g-computation": (lambda df: (g_computation(df)["ATE"], None), "ATE"),
    "matching (bias-corrected)": (lambda df: (match_att(df, on="mahalanobis", bias_correct=True)["ATT"], None), "ATT"),
    "IPW": (lambda df: (ipw(df)["ATE"], None), "ATE"),
    "AIPW": (lambda df: (lambda a: (a["ATE"], a["ATE_se"]))(aipw(df)), "ATE"),
    "2SLS with covariates": (lambda df: (lambda r: (r.params["joined"], r.std_errors["joined"]))(two_sls(df)), "LATE"),
}

rows = []
for world_label, hidden in [("A: no hidden confounder", 0.0), ("B: hidden affinity", 1.0)]:
    errs = {k: [] for k in ESTIMATORS}
    cover = {k: 0 for k in ESTIMATORS}
    for s in range(30):
        full = world(seed=5000 + s, hidden=hidden)
        df = public(full).to_pandas()
        t = truth(full)
        for name, (f, estimand) in ESTIMATORS.items():
            est, se = f(df)
            errs[name].append(est - t[estimand])
            if se is not None:
                cover[name] += abs(est - t[estimand]) <= 1.96 * se
    for name, (_, estimand) in ESTIMATORS.items():
        rows.append({"world": world_label, "estimator": name, "vs": estimand,
                     "average error": round(np.mean(errs[name]), 1), "SD": round(np.std(errs[name]), 1),
                     "95% CI coverage": f"{cover[name] / 30:.0%}" if name in ("AIPW", "2SLS with covariates") else ""})

with pl.Config(tbl_rows=-1, fmt_str_lengths=30, tbl_width_chars=120):
    print(pl.DataFrame(rows))

python -m day39.horse_race

│ world                   ┆ estimator                 ┆ vs   ┆ average error ┆ SD   ┆ 95% CI coverage │
│ A: no hidden confounder ┆ naive difference          ┆ ATE  ┆ 426.5         ┆ 8.9  ┆                 │
│ A: no hidden confounder ┆ OLS, flexible controls    ┆ ATE  ┆ -5.8          ┆ 2.5  ┆                 │
│ A: no hidden confounder ┆ g-computation             ┆ ATE  ┆ -0.2          ┆ 2.0  ┆                 │
│ A: no hidden confounder ┆ matching (bias-corrected) ┆ ATT  ┆ -0.0          ┆ 2.3  ┆                 │
│ A: no hidden confounder ┆ IPW                       ┆ ATE  ┆ -7.0          ┆ 21.8 ┆                 │
│ A: no hidden confounder ┆ AIPW                      ┆ ATE  ┆ -0.2          ┆ 2.1  ┆ 97%             │
│ A: no hidden confounder ┆ 2SLS with covariates      ┆ LATE ┆ 3.0           ┆ 10.6 ┆ 93%             │
│ B: hidden affinity      ┆ naive difference          ┆ ATE  ┆ 443.4         ┆ 8.2  ┆                 │
│ B: hidden affinity      ┆ OLS, flexible controls    ┆ ATE  ┆ 67.8          ┆ 2.1  ┆                 │
│ B: hidden affinity      ┆ g-computation             ┆ ATE  ┆ 72.4          ┆ 1.7  ┆                 │
│ B: hidden affinity      ┆ matching (bias-corrected) ┆ ATT  ┆ 73.3          ┆ 2.2  ┆                 │
│ B: hidden affinity      ┆ IPW                       ┆ ATE  ┆ 69.7          ┆ 20.1 ┆                 │
│ B: hidden affinity      ┆ AIPW                      ┆ ATE  ┆ 73.5          ┆ 1.8  ┆ 0%              │
│ B: hidden affinity      ┆ 2SLS with covariates      ┆ LATE ┆ 0.8           ┆ 13.9 ┆ 100%            │

39.3 Reading the results (55 min)

World A. G-computation, bias-corrected matching and AIPW are all unbiased with SDs around $2. IPW is unbiased-ish but ten times noisier. OLS with flexible controls is $6 off: its coefficient targets a regression-weighted average, not the ATE. 2SLS is right, but five times noisier than AIPW.

World B. Every selection-on-observables method is off by $68–74, and they agree with each other to within a few dollars. AIPW's 95% interval covered the truth in 0 of 30 datasets. 2SLS is the only method that works.

Three takeaways:

1. Agreement between estimators that share an assumption is not robustness. Five methods agreed in world B and all were wrong.
2. Precision is not accuracy. The most precise estimators in world B had the worst coverage.
3. The choice of strategy matters more than the choice of estimator within it. Within selection on observables, the differences are a few dollars; between strategies, $70.

39.4 A decision guide (30 min)

Write this table into docs/ESTIMATOR_GUIDE.md in your own words, with one NorthMart example per row.

39.5 Day 39 mini-project: how much hidden confounding? (65 min)

World B has hidden=1.0. Real confounding is rarely all or nothing. Run AIPW and 2SLS (with covariates) at hidden = 0, 0.25, 0.5 and 1.0, ten datasets each (world(seed=6000 + s, hidden=h)), and print the average error of each. Create day39/how_much_hidden.py.

# day39/how_much_hidden.py   (python -m day39.how_much_hidden)
import numpy as np

from causalkit.estimators import aipw, two_sls
from causalkit.loyalty import public, truth, world

for hidden in [0.0, 0.25, 0.5, 1.0]:
    e_a, e_iv = [], []
    for s in range(10):
        full = world(seed=6000 + s, hidden=hidden)
        df, t = public(full).to_pandas(), truth(full)
        e_a.append(aipw(df)["ATE"] - t["ATE"])
        e_iv.append(two_sls(df).params["joined"] - t["LATE"])
    print(f"hidden = {hidden:4.2f}: AIPW average error {np.mean(e_a):+6.1f}   2SLS average error {np.mean(e_iv):+6.1f}")

hidden = 0.00: AIPW average error   -0.5   2SLS average error   -2.6
hidden = 0.25: AIPW average error   +4.6   2SLS average error   -2.6
hidden = 0.50: AIPW average error  +20.4   2SLS average error   -2.7
hidden = 1.00: AIPW average error  +73.8   2SLS average error   -2.3

39.6 Save your work with Git (20 min)

git add .
git commit -m "Day 39: estimator horse race in two worlds, estimator guide"
git push

Day 39 checklist

• I can name the assumption each Module 4 estimator bets on, and its estimand
• I can explain why agreement among estimators that share an assumption isn't robustness
• I can choose a strategy from the situation, using a written decision guide
• I can show how bias grows with the strength of a hidden confounder