Sensitivity to Hidden Confounding

61.1 What a hidden confounder does, in one formula (40 min)

The coupon world of Day 54: NorthMart's targeting model also used a real-time browsing signal that was never stored. coupons(hidden=0.5) builds it in. Because it's a simulator, we can run the regression the analyst can't.

Create day61/ovb_by_hand.py:

# day61/ovb_by_hand.py   (python -m day61.ovb_by_hand)
"""
The omitted-variable bias formula, with a confounder we can see because we built the simulator.
NorthMart's targeting model also used a real-time browsing signal that was never stored (hidden=0.5).
  short regression: spend on coupon + features            (what the analyst can run)
  long regression:  spend on coupon + features + browsing (what the analyst wishes they could run)
OVB = (effect of browsing on spend) x (difference in browsing between couponed and other customers, given X).
"""
import statsmodels.formula.api as smf

from causalkit.coupons import X_COLS, coupons

df = coupons(hidden=0.5).to_pandas()
controls = " + ".join(X_COLS)
short = smf.ols(f"spend ~ coupon + {controls}", df).fit()
long = smf.ols(f"spend ~ coupon + _browsing + {controls}", df).fit()
imbalance = smf.ols(f"_browsing ~ coupon + {controls}", df).fit()

gamma, delta = long.params["_browsing"], imbalance.params["coupon"]
print(f"short regression (browsing omitted):  {short.params['coupon']:6.2f}")
print(f"long regression (browsing included):  {long.params['coupon']:6.2f}")
print(f"difference:                           {short.params['coupon'] - long.params['coupon']:6.2f}")
print(f"OVB formula: gamma x delta = {gamma:.2f} x {delta:.3f} = {gamma * delta:.2f}")
print("   gamma: extra $ of spend per 1 SD of browsing;  delta: how much more browsing couponed customers did")

python -m day61.ovb_by_hand

short regression (browsing omitted):   21.61
long regression (browsing included):   13.34
difference:                             8.28
OVB formula: gamma x delta = 14.85 x 0.557 = 8.28
   gamma: extra $ of spend per 1 SD of browsing;  delta: how much more browsing couponed customers did

The omitted-variable bias formula is exact: bias = (how much the confounder moves the outcome) × (how unequal it is between treated and untreated, given the controls). A confounder that does only one of the two does no harm. Every sensitivity method in this module is a way of putting plausible numbers on those two factors without seeing the confounder.

(The long regression's $13.34 is below the true $16.51 because linear controls can't capture the confounding by the observed features; that's Day 51's lesson, separate from today's.)

61.2 Sensitivity analysis with DoubleML (50 min)

Concept. Chernozhukov, Cinelli, Newey, Sharma & Syrgkanis (2022, "Long story short") extend the OVB formula to DML. A hidden confounder is described by two numbers, both shares between 0 and 1:

• cf_y: how much of the outcome's remaining variation it would explain;
• cf_d: roughly, how much of the treatment's remaining variation it would explain.

(A third parameter, ρ, is the correlation between the two; ρ = 1 is the worst case.) For any (cf_y, cf_d), DoubleML returns the range the true effect could lie in. The robustness value (RV) is the smallest equal strength (cf_y = cf_d) that could move the estimate to a chosen threshold.

Create day61/dml_sensitivity.py. Its fit_irm function is reused tomorrow:

# day61/dml_sensitivity.py   (python -m day61.dml_sensitivity)
"""
In real data you can't run the long regression. Sensitivity analysis asks instead: how strong would a hidden
confounder have to be to change the conclusion? DoubleML implements Chernozhukov, Cinelli, Newey, Sharma &
Syrgkanis (2022): a confounder's strength is described by
    cf_y: share of the outcome's remaining variation it explains,
    cf_d: how much it changes the treatment's (Riesz representer's) variation, roughly the same idea for the coupon,
and the bounds say how far the estimate could move.
The business threshold: a coupon pays for itself if it raises spend by more than $16.67 (Day 65).
"""
import doubleml as dml
import matplotlib.pyplot as plt
import numpy as np

from causalkit.coupons import X_COLS, coupons, public, truth
from causalkit.learners import clf, reg

np.random.seed(61)  # DoubleML draws its cross-fitting folds from numpy's global RNG
BREAK_EVEN = 16.67


def fit_irm(df):
    data = dml.DoubleMLData(df[X_COLS + ["coupon", "spend"]], "spend", "coupon", X_COLS)
    return dml.DoubleMLIRM(data, ml_g=reg(), ml_m=clf(), n_folds=5, trimming_threshold=0.02).fit()


if __name__ == "__main__":
    full = coupons(hidden=0.5)
    irm = fit_irm(public(full).to_pandas())
    print(f"IRM ATE: {irm.coef[0]:.2f} (SE {irm.se[0]:.2f})   truth {truth(full)['ATE']:.2f}")
    for cf in [0.01, 0.03, 0.05, 0.10]:
        irm.sensitivity_analysis(cf_y=cf, cf_d=cf, rho=1.0)
        p = irm.sensitivity_params
        print(f"confounder with cf_y = cf_d = {cf:.2f}: estimate between {p['theta']['lower'][0]:6.2f} and "
              f"{p['theta']['upper'][0]:6.2f}  (CI {p['ci']['lower'][0]:6.2f} to {p['ci']['upper'][0]:6.2f})")
    irm.sensitivity_analysis(null_hypothesis=0.0)
    rv0 = irm.sensitivity_params["rv"][0]
    irm.sensitivity_analysis(null_hypothesis=BREAK_EVEN)
    rv_be = irm.sensitivity_params["rv"][0]
    print(f"Robustness value, effect = 0:            {rv0:.1%}")
    print(f"Robustness value, effect = break-even:   {rv_be:.1%}")

    grid = np.linspace(0.0, 0.2, 21)
    lower = np.zeros((len(grid), len(grid)))
    for i, cy in enumerate(grid):
        for j, cd in enumerate(grid):
            irm.sensitivity_analysis(cf_y=cy, cf_d=min(cd, 0.99), rho=1.0)
            lower[i, j] = irm.sensitivity_params["theta"]["lower"][0]
    fig, ax = plt.subplots(figsize=(6, 5))
    cs = ax.contour(grid, grid, lower, levels=[0, 8, BREAK_EVEN, 20, 22, 24], colors="#64748b")
    ax.clabel(cs, fmt="%.1f")
    ax.contour(grid, grid, lower, levels=[BREAK_EVEN], colors="#dc2626", linewidths=2)
    ax.set_xlabel("cf_d (strength with the coupon)")
    ax.set_ylabel("cf_y (strength with spend)")
    ax.set_title("Lower bound of the coupon effect ($)")
    fig.tight_layout()
    fig.savefig("figures/day61_sensitivity_contour.png", dpi=120)
    print("Saved figures/day61_sensitivity_contour.png")

python -m day61.dml_sensitivity

IRM ATE: 24.61 (SE 0.64)   truth 16.51
confounder with cf_y = cf_d = 0.01: estimate between  23.74 and  25.47  (CI  22.69 to  26.53)
confounder with cf_y = cf_d = 0.03: estimate between  21.97 and  27.24  (CI  20.92 to  28.29)
confounder with cf_y = cf_d = 0.05: estimate between  20.17 and  29.04  (CI  19.12 to  30.10)
confounder with cf_y = cf_d = 0.10: estimate between  15.49 and  33.72  (CI  14.44 to  34.78)
Robustness value, effect = 0:            24.7%
Robustness value, effect = break-even:   8.8%
Saved figures/day61_sensitivity_contour.png

Two very different statements from the same estimate:

• "Coupons raise spending": robust. A confounder would need 25% strength on both sides to explain it away entirely.
• "Coupons pay for themselves": fragile. A confounder of 8.8% strength could push the effect below $16.67.

The robustness value depends on the question. Always compute it against the threshold the decision depends on, not against zero.

61.3 Reading the contour plot (45 min)

Open figures/day61_sensitivity_contour.png. Each point is a hypothetical confounder (strength with the coupon on the x-axis, with spend on the y-axis); the lines show the lowest plausible effect. The red line is the break-even: any confounder above and to the right of it could make the coupon unprofitable.

The plot turns an argument nobody can win ("there might be confounders") into one people can have ("is there plausibly anything as strong as this, that we didn't measure?"). That's a question for the people who built the targeting model, not for the data.

61.4 Robustness values say nothing about whether confounding exists (40 min)

Try it. Rerun the sensitivity analysis on coupons(hidden=0.0), a world with no hidden confounder. What's the robustness value for break-even? Create day61/no_hidden.py.

# day61/no_hidden.py   (python -m day61.no_hidden)
import numpy as np

from causalkit.coupons import coupons, public, truth
from day61.dml_sensitivity import BREAK_EVEN, fit_irm

np.random.seed(61)
full = coupons(hidden=0.0)
irm = fit_irm(public(full).to_pandas())
irm.sensitivity_analysis(null_hypothesis=BREAK_EVEN)
print(f"No hidden confounder: ATE {irm.coef[0]:.2f} (truth {truth(full)['ATE']:.2f}); "
      f"robustness value for break-even {irm.sensitivity_params['rv'][0]:.1%}")

No hidden confounder: ATE 16.25 (truth 16.51); robustness value for break-even 0.5%

61.5 Day 61 mini-project: benchmarking (75 min)

"cf = 0.088" means nothing to a marketing director. Benchmarking gives it meaning: pretend an observed feature had been left out, and measure its strength on the same scale. Then say: "a hidden confounder as strong as app use would move the estimate by $X". Create day61/benchmarking.py:

# day61/benchmarking.py   (python -m day61.benchmarking)
"""
Mini-project: 'cf = 0.05' means nothing to a marketing director. Benchmarking does: pretend an OBSERVED
feature had been left out, and measure how strong it is on the same scale. 'A hidden confounder as strong
as app use would move the estimate by $X.' Then, because this is a simulator, measure the real hidden
browsing signal the same way.
"""
import doubleml as dml
import numpy as np
import polars as pl

from causalkit.coupons import X_COLS, coupons, public
from causalkit.learners import clf, reg
from day61.dml_sensitivity import fit_irm

np.random.seed(61)
full = coupons(hidden=0.5)
irm = fit_irm(public(full).to_pandas())
rows = []
for feature in ["deal_share", "app_user", "email_open", "pre_spend"]:
    b = irm.sensitivity_benchmark(benchmarking_set=[feature]).iloc[0]
    rows.append({"left out": feature, "cf_y": round(b.cf_y, 3), "cf_d": round(b.cf_d, 3), "rho": round(b.rho, 2),
                 "estimate moves by": round(b.delta_theta, 2)})

df = full.to_pandas()
cols = X_COLS + ["_browsing"]
np.random.seed(61)
oracle = dml.DoubleMLIRM(dml.DoubleMLData(df[cols + ["coupon", "spend"]], "spend", "coupon", cols),
                         ml_g=reg(), ml_m=clf(), n_folds=5, trimming_threshold=0.02).fit()
b = oracle.sensitivity_benchmark(benchmarking_set=["_browsing"]).iloc[0]
rows.append({"left out": "browsing (the real hidden one)", "cf_y": round(b.cf_y, 3), "cf_d": round(b.cf_d, 3),
             "rho": round(b.rho, 2), "estimate moves by": round(b.delta_theta, 2)})
with pl.Config(tbl_width_chars=110, fmt_str_lengths=40):
    print(pl.DataFrame(rows))
print(f"Estimate with browsing included: {oracle.coef[0]:.2f}")

python -m day61.benchmarking

│ left out                       ┆ cf_y  ┆ cf_d  ┆ rho  ┆ estimate moves by │
│ deal_share                     ┆ 0.034 ┆ 0.138 ┆ 0.45 ┆ 2.48              │
│ app_user                       ┆ 0.165 ┆ 0.049 ┆ 0.63 ┆ 4.77              │
│ email_open                     ┆ 0.074 ┆ 0.002 ┆ 0.21 ┆ 0.2               │
│ pre_spend                      ┆ 1.0   ┆ 0.104 ┆ -0.71┆ -20.68            │
│ browsing (the real hidden one) ┆ 0.154 ┆ 0.114 ┆ 0.78 ┆ 8.36              │
Estimate with browsing included: 16.24

• A confounder as strong as app use (cf_y 0.165, cf_d 0.049) moves the estimate by $4.77; one as strong as deal-seeking by $2.48.
• Past spending is in a different league (it explains almost all of spend's variation). Nobody believes a confounder that strong is hiding, so it isn't a useful benchmark.
• The real hidden browsing signal: cf_y 0.154, cf_d 0.114, and it moves the estimate by $8.36, from $24.61 to $16.24. It's about as strong as app use on spend and stronger on the coupon. A careful analyst who said "something about as strong as app use could be missing" would have concluded that the coupon's profitability was in doubt, which was right.

Write in day61/NOTES.md: which observed feature is the most plausible benchmark for "what we didn't measure", and why?

61.6 Save your work with Git (20 min)

git add .
git commit -m "Day 61: OVB formula, DoubleML sensitivity bounds, robustness values, benchmarking"
git push

Day 61 checklist

• I can state the omitted-variable bias formula and explain why both factors are needed
• I can run DoubleML's sensitivity analysis and read bounds and robustness values
• I compute robustness values against the decision threshold, not just against zero
• I can benchmark a hypothetical confounder against observed features
• I know that a low robustness value doesn't prove confounding exists