From Prediction to Causal ML

51.1 Two different questions (40 min)

Machine learning is built for the first column. Days 51–54 are about borrowing its power for the second without inheriting its blind spots.

51.2 A great predictor, a biased effect (50 min)

NorthMart's targeting model sends $10-off coupons mostly to deal-seekers and lapsing customers. Lapsing customers spend less, so the people who get coupons would have spent less anyway.

Create day51/prediction_vs_causation.py:

# day51/prediction_vs_causation.py   (python -m day51.prediction_vs_causation)
"""
NorthMart's targeting model sends $10-off coupons. Did they raise next-month spend?
A model that PREDICTS spend very well is not automatically a model that measures the coupon's effect.
"""
import numpy as np
import statsmodels.formula.api as smf
from sklearn.model_selection import cross_val_score

from causalkit.coupons import X_COLS, coupons, public, truth
from causalkit.learners import reg

full = coupons()
df = public(full).to_pandas()
t = truth(full)
print(f"{len(df):,} customers, {len(X_COLS)} features; {df.coupon.mean():.0%} got a coupon")

naive = df.spend[df.coupon == 1].mean() - df.spend[df.coupon == 0].mean()
ols = smf.ols("spend ~ coupon + " + " + ".join(X_COLS), df).fit()

model = reg()
r2 = cross_val_score(model, df[X_COLS + ["coupon"]], df.spend, cv=5, scoring="r2").mean()
model.fit(df[X_COLS + ["coupon"]], df.spend)
s_learner = (model.predict(df[X_COLS + ["coupon"]].assign(coupon=1))
             - model.predict(df[X_COLS + ["coupon"]].assign(coupon=0))).mean()
rank = np.argsort(-model.feature_importances_).tolist().index(len(X_COLS)) + 1

print(f"\nnaive difference (coupon vs none):          {naive:+6.1f}")
print(f"OLS with all 24 features, linear:          {ols.params['coupon']:+6.1f}  (R2 {ols.rsquared:.2f})")
print(f"LightGBM 'S-learner' (predict with/without): {s_learner:+6.1f}  (cross-validated R2 {r2:.2f})")
print(f"   the coupon ranks {rank} of {len(X_COLS) + 1} in the model's feature importance")
print(f"\nTruth: ATE {t['ATE']:.1f}, ATT {t['ATT']:.1f} ($ per customer)")

python -m day51.prediction_vs_causation

20,000 customers, 24 features; 51% got a coupon

naive difference (coupon vs none):            +0.9
OLS with all 24 features, linear:           +13.4  (R2 0.63)
LightGBM 'S-learner' (predict with/without):  +14.9  (cross-validated R2 0.68)
   the coupon ranks 10 of 25 in the model's feature importance

Truth: ATE 16.5, ATT 20.3 ($ per customer)

• The naive comparison says coupons do nothing: confounding hides almost the whole effect.
• Linear OLS gets most of the way, but its straight lines can't remove all the confounding.
• LightGBM, a better predictor (R² 0.68 vs 0.63), is still $1.6 short. The "S-learner" (one model, predict everyone twice: with and without the coupon) is the most natural way to use a predictive model for effects, and it's biased.

51.3 Why: regularisation and confounding (45 min)

Concept. A predictive model trades a little bias for a lot less variance: that's what regularisation (shrinkage, early stopping, limited tree depth) does. It's the right trade for prediction. But the bias lands wherever it costs least predictive accuracy, and the coupon is a modest predictor of spend (rank 10 of 25), so its effect gets shrunk.

There's a second problem. Coupon and confounders are correlated (deal-seekers get coupons). When the model regularises, it can credit part of the coupon's effect to the confounders, or part of the confounders' effect to the coupon. Nothing in the fitting process cares which, because predictions are equally good either way.

Chernozhukov et al. (2018) call this regularisation bias, and show it doesn't disappear with more data at the rate needed for valid confidence intervals. Their fix, double/debiased machine learning (DML), is tomorrow.

51.4 Is it just confounding? (40 min)

Try it. Fit the same S-learner on coupons(randomized=True): same customers, but coupons given by a coin flip. There's no confounding. Is the S-learner still biased? Create day51/s_learner_rct.py.

# day51/s_learner_rct.py   (python -m day51.s_learner_rct)
from causalkit.coupons import X_COLS, coupons, public, truth
from causalkit.learners import reg

full = coupons(randomized=True)
df = public(full).to_pandas()
m = reg().fit(df[X_COLS + ["coupon"]], df.spend)
s = (m.predict(df[X_COLS + ["coupon"]].assign(coupon=1)) - m.predict(df[X_COLS + ["coupon"]].assign(coupon=0))).mean()
diff = df.spend[df.coupon == 1].mean() - df.spend[df.coupon == 0].mean()
print(f"Randomised coupons: difference in means {diff:.1f}, S-learner {s:.1f}, truth {truth(full)['ATE']:.1f}")

Randomised coupons: difference in means 16.3, S-learner 15.4, truth 16.5

51.5 Day 51 mini-project: is a better predictor a better estimator? (75 min)

Fit LightGBM S-learners of three sizes on 10 simulated datasets. Score each on held-out R² and on the error of its ATE. Create day51/better_predictor.py (about a minute):

# day51/better_predictor.py   (python -m day51.better_predictor)
"""
Mini-project: is a better predictor a better effect estimator? Fit LightGBM S-learners of increasing
complexity on 10 simulated datasets. Score each on out-of-sample R2 (prediction) and on the error of its
ATE (causal). The coupon is one feature among 25, and the trees spend their splits where the
predictive payoff is largest, which isn't the coupon.
"""
import numpy as np
import polars as pl
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split

from causalkit.coupons import X_COLS, coupons, public, truth
from causalkit.learners import reg

settings = {"shallow (50 trees, 4 leaves)": dict(n_estimators=50, num_leaves=4),
            "medium (300 trees, 15 leaves)": dict(n_estimators=300, num_leaves=15),
            "deep (1000 trees, 63 leaves)": dict(n_estimators=1000, num_leaves=63, min_child_samples=10)}
rows = []
for name, kw in settings.items():
    r2s, errs = [], []
    for s in range(10):
        full = coupons(seed=700 + s)
        df = public(full).to_pandas()
        tr, te = train_test_split(df, test_size=0.3, random_state=s)
        m = reg(**kw).fit(tr[X_COLS + ["coupon"]], tr.spend)
        r2s.append(r2_score(te.spend, m.predict(te[X_COLS + ["coupon"]])))
        ate = (m.predict(df[X_COLS + ["coupon"]].assign(coupon=1))
               - m.predict(df[X_COLS + ["coupon"]].assign(coupon=0))).mean()
        errs.append(ate - truth(full)["ATE"])
    rows.append({"model": name, "test R2": round(np.mean(r2s), 3), "ATE average error": round(np.mean(errs), 1),
                 "ATE error SD": round(np.std(errs), 1)})
with pl.Config(tbl_width_chars=100, fmt_str_lengths=40):
    print(pl.DataFrame(rows))

python -m day51.better_predictor

│ model                         ┆ test R2 ┆ ATE average error ┆ ATE error SD │
│ shallow (50 trees, 4 leaves)  ┆ 0.524   ┆ -13.2             ┆ 0.9          │
│ medium (300 trees, 15 leaves) ┆ 0.671   ┆ -1.9              ┆ 0.7          │
│ deep (1000 trees, 63 leaves)  ┆ 0.652   ┆ -1.8              ┆ 0.6          │

• The shallow model loses most of the effect (−$13 of $16.5): it barely uses the coupon at all.
• The medium and deep models are similar predictors and similarly biased (about −$1.9), with a small SD. The bias is systematic, so more data won't fix it, and a confidence interval around it would be confidently wrong.
• Model selection by R² doesn't select for good effect estimates.

Write two sentences in day51/NOTES.md: why would a marketing team that judges its model by R² end up underestimating what its coupons do?

51.6 Save your work with Git (20 min)

git add .
git commit -m "Day 51: coupon simulator, prediction vs causation, regularisation bias"
git push

Day 51 checklist

• I can explain the difference between predicting an outcome and estimating an effect
• I can estimate an effect with an S-learner, and explain why it's biased even in an experiment
• I know that a better R² doesn't mean a better effect estimate
• I can explain regularisation bias in plain language