Validating CATEs: RATE, Qini and Calibration

57.1 How do you test what you can't observe? (40 min)

In real data there is no true CATE to compare with. But a CATE model makes predictions that can be checked, as long as you have data where the treatment was randomised and the model never saw:

They test different things. Uplift tests ranking (what targeting needs). BLP and calibration test levels (what a budget or a price decision needs).

NorthMart has what's needed: last spring's coupon A/B test, on 10,000 customers who aren't in the training data.

57.2 Five models, one experiment (60 min)

EconML's DRTester runs all three checks. Create day57/validate.py. Its fit_models and experiment functions are reused below (about a minute):

# day57/validate.py   (python -m day57.validate)
"""
In real data there's no true CATE to compare with. How do you know a CATE model works?
Test it on a RANDOMISED experiment it wasn't trained on: last spring's coupon A/B test on 10,000 other
customers. EconML's DRTester computes, on that experiment:
  BLP slope:  regress doubly robust effect estimates on the model's predictions. 1 = well calibrated;
              above 1 = predictions too compressed; 0 = no information.
  AUTOC:      how much better than average are the customers the model ranks highest? (Yadlowsky et al.)
  calibration R2: do predicted group effects match measured group effects?
Then compare with what only a simulator can tell you: the true error.
"""
import numpy as np
import polars as pl
from econml.dr import DRLearner
from econml.metalearners import SLearner, TLearner
from econml.validate import DRTester

from causalkit.coupons import X_COLS, coupons
from causalkit.learners import clf, reg
from day56.causal_forest import fit_forest


class RandomScores:
    """A 'model' with no information, to calibrate what the metrics look like for noise."""
    def effect(self, X, T0=0, T1=1):
        return np.random.default_rng(len(X)).normal(16, 10, len(X))


def fit_models():
    """The five CATE models, all trained on the observational train half. Returns (models, train)."""
    cf, train, _ = fit_forest()
    models = {
        "causal forest": cf,
        "DR-learner": DRLearner(model_propensity=clf(), model_regression=reg(),
                                model_final=reg(n_estimators=100, num_leaves=4, min_child_samples=300), cv=5,
                                min_propensity=0.02, random_state=0).fit(train.spend, train.coupon, X=train[X_COLS]),
        "T-learner": TLearner(models=reg()).fit(train.spend, train.coupon, X=train[X_COLS]),
        "S-learner": SLearner(overall_model=reg()).fit(train.spend, train.coupon, X=train[X_COLS]),
        "random scores": RandomScores(),
    }
    return models, train


def experiment():
    """Last spring's randomised coupon test: 10,000 different customers."""
    return coupons(n=10_000, seed=570, randomized=True).to_pandas()


if __name__ == "__main__":
    models, train = fit_models()
    data = experiment()
    Xv, Xt = data[X_COLS].to_numpy(), train[X_COLS].to_numpy()
    rows = []
    for name, m in models.items():
        tester = DRTester(model_regression=reg(), model_propensity=clf(), cate=m)
        tester.fit_nuisance(Xv, data.coupon.to_numpy(), data.spend.to_numpy(),
                            Xt, train.coupon.to_numpy(), train.spend.to_numpy())
        s = tester.evaluate_all(Xv, Xt, n_groups=4, n_bootstrap=200).summary().iloc[0]
        pred = m.effect(Xv)
        rows.append({"model": name, "BLP slope": f"{s.blp_est:.2f} ({s.blp_se:.2f})", "AUTOC": round(s.autoc_est, 1),
                     "calibration R2": round(s.cal_r_squared, 2),
                     "true PEHE": round(float(np.sqrt(np.mean((pred - data._tau) ** 2))), 1),
                     "true correlation": round(float(np.corrcoef(pred, data._tau)[0, 1]), 2)})
    with pl.Config(tbl_width_chars=120, fmt_str_lengths=30):
        print(pl.DataFrame(rows))

python -m day57.validate

│ model         ┆ BLP slope   ┆ AUTOC ┆ calibration R2 ┆ true PEHE ┆ true correlation │
│ causal forest ┆ 1.64 (0.19) ┆ 9.5   ┆ 0.48           ┆ 8.7       ┆ 0.84             │
│ DR-learner    ┆ 0.89 (0.12) ┆ 9.2   ┆ 0.83           ┆ 6.2       ┆ 0.9              │
│ T-learner     ┆ 0.04 (0.07) ┆ 4.0   ┆ -2.08          ┆ 13.7      ┆ 0.68             │
│ S-learner     ┆ 0.94 (0.13) ┆ 8.2   ┆ 0.63           ┆ 7.4       ┆ 0.87             │
│ random scores ┆ 0.01 (0.14) ┆ 0.7   ┆ -26.2          ┆ 17.0      ┆ 0.01             │

The first three columns are what you'd have in real work; the last two only a simulator can give. Do the checks agree with the truth?

• Random scores fail everything: BLP slope 0, AUTOC near 0. The checks aren't fooled by noise.
• The causal forest's slope of 1.64 says its predictions are too compressed: a $1 difference in prediction means $1.64 of real difference. That's exactly Day 56's shrinkage, detected from experimental data alone.
• The T-learner's slope near 0 doesn't mean it has no information (its true correlation is 0.68): its predictions are so spread out that most of their variation is noise. Its calibration R² is negative: predicting the overall average would beat it.
• The DR-learner has the best slope, calibration and true error. The S-learner is close behind. The checks pick the right model.

57.3 Uplift curves by hand (50 min)

Concept. Rank the experiment's customers by predicted effect. For the top q%, measure the effect by a difference in means (it's an A/B test, so that's unbiased), and subtract the overall effect. Plotted against q, that's the targeting operating characteristic (TOC); its average is the AUTOC. Create day57/uplift_by_hand.py:

# day57/uplift_by_hand.py   (python -m day57.uplift_by_hand)
"""
The targeting operating characteristic (TOC) by hand, on the randomised experiment: rank customers by a
model's predicted effect; for the top q% measure the effect by a difference in means (it's an A/B test, so
that's unbiased) and subtract the overall effect. A useful model's top customers respond much more than
average. AUTOC = the area under that curve.
"""
import matplotlib.pyplot as plt
import numpy as np

from causalkit.coupons import X_COLS
from day57.validate import experiment, fit_models

models, _ = fit_models()
data = experiment()
y, d = data.spend.to_numpy(), data.coupon.to_numpy()
overall = y[d == 1].mean() - y[d == 0].mean()
qs = np.arange(5, 101, 5)
fig, ax = plt.subplots(figsize=(8, 4))
print(f"Overall effect in the experiment: {overall:.1f}")
for name, m in models.items():
    order = np.argsort(-m.effect(data[X_COLS].to_numpy()))
    toc = []
    for q in qs:
        top = order[: int(len(order) * q / 100)]
        toc.append(y[top][d[top] == 1].mean() - y[top][d[top] == 0].mean() - overall)
    print(f"{name:<14} top 10%: {toc[1] + overall:5.1f}   top 30%: {toc[5] + overall:5.1f}   AUTOC {np.mean(toc):5.1f}")
    ax.plot(qs, toc, marker="o", ms=3, label=name)
ax.axhline(0, color="grey", lw=0.8)
ax.set_xlabel("share of customers targeted, ranked by the model (%)")
ax.set_ylabel("effect in top q% minus overall ($)")
ax.legend()
fig.tight_layout()
fig.savefig("figures/day57_toc.png", dpi=120)
print("Saved figures/day57_toc.png")

python -m day57.uplift_by_hand

Overall effect in the experiment: 15.4
causal forest  top 10%:  37.3   top 30%:  31.9   AUTOC  10.8
DR-learner     top 10%:  37.6   top 30%:  30.7   AUTOC  10.7
T-learner      top 10%:  36.7   top 30%:  29.5   AUTOC   8.6
S-learner      top 10%:  39.8   top 30%:  31.2   AUTOC  10.5
random scores  top 10%:  15.6   top 30%:  15.9   AUTOC  -0.7
Saved figures/day57_toc.png

For targeting, the forest, DR-learner and S-learner are almost equally good: each one's top 30% responds about twice as much as average. The forest's compression doesn't hurt the ranking at all. The T-learner is somewhat worse. (These numbers differ slightly from DRTester's AUTOC, which uses doubly robust scores and a finer grid; the conclusions are the same.)

Open the figure: every real model's curve starts high and falls towards zero as q grows to 100% (targeting everyone = the average effect).

57.4 Can you validate without an experiment? (35 min)

Try it. Run the same DRTester checks on the observational test half of the coupon data (no experiment). DRTester uses doubly robust scores, which adjust for the targeting. Create day57/validate_observational.py.

# day57/validate_observational.py   (python -m day57.validate_observational)
from econml.validate import DRTester

from causalkit.coupons import X_COLS
from causalkit.learners import clf, reg
from day56.causal_forest import fit_forest
from day57.validate import fit_models

models, train = fit_models()
_, _, test = fit_forest()
for name in ["causal forest", "DR-learner", "T-learner"]:
    tester = DRTester(model_regression=reg(), model_propensity=clf(), cate=models[name])
    tester.fit_nuisance(test[X_COLS].to_numpy(), test.coupon.to_numpy(), test.spend.to_numpy(),
                        train[X_COLS].to_numpy(), train.coupon.to_numpy(), train.spend.to_numpy())
    s = tester.evaluate_all(test[X_COLS].to_numpy(), train[X_COLS].to_numpy(), n_bootstrap=200).summary().iloc[0]
    print(f"{name:<14} BLP slope {s.blp_est:.2f} ({s.blp_se:.2f})   AUTOC {s.autoc_est:.1f}   "
          f"calibration R2 {s.cal_r_squared:.2f}")

causal forest  BLP slope 1.55 (0.12)   AUTOC 9.2   calibration R2 0.53
DR-learner     BLP slope 1.03 (0.07)   AUTOC 10.0   calibration R2 0.88
T-learner      BLP slope 0.52 (0.05)   AUTOC 7.3   calibration R2 -0.02

57.5 Day 57 mini-project: calibration by hand (65 min)

Cut the experiment's customers into quartiles of predicted effect (cut points from the training data), measure each quartile's effect by a difference in means, and plot it against the average prediction. Create day57/calibration.py:

# day57/calibration.py   (python -m day57.calibration)
"""
Mini-project: calibration by hand. Cut the experiment's customers into quartiles of predicted effect
(cut points from the training data), measure each quartile's effect by difference in means, and plot it
against the average prediction. On the 45-degree line = predictions mean what they say.
"""
import matplotlib.pyplot as plt
import numpy as np
import polars as pl

from causalkit.coupons import X_COLS
from day57.validate import experiment, fit_models

models, train = fit_models()
data = experiment()
y, d = data.spend.to_numpy(), data.coupon.to_numpy()
fig, ax = plt.subplots(figsize=(5.5, 5))
rows = []
for name in ["causal forest", "DR-learner", "T-learner"]:
    m = models[name]
    cuts = np.quantile(m.effect(train[X_COLS].to_numpy()), [0.25, 0.5, 0.75])
    pred = m.effect(data[X_COLS].to_numpy())
    q = np.digitize(pred, cuts)
    xs, ys = [], []
    for g in range(4):
        sel = q == g
        diff = y[sel][d[sel] == 1].mean() - y[sel][d[sel] == 0].mean()
        se = np.sqrt(y[sel][d[sel] == 1].var() / (d[sel] == 1).sum() + y[sel][d[sel] == 0].var() / (d[sel] == 0).sum())
        xs.append(pred[sel].mean())
        ys.append(diff)
        rows.append({"model": name, "quartile": g + 1, "predicted": round(pred[sel].mean(), 1),
                     "measured in experiment": round(diff, 1), "SE": round(se, 1)})
    ax.plot(xs, ys, marker="o", label=name)
lim = [-10, 45]
ax.plot(lim, lim, color="grey", ls="--", lw=0.8)
ax.set_xlabel("average predicted effect ($)")
ax.set_ylabel("effect measured in the experiment ($)")
ax.legend()
fig.tight_layout()
fig.savefig("figures/day57_calibration.png", dpi=120)
with pl.Config(tbl_rows=12):
    print(pl.DataFrame(rows))
print("Saved figures/day57_calibration.png")

python -m day57.calibration

│ model         ┆ quartile ┆ predicted ┆ measured in experiment ┆ SE  │
│ causal forest ┆ 1        ┆ 8.5       ┆ 3.9                    ┆ 2.5 │
│ causal forest ┆ 2        ┆ 11.5      ┆ 6.0                    ┆ 2.4 │
│ causal forest ┆ 3        ┆ 16.3      ┆ 21.1                   ┆ 2.4 │
│ causal forest ┆ 4        ┆ 27.2      ┆ 32.4                   ┆ 2.5 │
│ DR-learner    ┆ 1        ┆ 2.9       ┆ 3.2                    ┆ 2.7 │
│ DR-learner    ┆ 2        ┆ 11.3      ┆ 6.6                    ┆ 2.3 │
│ DR-learner    ┆ 3        ┆ 18.1      ┆ 20.6                   ┆ 2.3 │
│ DR-learner    ┆ 4        ┆ 32.3      ┆ 32.4                   ┆ 2.3 │
│ T-learner     ┆ 1        ┆ -7.5      ┆ 7.7                    ┆ 2.8 │
│ T-learner     ┆ 2        ┆ 9.3       ┆ 11.3                   ┆ 2.3 │
│ T-learner     ┆ 3        ┆ 21.8      ┆ 13.2                   ┆ 2.4 │
│ T-learner     ┆ 4        ┆ 40.1      ┆ 31.5                   ┆ 2.3 │
Saved figures/day57_calibration.png

Open the figure. The DR-learner's points sit on the 45-degree line. The forest's line is steeper than 45 degrees (predicted 8.5 to 27.2; measured 3.9 to 32.4): compressed. The T-learner's is flatter (predicted −7.5 to 40.1; measured 7.7 to 31.5): over-dispersed. Its bottom quartile is predicted to lose $7.50 per coupon, and actually gains $7.70.

That matters for decisions. If marketing used the T-learner to stop couponing customers with "negative" predicted effects, it would drop customers who respond.

57.6 Save your work with Git (20 min)

git add .
git commit -m "Day 57: CATE validation on an experiment: BLP, TOC/AUTOC, calibration"
git push

Coming next (Days 58–60). Everything so far had a yes/no treatment. Prices are continuous, set by a pricing team that reacts to demand, and they differ by segment. Days 58–60 put DML and heterogeneity to work on NorthMart's most consequential question.

Day 57 checklist

• I can validate a CATE model on a randomised experiment it wasn't trained on
• I can read a BLP slope, an AUTOC and a calibration R², and say what each one tests
• I can draw a TOC curve and a calibration plot by hand
• I know that validation on observational data inherits the no-hidden-confounders assumption