Uplift Modelling

64.1 Uplift is a CATE in marketing's language (40 min)

Marketing teams have estimated heterogeneous effects for decades, under the name uplift modelling: who buys because of the campaign? It's Day 55's CATE, usually estimated on randomised campaigns. Marketing sorts customers into four groups:

A response model targets everyone likely to buy, including sure things; an uplift model targets persuadables and avoids sleeping dogs.

The data: last spring's coupon test, 40,000 customers, coupons by coin flip. Half to train, half to evaluate.

64.2 Uplift metrics, from scratch (40 min)

Concept. Sort customers by predicted uplift, highest first. For the top k customers:

• uplift curve: the measured uplift among them (difference in means: it's an experiment) × k, the extra outcome you'd get by treating exactly those k;
• Qini curve: treated customers' total outcome minus control customers' total scaled to the same number of treated customers (Radcliffe, 2007). Same idea, slightly different scaling.

A good model's curve rises steeply and then flattens or falls (the last customers added are sleeping dogs). Random targeting is the straight line from zero to the overall effect. The area between the curve and that line scores the model.

The usual Python package for this, scikit-uplift, no longer runs with current numpy and scikit-learn. It's 40 lines; write them. Create causalkit/uplift.py:

"""Uplift metrics for randomised campaigns (Day 64), written out because the usual package no longer runs.

Sort customers by a model's score (highest predicted uplift first). For the top k customers:
    uplift curve:  (mean outcome of treated - mean outcome of control among the top k) x k
                   = extra outcome if we had treated exactly those k customers
    Qini curve:    sum of treated outcomes - sum of control outcomes x (treated count / control count),
                   the same idea, scaled by the number of treated customers among the top k
Area under each curve, minus the area for random targeting (a straight line), is the model's score.
"""
import numpy as np
import polars as pl


def _sorted(y, t, score):
    order = np.argsort(-np.asarray(score), kind="stable")
    return np.asarray(y, float)[order], np.asarray(t, int)[order]


def uplift_curve(y, t, score, points=50):
    """Returns (share targeted, extra outcome per customer in the whole population)."""
    y, t = _sorted(y, t, score)
    n = len(y)
    ks = np.unique(np.linspace(n // points, n, points).astype(int))
    out = []
    for k in ks:
        yt, tt = y[:k], t[:k]
        if tt.sum() == 0 or (1 - tt).sum() == 0:
            out.append(0.0)
            continue
        out.append((yt[tt == 1].mean() - yt[tt == 0].mean()) * k / n)
    return ks / n, np.array(out)


def qini_curve(y, t, score, points=50):
    """Returns (share targeted, Qini value per customer in the whole population)."""
    y, t = _sorted(y, t, score)
    n = len(y)
    ks = np.unique(np.linspace(n // points, n, points).astype(int))
    cy_t, cy_c = np.cumsum(y * t), np.cumsum(y * (1 - t))
    cn_t, cn_c = np.cumsum(t), np.cumsum(1 - t)
    q = cy_t[ks - 1] - cy_c[ks - 1] * cn_t[ks - 1] / np.maximum(cn_c[ks - 1], 1)
    return ks / n, q / n


def area_above_random(x, curve):
    """Area between a curve and the straight line from (0, 0) to its end point."""
    return float(np.trapezoid(curve - x * curve[-1], x))


def uplift_at_k(y, t, score, k=0.3):
    """Measured uplift (difference in means) among the top k share of customers."""
    y, t = _sorted(y, t, score)
    top = int(len(y) * k)
    return float(y[:top][t[:top] == 1].mean() - y[:top][t[:top] == 0].mean())


def by_decile(y, t, score):
    """Measured uplift in each decile of the score, with standard errors."""
    y, t = _sorted(y, t, score)
    rows = []
    for d, idx in enumerate(np.array_split(np.arange(len(y)), 10), start=1):
        yd, td = y[idx], t[idx]
        a, b = yd[td == 1], yd[td == 0]
        rows.append({"decile": d, "customers": len(idx), "uplift": round(a.mean() - b.mean(), 1),
                     "SE": round(float(np.sqrt(a.var(ddof=1) / len(a) + b.var(ddof=1) / len(b))), 1)})
    return pl.DataFrame(rows)

64.3 Four uplift models on a 40,000-customer test (55 min)

Three classic uplift models and the DR-learner of Day 55. The transformed outcome is the oldest trick in uplift modelling: with treatment probability p, Y* = Y·(T − p)/(p(1 − p)) has expectation equal to the uplift, so any regression of Y* on X estimates it. Create day64/uplift_models.py:

# day64/uplift_models.py   (python -m day64.uplift_models)
"""
Uplift modelling is CATE estimation for randomised campaigns, in marketing's language. NorthMart ran a
40,000-customer coupon test last spring (coupons by coin flip). Train uplift models on half, score on the
other half. Three standard uplift models, plus Day 55's DR-learner:
  solo model (S-learner), two models (T-learner),
  transformed outcome: with p = 0.5, Y* = 2Y if couponed, -2Y if not, has expectation tau(x): regress it on X.
"""
import numpy as np
import polars as pl
from econml.dr import DRLearner
from econml.metalearners import SLearner, TLearner
from sklearn.model_selection import train_test_split

from causalkit.coupons import X_COLS, coupons
from causalkit.learners import clf, reg
from causalkit.uplift import area_above_random, qini_curve, uplift_at_k


def experiment():
    return train_test_split(coupons(n=40_000, seed=640, randomized=True).to_pandas(), test_size=0.5, random_state=0)


def fit_uplift_models(train):
    p = train.coupon.mean()
    y_star = train.spend * (train.coupon - p) / (p * (1 - p))
    return {
        "solo model (S)": SLearner(overall_model=reg()).fit(train.spend, train.coupon, X=train[X_COLS]),
        "two models (T)": TLearner(models=reg()).fit(train.spend, train.coupon, X=train[X_COLS]),
        "transformed outcome": reg(n_estimators=100, num_leaves=4, min_child_samples=300).fit(train[X_COLS], y_star),
        "DR-learner": DRLearner(model_propensity=clf(), model_regression=reg(),
                                model_final=reg(n_estimators=100, num_leaves=4, min_child_samples=300), cv=5,
                                min_propensity=0.02, random_state=0).fit(train.spend, train.coupon, X=train[X_COLS]),
    }


def score(model, X):
    return model.effect(X) if hasattr(model, "effect") else model.predict(X)


if __name__ == "__main__":
    train, test = experiment()
    models = fit_uplift_models(train)
    rows = []
    for name, m in models.items():
        s = score(m, test[X_COLS])
        x, q = qini_curve(test.spend, test.coupon, s)
        top = np.argsort(-s)[: int(0.3 * len(s))]
        rows.append({"model": name, "Qini area": round(area_above_random(x, q), 2),
                     "uplift in top 30%": round(uplift_at_k(test.spend, test.coupon, s, 0.3), 1),
                     "true uplift in top 30%": round(test._tau.to_numpy()[top].mean(), 1),
                     "true PEHE": round(float(np.sqrt(np.mean((s - test._tau) ** 2))), 1)})
    with pl.Config(tbl_width_chars=120, fmt_str_lengths=30):
        print(pl.DataFrame(rows))
    print(f"Average uplift (everyone): {test.spend[test.coupon == 1].mean() - test.spend[test.coupon == 0].mean():.1f}")

python -m day64.uplift_models

│ model               ┆ Qini area ┆ uplift in top 30% ┆ true uplift in top 30% ┆ true PEHE │
│ solo model (S)      ┆ 2.09      ┆ 35.3              ┆ 32.1                   ┆ 5.4       │
│ two models (T)      ┆ 1.87      ┆ 32.3              ┆ 30.2                   ┆ 9.8       │
│ transformed outcome ┆ 1.29      ┆ 29.6              ┆ 27.4                   ┆ 14.7      │
│ DR-learner          ┆ 2.17      ┆ 35.6              ┆ 32.6                   ┆ 4.0       │
Average uplift (everyone): 17.5

• With 20,000 training customers from an experiment, the DR-learner and the solo model are best; the transformed outcome is worst (Y* is extremely noisy: each customer's value is ±2 × spend).
• The Qini area ranks the models the same way as the true PEHE. That's the point: the Qini area is computable without the truth.
• "Uplift in top 30%" is measured on the held-out half and is a few dollars above the truth: the top 30% is chosen by the model, and the measurement adds noise. With one model it's unbiased; when you compare many models and report the best, it's optimistic.

64.4 Qini and uplift curves (40 min)

Create day64/qini_plot.py:

# day64/qini_plot.py   (python -m day64.qini_plot)
"""Qini and uplift curves for the four models, plus random targeting and the oracle (true uplift)."""
import matplotlib.pyplot as plt

from causalkit.coupons import X_COLS
from causalkit.uplift import qini_curve, uplift_curve
from day64.uplift_models import experiment, fit_uplift_models, score

train, test = experiment()
models = fit_uplift_models(train)
fig, axes = plt.subplots(1, 2, figsize=(11, 4))
scores = {name: score(m, test[X_COLS]) for name, m in models.items()} | {"oracle": test._tau.to_numpy()}
for name, s in scores.items():
    for ax, curve in zip(axes, [qini_curve, uplift_curve]):
        x, c = curve(test.spend, test.coupon, s)
        ax.plot(x, c, label=name, ls="--" if name == "oracle" else "-")
for ax, title in zip(axes, ["Qini curve", "Uplift curve"]):
    x, c = qini_curve(test.spend, test.coupon, scores["oracle"]) if title == "Qini curve" else \
        uplift_curve(test.spend, test.coupon, scores["oracle"])
    ax.plot([0, 1], [0, c[-1]], color="grey", lw=0.8, label="random")
    ax.set_title(title)
    ax.set_xlabel("share of customers targeted (highest score first)")
axes[0].set_ylabel("extra $ per customer in the base")
axes[1].legend(fontsize=8)
fig.tight_layout()
fig.savefig("figures/day64_qini.png", dpi=120)
print("Saved figures/day64_qini.png")

python -m day64.qini_plot

Saved figures/day64_qini.png

Open the figure. The oracle (true uplift) rises fastest, then falls at the end: the last customers it adds have negative uplift. The DR-learner and solo model follow it closely. The uplift curve's right-hand end is the same for every model (treating everyone gives the average effect, whatever the ranking).

Try it. Rerun the four models on a 4,000-customer experiment instead of 40,000. Which model's Qini area falls most? Create day64/small_experiment.py.

# day64/small_experiment.py   (python -m day64.small_experiment)
import numpy as np
from sklearn.model_selection import train_test_split

from causalkit.coupons import X_COLS, coupons
from causalkit.uplift import area_above_random, qini_curve
from day64.uplift_models import fit_uplift_models, score

for n in [4_000, 40_000]:
    train, test = train_test_split(coupons(n=n, seed=640, randomized=True).to_pandas(), test_size=0.5, random_state=0)
    out = []
    for name, m in fit_uplift_models(train).items():
        x, q = qini_curve(test.spend, test.coupon, score(m, test[X_COLS]))
        out.append(f"{name.split(' (')[0]} {area_above_random(x, q):.2f}")
    print(f"n = {n:>6,}: Qini area  " + " | ".join(out))

n =  4,000: Qini area  solo model 1.28 | two models 0.62 | transformed outcome 0.00 | DR-learner 1.03
n = 40,000: Qini area  solo model 2.09 | two models 1.87 | transformed outcome 1.29 | DR-learner 2.17

64.5 Day 64 mini-project: the decile table and the sleeping dogs (75 min)

The table marketing reads: uplift measured in each decile of the model's score. Create day64/deciles.py:

# day64/deciles.py   (python -m day64.deciles)
"""
Mini-project: the table marketing actually reads. Uplift measured in each decile of the DR-learner's score
(decile 1 = highest predicted uplift), with standard errors. Look at the bottom: customers for whom a coupon
does nothing, or reduces spending ('sleeping dogs': loyal customers who'd have paid full price).
"""
import numpy as np
import polars as pl

from causalkit.coupons import X_COLS
from causalkit.uplift import by_decile
from day64.uplift_models import experiment, fit_uplift_models, score

train, test = experiment()
s = score(fit_uplift_models(train)["DR-learner"], test[X_COLS])
table = by_decile(test.spend, test.coupon, s)
true = test.assign(score=s).sort_values("score", ascending=False)._tau.to_numpy()
table = table.with_columns(true_uplift=pl.Series([round(c.mean(), 1) for c in np.array_split(true, 10)]))
print(table)
print(f"Share of customers with a negative true uplift: {(test._tau < 0).mean():.1%}; "
      f"in the bottom two deciles: {(true[-len(true) // 5:] < 0).mean():.1%}")

python -m day64.deciles

│ decile ┆ customers ┆ uplift ┆ SE  ┆ true_uplift │
│ 1      ┆ 2000      ┆ 41.3   ┆ 2.7 ┆ 40.6        │
│ 2      ┆ 2000      ┆ 37.7   ┆ 2.5 ┆ 31.4        │
│ 3      ┆ 2000      ┆ 26.8   ┆ 2.5 ┆ 25.7        │
│ 4      ┆ 2000      ┆ 22.6   ┆ 2.7 ┆ 21.7        │
│ 5      ┆ 2000      ┆ 19.0   ┆ 2.9 ┆ 17.6        │
│ 6      ┆ 2000      ┆ 15.5   ┆ 2.8 ┆ 13.9        │
│ 7      ┆ 2000      ┆ 10.1   ┆ 2.7 ┆ 10.3        │
│ 8      ┆ 2000      ┆ 5.9    ┆ 2.8 ┆ 7.0         │
│ 9      ┆ 2000      ┆ -1.9   ┆ 2.9 ┆ 3.0         │
│ 10     ┆ 2000      ┆ -5.0   ┆ 3.2 ┆ -4.8        │
Share of customers with a negative true uplift: 10.5%; in the bottom two deciles: 50.0%

A clean staircase from $41 to −$5. The bottom decile are the sleeping dogs: big spenders who'd have bought anyway, and for whom a coupon just lowers what they pay. Half of the bottom fifth have a negative true uplift. Couponing them costs money twice: the coupon, and the margin given away.

With a $5 coupon and 30% margin, a coupon pays above $16.67 of uplift: deciles 1–5 clearly, decile 6 roughly at break-even, deciles 7–10 not. Tomorrow turns this reasoning into a policy.

64.6 Save your work with Git (20 min)

git add .
git commit -m "Day 64: uplift metrics from scratch, four uplift models, Qini curves, deciles"
git push

Day 64 checklist

• I can explain uplift modelling and the four customer types
• I can compute uplift and Qini curves, areas and uplift-at-k by hand
• I can fit S, T, transformed-outcome and DR uplift models and compare them by Qini area
• I can produce and read a decile table, and spot sleeping dogs