Matching

33.1 Matching as a constructed control group (40 min)

Concept. For each joiner, find the non-joiner who looks most like them and use that person's spending as the joiner's "would-have-spent" value. Average the gaps and you have the ATT: the effect for the people who actually joined. Matching is popular with decision-makers because the comparison is concrete: "customers like Ana who didn't join spent $X".

The choices that matter:

Type MATCH_VARS and match_att into estimators.py. Mahalanobis distance rescales covariates so that each counts in units of its own spread, and accounts for their correlation; after the Cholesky transform it is plain Euclidean distance, so scikit-learn's NearestNeighbors does the search.

33.2 Four ways to match (60 min)

Create day33/matching.py:

# day33/matching.py   (python -m day33.matching)
"""For every joiner, find the most similar non-joiner; compare their spending. Four versions."""
import polars as pl

from causalkit.estimators import match_att
from causalkit.loyalty import public, truth, world

full = world()
df = public(full).to_pandas()
t = truth(full)

rows = []
for name, kw in [("propensity score", {"on": "ps"}),
                 ("propensity score, caliper 0.05 SD", {"on": "ps", "caliper": 0.05}),
                 ("Mahalanobis on covariates", {"on": "mahalanobis"}),
                 ("Mahalanobis + bias correction", {"on": "mahalanobis", "bias_correct": True})]:
    r = match_att(df, **kw)
    rows.append({"matching on": name, "ATT": round(r["ATT"], 1), "joiners matched": r["matched_treated"],
                 "distinct non-joiners used": r["distinct_controls_used"]})
with pl.Config(fmt_str_lengths=40, tbl_width_chars=110):
    print(pl.DataFrame(rows))
print(f"Truth: ATT {t['ATT']:.1f}")

python -m day33.matching

│ matching on                       ┆ ATT   ┆ joiners matched ┆ distinct non-joiners used │
│ propensity score                  ┆ 203.6 ┆ 9630            ┆ 4424                      │
│ propensity score, caliper 0.05 SD ┆ 189.2 ┆ 9547            ┆ 4424                      │
│ Mahalanobis on covariates         ┆ 229.1 ┆ 9630            ┆ 4726                      │
│ Mahalanobis + bias correction     ┆ 185.2 ┆ 9630            ┆ 4726                      │
Truth: ATT 186.0

• Plain matching overstates the ATT by $18–43, even with 20,000 customers.
• The caliper drops 83 joiners who had no close match. The estimate improves, but notice the cost: it is now the effect for 9,547 joiners, a slightly different population. Every caliper changes the estimand a little.
• Only ~4,500 distinct non-joiners serve as matches for 9,630 joiners: some are reused many times. Matching with replacement trades variance for less bias.
• Bias correction almost removes the error ($185.2 vs $186.0). It fits an outcome model on non-joiners and uses it to adjust each pair for the gap the match left. That's matching and g-computation combined, a preview of Day 36's idea of using two models at once.

33.3 Judging a match: balance and the Love plot (55 min)

Concept. You judge a match by balance, never by the estimate it gives. Standard practice: every covariate's SMD below 0.1 after matching, shown in a Love plot (one dot per covariate, before and after).

Create day33/balance_after_matching.py:

# day33/balance_after_matching.py   (python -m day33.balance_after_matching)
"""
Mini-project: did matching balance the covariates? Standardised mean differences before and after,
for propensity-score and Mahalanobis matching, and a 'love plot' of the result.
"""
import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
import polars as pl

from causalkit.estimators import COVARIATES, match_att, smd
from causalkit.loyalty import public, world

df = public(world()).to_pandas()

pairs = {name: match_att(df, on=on)["_pairs"] for name, on in [("PS matching", "ps"), ("Mahalanobis", "mahalanobis")]}
rows = []
for col in COVARIATES:
    row = {"covariate": col,
           "before": round(smd(df[col][df.joined == 1], df[col][df.joined == 0]), 3)}
    for name, (t, c) in pairs.items():
        row[name] = round(smd(df[col].iloc[t], df[col].iloc[c]), 3)
    rows.append(row)
table = pl.DataFrame(rows)
print(table)

fig, ax = plt.subplots(figsize=(6.5, 3.8))
y = np.arange(len(COVARIATES))
for col, marker in [("before", "o"), ("PS matching", "s"), ("Mahalanobis", "^")]:
    ax.scatter(table[col].abs(), y, marker=marker, label=col, s=45)
ax.axvline(0.1, color="grey", linestyle="--", linewidth=1)
ax.set_yticks(y, COVARIATES); ax.set_xlabel("|standardised mean difference|"); ax.legend()
ax.set_title("Balance before and after matching")
fig.tight_layout(); fig.savefig("figures/day33_love_plot.png", dpi=150)
print("Saved figures/day33_love_plot.png")

python -m day33.balance_after_matching

│ covariate    ┆ before ┆ PS matching ┆ Mahalanobis │
│ pre_spend    ┆ 0.649  ┆ 0.014       ┆ 0.051       │
│ pre_orders   ┆ 0.595  ┆ 0.014       ┆ 0.025       │
│ engaged      ┆ 0.465  ┆ -0.002      ┆ 0.0         │
│ tenure_years ┆ 0.553  ┆ 0.009       ┆ 0.009       │
│ age          ┆ -0.097 ┆ 0.024       ┆ -0.001      │
│ mobile       ┆ 0.017  ┆ 0.006       ┆ -0.005      │
Saved figures/day33_love_plot.png

By the usual rule, both matches pass easily: every SMD is under 0.06.

33.4 Why good balance can still give a biased answer (30 min)

Yet PS matching overstates the ATT by $18. How?

The SMD compares means. The outcome depends on pre_spend²: next year's spending grows faster than linearly with last year's. Two groups can have the same mean pre_spend but different spreads, and then they have different means of pre_spend², and different average spending even without the programme.

Two lessons:

1. Check balance on the functions of covariates the outcome depends on (squares, logs, interactions), and on spreads, not just means.
2. Matching's leftover gaps are where bias correction (or Day 36's AIPW) earns its keep.

33.5 Day 33 mini-project: check the whole distribution (65 min)

Compute, before and after PS matching, the SMD of pre_spend, the SMD of pre_spend², and the variance ratio of pre_spend (joiners' variance divided by matched non-joiners'; good balance is between 0.8 and 1.25). Create day33/variance_ratio.py. Use match_att(df, on="ps")["_pairs"] to get the matched rows.

# day33/variance_ratio.py   (python -m day33.variance_ratio)
import numpy as np

from causalkit.estimators import match_att, smd
from causalkit.loyalty import public, world

df = public(world()).to_pandas()
t, c = match_att(df, on="ps")["_pairs"]
x = df.pre_spend
for name, a, b in [("before matching", x[df.joined == 1], x[df.joined == 0]),
                   ("after PS matching", x.iloc[t], x.iloc[c])]:
    print(f"{name:<18} SMD of pre_spend {smd(a, b):+.3f}   SMD of pre_spend^2 {smd(a**2, b**2):+.3f}"
          f"   variance ratio {np.var(a) / np.var(b):.2f}")

before matching    SMD of pre_spend +0.649   SMD of pre_spend^2 +0.412   variance ratio 3.02
after PS matching  SMD of pre_spend +0.014   SMD of pre_spend^2 +0.038   variance ratio 1.18

33.6 Save your work with Git (20 min)

git add .
git commit -m "Day 33: PS and Mahalanobis matching, calipers, bias correction, Love plot"
git push

Coming next (Days 34–36). Matching throws away most non-joiners and reuses a few. Weighting keeps everyone. Days 34–36 build inverse probability weighting, confront its main weakness (extreme weights when overlap is poor), and then combine an outcome model with weights in the doubly robust estimator, AIPW.

Day 33 checklist

• I can match on the propensity score or on Mahalanobis distance, with replacement and a caliper
• I can explain how a caliper changes the population the estimate describes
• I judge a match by covariate balance and a Love plot, not by its estimate
• I check balance on squares and spreads, not just means
• I can explain what bias-corrected matching adds