Overlap and Trimming

35.1 Positivity: the assumption you can see (40 min)

Concept. Overlap (positivity) requires that every kind of customer had some chance of both joining and not joining. If customers who spend over $3,000 always join, there's nobody to compare them with. Every method fails there, in its own way:

Unlike unconfoundedness, overlap can be checked: plot the propensity scores (Day 32's figure) and count customers near 0 or 1.

35.2 A world with poor overlap (60 min)

world(overlap=2.5) makes selection 2.5 times sharper: the same customers, but joining is much more predictable. Create day35/poor_overlap.py:

# day35/poor_overlap.py   (python -m day35.poor_overlap)
"""
Poor overlap: when the covariates almost DETERMINE who joins, some customers have no comparable
counterpart in the other group. Same customers, but joining is 2.5x more predictable.
"""
import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np

from causalkit.estimators import effective_sample_size, ipw, ipw_weights, propensity
from causalkit.loyalty import public, truth, world

for label, k in [("normal overlap", 1.0), ("poor overlap", 2.5)]:
    full = world(overlap=k)
    df = public(full).to_pandas()
    ps = propensity(df)
    d = df.joined.to_numpy()
    w = ipw_weights(d, ps, "ATE")
    print(f"{label}: propensity range [{ps.min():.4f}, {ps.max():.4f}], "
          f"share below 0.05 or above 0.95: {np.mean((ps < 0.05) | (ps > 0.95)):.1%}")
    print(f"   largest ATE weight {w.max():,.0f};  effective sample size {effective_sample_size(w):,.0f} of {len(w):,}")
    print(f"   IPW ATE {ipw(df, ps)['ATE']:.1f}   truth {truth(full)['ATE']:.1f}")

full = world(overlap=2.5)
df = public(full).to_pandas()
ps = propensity(df)
fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(ps[df.joined == 0], bins=60, alpha=0.6, label="did not join", color="#d95926")
ax.hist(ps[df.joined == 1], bins=60, alpha=0.6, label="joined", color="#3987e5")
ax.set_xlabel("Estimated propensity to join"); ax.set_ylabel("Customers"); ax.legend()
ax.set_title("Poor overlap: piles near 0 and 1 with few comparisons")
fig.tight_layout(); fig.savefig("figures/day35_poor_overlap.png", dpi=150)
print("Saved figures/day35_poor_overlap.png")

python -m day35.poor_overlap

normal overlap: propensity range [0.0127, 0.9724], share below 0.05 or above 0.95: 0.4%
   largest ATE weight 23;  effective sample size 14,549 of 20,000
   IPW ATE 176.1   truth 158.4
poor overlap: propensity range [0.0000, 0.9996], share below 0.05 or above 0.95: 30.6%
   largest ATE weight 404;  effective sample size 2,273 of 20,000
   IPW ATE 205.9   truth 158.4
Saved figures/day35_poor_overlap.png

With poor overlap, almost a third of customers have scores below 0.05 or above 0.95, one customer carries a weight of 404, and 20,000 customers are worth 2,273. Open the figure: the two histograms barely touch.

Rule of thumb. If more than a few percent of scores are outside [0.05, 0.95], or the largest weight is a large share of the total, the ATE for everyone is not identified well by these data. Change the question before you change the estimator.

35.3 Three remedies, three estimands (50 min)

All three buy precision by answering a different question. That's fine, as long as the report says so. "The programme raised spending by $160 among customers whose decision to join wasn't a foregone conclusion" is honest. Calling the same number "the effect" is not.

35.4 Who gets trimmed? (35 min)

Try it. In the poor-overlap world, trim to [0.05, 0.95]. Compare the true ATE for everyone with the true ATE for the customers you kept (use the hidden _tau column: we're grading, not analysing). Then compare the kept and dropped customers' average pre_spend and share engaged. Create day35/who_is_trimmed.py.

# day35/who_is_trimmed.py   (python -m day35.who_is_trimmed)
import numpy as np

from causalkit.estimators import propensity
from causalkit.loyalty import public, world

full = world(overlap=2.5)
df = public(full).to_pandas()
ps = propensity(df)
keep = (ps >= 0.05) & (ps <= 0.95)
tau = full["_tau"].to_numpy()
print(f"True ATE, everyone:            {tau.mean():.1f}")
print(f"True ATE, kept after trimming: {tau[keep].mean():.1f}")
print(f"Average pre_spend, kept {df.pre_spend[keep].mean():.0f} vs dropped {df.pre_spend[~keep].mean():.0f}")
print(f"Share engaged, kept {df.engaged[keep].mean():.2f} vs dropped {df.engaged[~keep].mean():.2f}")

True ATE, everyone:            158.4
True ATE, kept after trimming: 161.9
Average pre_spend, kept 718 vs dropped 707
Share engaged, kept 0.64 vs dropped 0.39

35.5 Day 35 mini-project: trimming rules compared (65 min)

Create day35/trimming.py. Over 50 poor-overlap datasets, each estimate is compared with the true effect for the population it describes:

# day35/trimming.py   (python -m day35.trimming)
"""
Mini-project: trimming (dropping customers with propensity outside [a, 1 - a]) and overlap weights.
Both trade the estimand for precision: each estimate is compared with the truth for ITS population.
50 simulated datasets with poor overlap.
"""
import numpy as np
import polars as pl

from causalkit.estimators import ipw, propensity
from causalkit.loyalty import public, world

RULES = [0.0, 0.01, 0.05, 0.10]
res = {a: {"err": [], "kept": []} for a in RULES}
ato = {"err": []}
for s in range(50):
    full = world(seed=2000 + s, overlap=2.5)
    df = public(full).to_pandas()
    ps = propensity(df)
    tau = full["_tau"].to_numpy()
    for a in RULES:
        keep = (ps >= a) & (ps <= 1 - a)
        est = ipw(df[keep].reset_index(drop=True), ps[keep])["ATE"]
        res[a]["err"].append(est - tau[keep].mean())          # truth for the trimmed population
        res[a]["kept"].append(keep.mean())
    # Overlap weights: joiners weighted by 1 - ps, non-joiners by ps (estimand: the 'overlap population')
    d, y = df.joined.to_numpy(), df.spend.to_numpy()
    est = np.average(y[d == 1], weights=1 - ps[d == 1]) - np.average(y[d == 0], weights=ps[d == 0])
    ato["err"].append(est - np.average(tau, weights=ps * (1 - ps)))

rows = [{"rule": "no trimming" if a == 0 else f"trim to [{a}, {1 - a}]",
         "customers kept": f"{np.mean(r['kept']):.0%}", "average error": round(np.mean(r["err"]), 1),
         "SD of error": round(np.std(r["err"]), 1)} for a, r in res.items()]
rows.append({"rule": "overlap weights (ATO)", "customers kept": "100%",
             "average error": round(np.mean(ato["err"]), 1), "SD of error": round(np.std(ato["err"]), 1)})
with pl.Config(fmt_str_lengths=30):
    print(pl.DataFrame(rows))

python -m day35.trimming

│ rule                  ┆ customers kept ┆ average error ┆ SD of error │
│ no trimming           ┆ 100%           ┆ 2.4           ┆ 116.1       │
│ trim to [0.01, 0.99]  ┆ 90%            ┆ 5.2           ┆ 16.6        │
│ trim to [0.05, 0.95]  ┆ 70%            ┆ 0.3           ┆ 8.7         │
│ trim to [0.1, 0.9]    ┆ 56%            ┆ -1.2          ┆ 7.0         │
│ overlap weights (ATO) ┆ 100%           ┆ 0.6           ┆ 3.5         │

• Untrimmed IPW is unbiased on average and useless in practice: SD $116.
• Trimming at 0.05 or 0.1 cuts the SD by more than 90%, while dropping 30–44% of customers.
• Overlap weights have the smallest error of all and drop nobody. Their weights are bounded between 0 and 1, so they can't explode. The price is an estimand (ATO) that emphasises customers near e(X) = 0.5.

Write in day35/NOTES.md: if the director asks "what's the effect of the programme?" in the poor-overlap world, what would you report, and how would you word the population it applies to?

35.6 Save your work with Git (20 min)

git add .
git commit -m "Day 35: overlap diagnostics, trimming and overlap weights"
git push

Day 35 checklist

• I check overlap before estimating: propensity ranges, extreme shares, largest weight, ESS
• I can explain how g-computation, matching and IPW each fail without overlap
• I can trim or use overlap weights, and name the estimand each one targets
• I can describe who was trimmed, and whether that matters for the decision