Inverse Probability Weighting

34.1 Weighting creates a pseudo-population (45 min)

Concept. A customer with propensity 0.2 who joined is unusual: four in five customers like them didn't join. In the data they stand in for the missing joiners like them, so give them weight 1/0.2 = 5. A non-joiner with propensity 0.2 is typical: weight 1/0.8 = 1.25. After weighting, each group looks like the whole population, as if joining had been randomised. This is inverse probability weighting (IPW), the survey statistician's idea from Day 12 applied to treatment.

Type ipw_weights, ipw and effective_sample_size into estimators.py.

34.2 Horvitz–Thompson and Hajek estimators (50 min)

There are two ways to turn weights into an average:

• Horvitz–Thompson (HT): weighted sum divided by the number of customers. Unbiased in theory, but if the weights happen to sum to more than n, the whole estimate is inflated.
• Hajek: weighted sum divided by the sum of the weights: an ordinary weighted mean. Slightly biased in small samples, usually much less variable. This is the default in practice.

Create day34/ipw.py:

# day34/ipw.py   (python -m day34.ipw)
"""
Inverse probability weighting: a joiner with a 25% chance of joining stands in for four customers
like them. Re-weighted, each group looks like the population the estimand is about.
"""
import numpy as np

from causalkit.estimators import effective_sample_size, ipw, ipw_weights, propensity
from causalkit.loyalty import public, truth, world

full = world()
df = public(full).to_pandas()
t = truth(full)
ps = propensity(df)
d = df.joined.to_numpy()

for name, normalised in [("Horvitz-Thompson", False), ("Hajek (normalised)", True)]:
    e = ipw(df, ps, normalised=normalised)
    print(f"{name:<20} ATE {e['ATE']:7.1f}   ATT {e['ATT']:7.1f}")
print(f"{'Truth':<20} ATE {t['ATE']:7.1f}   ATT {t['ATT']:7.1f}")

w = ipw_weights(d, ps, "ATE")
print(f"\nATE weights: min {w.min():.2f}, median {np.median(w):.2f}, max {w.max():.1f}")
print(f"Effective sample size: {effective_sample_size(w[d == 1]):,.0f} of {(d == 1).sum():,} joiners, "
      f"{effective_sample_size(w[d == 0]):,.0f} of {(d == 0).sum():,} non-joiners")

python -m day34.ipw

Horvitz-Thompson     ATE   180.8   ATT   229.4
Hajek (normalised)   ATE   176.1   ATT   223.5
Truth                ATE   158.4   ATT   186.0

ATE weights: min 1.01, median 1.61, max 23.4
Effective sample size: 6,760 of 9,630 joiners, 7,879 of 10,370 non-joiners

Both are well off: +$18 on the ATE, +$37 on the ATT. G-computation got within $2 on the same data. Before blaming the method, look at the weights.

34.3 Weighted balance and the effective sample size (50 min)

Concept. The effective sample size (Kish): ESS = (Σw)² / Σw². With equal weights it equals n. When a few customers carry large weights, the estimate rests mostly on them, and the ESS falls. Here 9,630 joiners are "worth" 6,760 equally weighted ones: about 30% of the information is lost to weighting.

Create day34/weighted_balance.py:

# day34/weighted_balance.py   (python -m day34.weighted_balance)
"""Do the weights balance the covariates? Weighted standardised mean differences, ATE and ATT weights."""
import polars as pl

from causalkit.estimators import COVARIATES, ipw_weights, propensity, smd
from causalkit.loyalty import public, world

df = public(world()).to_pandas()
ps = propensity(df)
d = df.joined.to_numpy()
t, c = d == 1, d == 0

rows = []
for col in COVARIATES:
    x = df[col]
    w_ate, w_att = ipw_weights(d, ps, "ATE"), ipw_weights(d, ps, "ATT")
    rows.append({"covariate": col,
                 "unweighted": round(smd(x[t], x[c]), 3),
                 "ATE weights": round(smd(x[t], x[c], w_ate[t], w_ate[c]), 3),
                 "ATT weights": round(smd(x[t], x[c], w_att[t], w_att[c]), 3)})
print(pl.DataFrame(rows))

python -m day34.weighted_balance

│ covariate    ┆ unweighted ┆ ATE weights ┆ ATT weights │
│ pre_spend    ┆ 0.649      ┆ 0.024       ┆ 0.051       │
│ pre_orders   ┆ 0.595      ┆ 0.013       ┆ 0.03        │
│ engaged      ┆ 0.465      ┆ -0.003      ┆ 0.001       │
│ tenure_years ┆ 0.553      ┆ 0.003       ┆ 0.0         │
│ age          ┆ -0.097     ┆ -0.003      ┆ -0.007      │
│ mobile       ┆ 0.017      ┆ 0.006       ┆ 0.011       │

Weighted balance is excellent. So the propensity model is fine, and the estimate is still $18 off. Is IPW biased here, or did this dataset just get unlucky?

34.4 Is IPW biased, or just noisy? (40 min)

One dataset can't tell you. The simulator can: draw 100 datasets from the same world and look at the distribution of errors. Create day34/ipw_noise.py:

# day34/ipw_noise.py   (python -m day34.ipw_noise)
"""
Mini-project: is IPW wrong, or just noisy? Simulate 100 loyalty datasets (new customers each time)
and compare the error of IPW and g-computation against each dataset's own truth.
"""
import numpy as np
import polars as pl

from causalkit.estimators import g_computation, ipw
from causalkit.loyalty import public, truth, world

errors = {"IPW (Hajek)": [], "g-computation": []}
for s in range(100):
    full = world(seed=1000 + s)
    df = public(full).to_pandas()
    ate = truth(full)["ATE"]
    errors["IPW (Hajek)"].append(ipw(df)["ATE"] - ate)
    errors["g-computation"].append(g_computation(df)["ATE"] - ate)

print(pl.DataFrame([{"estimator": k, "average error (bias)": round(np.mean(v), 2),
                     "SD of error": round(np.std(v), 2), "worst error": round(max(v, key=abs), 1)}
                    for k, v in errors.items()]))

python -m day34.ipw_noise

│ estimator     ┆ average error (bias) ┆ SD of error ┆ worst error │
│ IPW (Hajek)   ┆ 1.17                 ┆ 17.37       ┆ -132.2      │
│ g-computation ┆ 0.08                 ┆ 1.98        ┆ -5.0        │

IPW is essentially unbiased (+$1 on average), but its standard deviation is nine times g-computation's, and one dataset in a hundred was $132 off. Our dataset's +$18 error was one ordinary draw from that wide distribution.

Why so noisy? Spending is very spread out (some customers spend $5,000), and IPW multiplies each customer's spending by their weight. One big spender with a weight of 20 moves the average by a lot. G-computation uses a model to smooth over those customers; IPW uses their raw spending.

The lesson is about method, not just IPW: when one estimate surprises you, simulate before you conclude. A single draw can't distinguish bias from noise.

34.5 Day 34 mini-project: Horvitz–Thompson vs Hajek (65 min)

Repeat 34.4 for both the HT and Hajek ATE estimators, using the same 100 seeds (world(seed=1000 + s)). Report the average error and SD of each. Create day34/ht_vs_hajek.py.

# day34/ht_vs_hajek.py   (python -m day34.ht_vs_hajek)
import numpy as np

from causalkit.estimators import ipw, propensity
from causalkit.loyalty import public, truth, world

errs = {"Horvitz-Thompson": [], "Hajek": []}
for s in range(100):
    full = world(seed=1000 + s)
    df = public(full).to_pandas()
    ps, ate = propensity(df), truth(full)["ATE"]
    errs["Horvitz-Thompson"].append(ipw(df, ps, normalised=False)["ATE"] - ate)
    errs["Hajek"].append(ipw(df, ps, normalised=True)["ATE"] - ate)
for k, v in errs.items():
    print(f"{k:<17} average error {np.mean(v):+6.1f}   SD {np.std(v):5.1f}")

Horvitz-Thompson  average error   +0.2   SD  18.7
Hajek             average error   +1.2   SD  17.4

34.6 Save your work with Git (20 min)

git add .
git commit -m "Day 34: IPW (HT and Hajek), weighted balance, ESS, Monte Carlo check"
git push

Day 34 checklist

• I can write IPW weights for the ATE and the ATT and explain the pseudo-population
• I can explain the difference between Horvitz–Thompson and Hajek estimators
• I report weighted balance and the effective sample size with every IPW estimate
• I use simulation to tell bias from noise before trusting a surprising estimate