Weak Instruments and LATE

38.1 Three ways IV goes wrong (30 min)

IV fixed world B, but it rests on its own assumptions. Today breaks them one at a time:

38.2 Weak instruments (60 min)

Concept. IV divides by the first stage. When the first stage is close to zero, small amounts of noise in the numerator become huge swings in the ratio. The rule of thumb (Staiger & Stock, 1997): first-stage F above 10. Recent work (Lee, McCrary, Moreira & Porter, 2022) suggests much higher, around 104, for standard 5% tests to be reliable.

world(invite_effect=...) controls how strongly the invitation shifts joining. Create day38/weak_instrument.py, which runs 60 datasets per strength:

# day38/weak_instrument.py   (python -m day38.weak_instrument)
"""
Weak instruments: what if the $20 offer barely changed who joined? 60 simulated datasets per
strength. Watch the first-stage F, the drift towards the OLS bias, and the spread of the estimates.
(We report percentiles: with very weak instruments a few estimates are absurd, e.g. +16,000.)
"""
import numpy as np
import polars as pl

from causalkit.estimators import two_sls
from causalkit.loyalty import public, truth, world

rows = []
for strength in [0.9, 0.3, 0.1, 0.05]:
    est, F, cover = [], [], 0
    for s in range(60):
        full = world(seed=4000 + s, hidden=1.0, invite_effect=strength)
        iv = two_sls(public(full).to_pandas())
        b, se = iv.params["joined"], iv.std_errors["joined"]
        late = truth(full)["LATE"]
        est.append(b - late)
        F.append(iv.first_stage.diagnostics["f.stat"].iloc[0])
        cover += abs(b - late) <= 1.96 * se
    rows.append({"invitation effect (log-odds)": strength, "median first-stage F": round(np.median(F), 1),
                 "median error": round(np.median(est), 1),
                 "middle 50% of errors": f"[{np.percentile(est, 25):.0f}, {np.percentile(est, 75):.0f}]",
                 "95% CI coverage": f"{cover / 60:.0%}"})
with pl.Config(tbl_width_chars=120):
    print(pl.DataFrame(rows))
print("For reference, OLS with controls overstates the effect by about +70 in this world.")

python -m day38.weak_instrument

│ invitation effect (log-odds) ┆ median first-stage F ┆ median error ┆ middle 50% of errors ┆ 95% CI coverage │
│ 0.9                          ┆ 607.7                ┆ 2.4          ┆ [-10, 13]            ┆ 92%             │
│ 0.3                          ┆ 71.2                 ┆ 9.2          ┆ [-29, 36]            ┆ 93%             │
│ 0.1                          ┆ 9.3                  ┆ 17.1         ┆ [-80, 104]           ┆ 98%             │
│ 0.05                         ┆ 3.0                  ┆ 41.4         ┆ [-81, 199]           ┆ 98%             │
For reference, OLS with controls overstates the effect by about +70 in this world.

With F ≈ 3, half the estimates fall in a $280-wide range, and the median error ($41) is drifting towards OLS's bias. A weak instrument doesn't just add noise: 2SLS is biased towards OLS, because the first stage starts fitting noise that's correlated with the confounder.

Why use percentiles instead of the mean and SD? With a weak first stage, some datasets divide by almost zero and produce estimates in the thousands; the mean is meaningless. (The CIs show high coverage here only because they are enormous.)

Always report the first-stage F. If it's below 10, report the reduced form (the ITT) and an Anderson–Rubin confidence set instead of a 2SLS point estimate.

38.3 The exclusion restriction (55 min)

Concept. The invitation may affect spending only by changing who joins. But the $20 bonus is money. Suppose invited customers spend part of the $20 at NorthMart even when they don't join. Then the instrument has a direct path to the outcome, and IV attributes that spending to joining.

world(bonus_spend_effect=...) adds that leak. Create day38/exclusion_violation.py:

# day38/exclusion_violation.py   (python -m day38.exclusion_violation)
"""
The exclusion restriction: the invitation may affect spend ONLY by changing who joins.
But the $20 bonus is money to spend. If invited customers spend part of it even without joining,
the instrument has a direct path to the outcome, and IV divides that leak by a small first stage.
"""
from causalkit.estimators import two_sls
from causalkit.loyalty import public, truth, world

for leak in [0, 5, 10, 20]:
    full = world(hidden=1.0, bonus_spend_effect=leak)
    iv = two_sls(public(full).to_pandas())
    print(f"Invitation adds ${leak:>2} of spend directly: 2SLS {iv.params['joined']:6.1f} "
          f"(SE {iv.std_errors['joined']:.1f})   truth LATE {truth(full)['LATE']:.1f}")
print("Bias is roughly leak / first stage = leak / 0.16, i.e. about 6x the leak.")

python -m day38.exclusion_violation

Invitation adds $ 0 of spend directly: 2SLS  141.3 (SE 13.5)   truth LATE 153.6
Invitation adds $ 5 of spend directly: 2SLS  172.1 (SE 13.3)   truth LATE 153.6
Invitation adds $10 of spend directly: 2SLS  202.9 (SE 13.1)   truth LATE 153.6
Invitation adds $20 of spend directly: 2SLS  264.6 (SE 13.2)   truth LATE 153.6
Bias is roughly leak / first stage = leak / 0.16, i.e. about 6x the leak.

A $5 leak becomes a $31 bias; a $20 leak becomes $123. The bias equals the leak divided by the first stage, so the weaker the instrument, the more any exclusion violation is amplified. And the standard error doesn't change: nothing in the output warns you.

The exclusion restriction can't be tested; it has to be argued, and ideally designed in. NorthMart could have made the bonus redeemable only as programme points, so it can't be spent outside the programme. Write in day38/NOTES.md two other ways the invitation could affect spending directly (hint: think about what an email from NorthMart does).

38.4 Whose effect is the LATE? (40 min)

Concept. The LATE is the effect for compliers: customers whose joining was decided by the invitation. Not for always-takers (the fans who join anyway) and not for never-takers. Three effects, three groups, three decisions:

In world B, the LATE ($154) is close to the ATE ($158), but the ATT is $183. That's not an accident of the method: compliers are less extreme than self-selected joiners, as the mini-project shows.

38.5 Day 38 mini-project: profile the compliers (65 min)

We can't list the compliers (we never see what a customer would have done under the other invitation status), but we can describe them. The average of any covariate X among compliers is:

( E[X · joined | invited] − E[X · joined | not invited] ) / first stage

Create day38/compliers.py:

# day38/compliers.py   (python -m day38.compliers)
"""
Who are the compliers, the customers the LATE is about? We can't list them, but we CAN describe them:
the average of any covariate X among compliers is
    (E[X * joined | invited] - E[X * joined | not invited]) / first stage.
"""
import polars as pl

from causalkit.loyalty import public, truth, world

full = world(hidden=1.0)
df = public(full)
inv, not_inv = df.filter(pl.col("invited") == 1), df.filter(pl.col("invited") == 0)
first_stage = inv["joined"].mean() - not_inv["joined"].mean()

rows = []
for col in ["pre_spend", "engaged", "tenure_years", "age"]:
    complier_mean = ((inv[col] * inv["joined"]).mean() - (not_inv[col] * not_inv["joined"]).mean()) / first_stage
    rows.append({"covariate": col,
                 "all customers": round(df[col].mean(), 2),
                 "joined": round(df.filter(pl.col("joined") == 1)[col].mean(), 2),
                 "compliers (estimated)": round(complier_mean, 2),
                 "compliers (truth)": round(full.filter(pl.col("_complier") == 1)[col].mean(), 2)})
print(pl.DataFrame(rows))
t = truth(full)
print(f"Effects: LATE {t['LATE']:.1f}   ATE {t['ATE']:.1f}   ATT {t['ATT']:.1f}")

python -m day38.compliers

│ covariate    ┆ all customers ┆ joined ┆ compliers (estimated) ┆ compliers (truth) │
│ pre_spend    ┆ 714.65        ┆ 887.98 ┆ 625.82                ┆ 668.61            │
│ engaged      ┆ 0.56          ┆ 0.67   ┆ 0.61                  ┆ 0.58              │
│ tenure_years ┆ 4.01          ┆ 4.59   ┆ 3.94                  ┆ 4.0               │
│ age          ┆ 45.19         ┆ 44.4   ┆ 44.2                  ┆ 44.84             │
Effects: LATE 153.6   ATE 158.4   ATT 183.4

Joiners spent $888 last year; compliers about $650, slightly below the average customer. The $20 invitation persuades moderate spenders, while the big spenders were joining anyway. Since the effect grows with past spending, that's why the LATE is below the ATT.

The estimates are noisy (they divide by a first stage of 0.16), but they get the direction right. Put this table in every IV report: it tells the reader whom the number describes.

Try it. Without code: if NorthMart doubled the bonus to $40, the first stage would grow. Would you expect the new LATE to be larger or smaller than $154? Why?

38.6 Save your work with Git (20 min)

git add .
git commit -m "Day 38: weak instruments, exclusion violations, complier profiles"
git push

Day 38 checklist

• I report the first-stage F with every IV estimate and know the thresholds
• I can explain why weak instruments bias 2SLS towards OLS
• I can show that an exclusion violation's bias is the leak divided by the first stage
• I can match ATT, ATE and LATE to the decisions they inform
• I can estimate the complier profile and use it to say whom a LATE describes