Colliders and Selection Bias

16.1 Conditioning on a collider (40 min)

Concept. Day 14 showed that controlling for a collider creates association. "Controlling" includes more than adding a regression variable. You also condition on a variable when you:

• filter the sample on it (WHERE active = 1),
• stratify by it (report results by segment),
• only have data for some of its values (only customers who answered a survey).

When the variable is a collider (or a descendant of one) between treatment and outcome, any of these produces selection bias, even if treatment was randomised. Unlike confounding, no control variable fixes it: the damage is done by who is in the data.

16.2 Berkson's paradox in a VIP programme (40 min)

Story. NorthMart's VIP status goes to customers who spend a lot or engage a lot with emails. Among VIPs, an analyst notices that heavy spenders open fewer emails, and concludes "email engagement cannibalises spending".

Create day16/berkson.py:

# day16/berkson.py   (python -m day16.berkson)
"""Berkson's paradox: two unrelated traits look related once you only study a selected group."""
import numpy as np
import polars as pl

rng = np.random.default_rng(16)
n = 50_000
spend = rng.normal(size=n)          # how much a customer spends
engagement = rng.normal(size=n)     # how often they open emails: unrelated to spend
vip = (spend + engagement + rng.normal(scale=0.5, size=n)) > 1.5   # VIP status needs either

df = pl.DataFrame({"spend": spend, "engagement": engagement, "vip": vip})
print(f"All customers:   corr(spend, engagement) = {df.select(pl.corr('spend', 'engagement')).item():+.3f}")
vips = df.filter(pl.col("vip"))
print(f"VIPs only ({vips.height:,}): corr(spend, engagement) = {vips.select(pl.corr('spend', 'engagement')).item():+.3f}")

python -m day16.berkson

All customers:   corr(spend, engagement) = +0.001
VIPs only (7,875): corr(spend, engagement) = -0.543

Spend and engagement are unrelated by construction. VIP status is their collider (spend → vip ← engagement): a VIP with low spend must have high engagement to have qualified. Studying VIPs only manufactures a strong negative correlation.

Try it. Raise the VIP threshold from 1.5 to 2.5. Does the spurious correlation get weaker or stronger? Why?

16.3 "Active customers only": collider bias inside an experiment (70 min)

Story. An analyst says: "Inactive customers just add noise. Let's measure the email's effect only on customers who ordered at least once after it was sent."

Being active after the email is caused by the email and by shopping intensity: email → active ← intensity → orders. Filtering on it opens a path from email to orders through intensity. To isolate this bias from targeting, we randomise the email (via the truth file) 1,000 times.

Create day16/active_customers_only.py:

# day16/active_customers_only.py   (python -m day16.active_customers_only)
"""
'Let's only look at customers who were active after the email.'
Activity after the email is CAUSED by the email and by shopping intensity: a collider.
We randomise the email 1,000 times (using the truth file), so any bias is pure selection.
"""
import numpy as np
import polars as pl

po = pl.read_parquet("data/truth/potential_outcomes.parquet")
y0, y1, mu0 = po["y0"].to_numpy(), po["y1"].to_numpy(), po["mu0"].to_numpy()
rng = np.random.default_rng(16)

everyone, active_only, intensity_gap, n_active = [], [], [], []
for _ in range(1000):
    z = np.zeros(po.height, dtype=bool)
    z[rng.choice(po.height, size=1279, replace=False)] = True
    y = np.where(z, y1, y0)
    a = y > 0                                             # "active after the email"
    everyone.append(y[z].mean() - y[~z].mean())
    active_only.append(y[z & a].mean() - y[~z & a].mean())
    intensity_gap.append(mu0[z & a].mean() - mu0[~z & a].mean())
    n_active.append(a.sum())

print(f"True ATE                                   {(y1 - y0).mean():+.3f}")
print(f"Everyone, average of 1,000 experiments     {np.mean(everyone):+.3f}")
print(f"Active customers only, average             {np.mean(active_only):+.3f}   (~{np.mean(n_active):,.0f} customers)")
print(f"Intensity gap among the active (emailed - not): {np.mean(intensity_gap):+.3f} expected orders")

python -m day16.active_customers_only

True ATE                                   +0.319
Everyone, average of 1,000 experiments     +0.321
Active customers only, average             +0.263   (~3,137 customers)
Intensity gap among the active (emailed - not): -0.037 expected orders

Even with a perfect experiment, the filter cuts the estimated effect by 18%. The mechanism is in the last line: among active customers, the emailed ones are slightly lighter shoppers. The email pulled some light shoppers into activity; without it, only heavier shoppers would have made the cut. The comparison is no longer like for like.

This is why Module 1 kept customers with zero orders (Days 2–3): zero is an outcome, and dropping it is conditioning on a collider.

16.4 Where colliders hide in business data (35 min)

The rule: never filter on, stratify by, or control for anything the treatment can affect. Filtering on pre-treatment characteristics changes who you study (the estimand's population), which is fine when you say so.

16.5 Day 16 mini-project: audit four sample filters (70 min)

Goal. For four filters an analyst might apply, compute the average estimate across 1,000 randomised replays, and compare it with the true effect for the population the filter selects. Classify each filter as safe (changes the population, no bias) or collider (biased).

Create day16/selection_audit.py:

# day16/selection_audit.py   (python -m day16.selection_audit)
"""
Mini-project: audit three sample filters an analyst might apply.
A filter on something measured BEFORE the email changes WHO you study (fine, if you say so).
A filter on something the email can AFFECT creates collider bias.
"""
import numpy as np
import polars as pl

from causalkit.data import load_customers

po = pl.read_parquet("data/truth/potential_outcomes.parquet").select("customer_id", "y0", "y1")
df = load_customers().select("customer_id", "pre_orders").join(po, on="customer_id")
y0, y1 = df["y0"].to_numpy(), df["y1"].to_numpy()
pre_active = df["pre_orders"].to_numpy() > 0
rng = np.random.default_rng(160)

FILTERS = {
    "everyone": lambda y: np.ones_like(y, dtype=bool),
    "ordered BEFORE the email (pre_orders > 0)": lambda y: pre_active,
    "ordered AFTER the email (post_orders > 0)": lambda y: y > 0,
    "heavy buyers after the email (post_orders >= 3)": lambda y: y >= 3,
}
results = {name: [] for name in FILTERS}
for _ in range(1000):
    z = np.zeros(df.height, dtype=bool)
    z[rng.choice(df.height, size=1279, replace=False)] = True
    y = np.where(z, y1, y0)
    for name, rule in FILTERS.items():
        keep = rule(y)
        results[name].append(y[z & keep].mean() - y[~z & keep].mean())

rows = []
for name, est in results.items():
    target = (y1 - y0)[pre_active].mean() if "BEFORE" in name else (y1 - y0).mean()
    rows.append({"sample": name, "average estimate": round(np.mean(est), 3),
                 "true effect for that population": round(target, 3)})
with pl.Config(fmt_str_lengths=60, tbl_width_chars=120):
    print(pl.DataFrame(rows))

python -m day16.selection_audit

│ sample                                          ┆ average estimate ┆ true effect for that population │
│ everyone                                        ┆ 0.317            ┆ 0.318                           │
│ ordered BEFORE the email (pre_orders > 0)       ┆ 0.352            ┆ 0.355                           │
│ ordered AFTER the email (post_orders > 0)       ┆ 0.26             ┆ 0.318                           │
│ heavy buyers after the email (post_orders >= 3) ┆ 0.168            ┆ 0.318                           │

• Pre-period filter: safe. The estimate (0.352) matches the true effect for pre-period buyers (0.355). It's a different estimand, a CATE for a subgroup, and it's correctly estimated.
• Post-period filters: collider bias, growing with selectivity: −18% for "any order", −47% for "three or more".

Write your conclusions in docs/ESTIMANDS.md under "Sample filters": one line per filter, with its verdict and the reason.

16.6 Save your work with Git (15 min)

git add .
git commit -m "Day 16: colliders, Berkson's paradox, selection audit"
git push

Day 16 checklist

• I can explain why filtering, stratifying or limited data can all mean conditioning
• I can recognise Berkson's paradox and explain it with a DAG
• I can show collider bias inside a randomised experiment
• I can tell a safe pre-treatment filter from a biased post-treatment one