Simpson's Paradox in Business Data

19.1 When the total disagrees with every part (40 min)

Concept. Simpson's paradox: a relationship that holds in every subgroup reverses (or vanishes) when the subgroups are combined. Famous cases include university admissions (each department admitted women at similar or higher rates, yet overall rates favoured men) and medical treatments that look better in every patient group but worse overall.

It's not a statistical trick. It happens whenever the subgroups differ both in how often they're treated and in their baseline outcome. The interesting question isn't why it happens but which table to believe, and the data alone can't answer that.

19.2 Same numbers, two stories (60 min)

Create day19/simpson_two_stories.py:

# day19/simpson_two_stories.py   (python -m day19.simpson_two_stories)
"""
The same four numbers, two different stories. The data cannot tell you which table to believe;
the DAG can.
"""
import polars as pl

# Free-shipping offer: conversions out of visitors, split by a third variable
counts = pl.DataFrame({
    "group":       ["A", "A", "B", "B"],
    "offer":       [1, 0, 1, 0],
    "visitors":    [800, 200, 200, 800],
    "conversions": [160, 30, 80, 280],
})


def rates(df: pl.DataFrame, by: list[str]) -> pl.DataFrame:
    return (df.group_by(by).agg(pl.col("conversions").sum(), pl.col("visitors").sum())
              .with_columns(rate=(pl.col("conversions") / pl.col("visitors")).round(3)).sort(by))


print("Pooled:")
print(rates(counts, ["offer"]))
print("By group:")
print(rates(counts, ["group", "offer"]))

pooled = rates(counts, ["offer"])["rate"]
print(f"\nPooled effect of the offer:    {pooled[1] - pooled[0]:+.3f}")
for g in ["A", "B"]:
    r = rates(counts.filter(pl.col("group") == g), ["offer"])["rate"]
    print(f"Effect within group {g}:        {r[1] - r[0]:+.3f}")

# Standardisation: average the within-group effects, weighting by each group's share of ALL visitors
share = counts.group_by("group").agg(pl.col("visitors").sum()).with_columns(w=pl.col("visitors") / 2000)
eff = {g: rates(counts.filter(pl.col("group") == g), ["offer"])["rate"].diff()[1] for g in ["A", "B"]}
standardised = sum(eff[g] * share.filter(pl.col("group") == g)["w"].item() for g in ["A", "B"])
print(f"Standardised effect:           {standardised:+.3f}")

print("""
Story 1 - group = REGION (A = low-converting region). Marketing showed the offer mostly in A.
          region -> offer, region -> conversion: a confounder.  Believe the BY-GROUP table.
Story 2 - group = DELIVERY SPEED CHOSEN AT CHECKOUT (A = standard). The offer pushed buyers
          to standard delivery: offer -> group -> conversion, a mediator. Believe the POOLED table.""")

python -m day19.simpson_two_stories

Pooled:
│ offer ┆ conversions ┆ visitors ┆ rate │
│ 0     ┆ 310         ┆ 1000     ┆ 0.31 │
│ 1     ┆ 240         ┆ 1000     ┆ 0.24 │
By group:
│ group ┆ offer ┆ conversions ┆ visitors ┆ rate │
│ A     ┆ 0     ┆ 30          ┆ 200      ┆ 0.15 │
│ A     ┆ 1     ┆ 160         ┆ 800      ┆ 0.2  │
│ B     ┆ 0     ┆ 280         ┆ 800      ┆ 0.35 │
│ B     ┆ 1     ┆ 80          ┆ 200      ┆ 0.4  │

Pooled effect of the offer:    -0.070
Effect within group A:        +0.050
Effect within group B:        +0.050
Standardised effect:           +0.050

The offer raises conversion by 5 points in both groups, yet pooled it appears to lower conversion by 7 points. The reason is in the visitor counts: 80% of offer visitors are in group A, where everyone converts less.

Now the two stories, with identical numbers:

In Story 2 the offer really does reduce conversion overall: it works on everyone, but by changing their delivery choice it also triggers a mechanism that hurts. Stratifying by delivery choice would hide that harm.

19.3 Standardisation: averaging the parts properly (45 min)

Concept. When the subgroup variable is a confounder, the overall effect is the within-group effects averaged over the whole population's mix of groups, not the treated group's mix. This is standardisation (also called the adjustment formula, or g-formula):

effect = Σ over groups g:  (effect within g) × (share of all units in g)

In the script, both groups are 50% of visitors and both effects are +0.05, so the standardised effect is +0.05. With different within-group effects, the weights would matter, and choosing whose mix to weight by is choosing the estimand: weight by all units for the ATE, by treated units for the ATT.

Standardisation is backdoor adjustment written as arithmetic. Regression with group dummies does the same thing approximately, with weights you don't choose.

Try it. Change group A's offer conversions from 160 to 120 (so the offer lowers conversion in A) and rerun. What are the within-group effects, the pooled effect, and the standardised effect?

19.4 Simpson's paradox as a DAG question (40 min)

The resolution, due to Judea Pearl: stratify by a variable if and only if it's in a valid adjustment set.

• If the third variable is a confounder (it causes the treatment and the outcome): stratify.
• If it's a mediator or a collider (the treatment causes it): don't.
• If it's a pre-treatment collider (M-bias, Day 15): don't, even though it comes first in time.

No statistical test distinguishes Story 1 from Story 2: the counts are identical. What distinguishes them is when the group variable was determined, and by what. That's knowledge about the process, written down as a DAG. This is the strongest argument for drawing DAGs before looking at results: once you've seen that the pooled table says "−7" and the stratified table says "+5", it's very tempting to pick the story that gives the answer you hoped for.

19.5 Day 19 mini-project: NorthMart's segments (70 min)

Goal. NorthMart's naive email comparison, pooled and by loyalty segment, has a milder version of the paradox. Explain it, then decide whether stratifying by loyalty fixes the bias.

Create day19/northmart_segments.py:

# day19/northmart_segments.py   (python -m day19.northmart_segments)
"""
Mini-project: NorthMart's naive email comparison, pooled and by loyalty segment.
The pooled gap is LARGER than the gap in either segment. Why, and which number is closer to the truth?
"""
import polars as pl

from causalkit.data import load_customers

truth = pl.read_parquet("data/truth/potential_outcomes.parquet").select("customer_id", "tau")
df = load_customers().join(truth, on="customer_id")


def naive(d: pl.DataFrame) -> float:
    return d.filter(pl.col("email") == 1)["post_orders"].mean() - d.filter(pl.col("email") == 0)["post_orders"].mean()


rows = [{"segment": "all customers", "customers": df.height,
         "share_emailed": round(df["email"].mean(), 3), "naive_gap": round(naive(df), 3)}]
for flag, label in [(True, "loyalty members"), (False, "non-members")]:
    seg = df.filter(pl.col("is_loyalty") == flag)
    rows.append({"segment": label, "customers": seg.height,
                 "share_emailed": round(seg["email"].mean(), 3), "naive_gap": round(naive(seg), 3)})
table = pl.DataFrame(rows)
print(table)

# Standardise the segment gaps to the whole customer base
w = table.filter(pl.col("segment") != "all customers")
standardised = (w["naive_gap"] * w["customers"]).sum() / w["customers"].sum()
print(f"\nStandardised over segments:   {standardised:+.3f}")
print(f"True ATE:                     {df['tau'].mean():+.3f}")

python -m day19.northmart_segments

│ segment         ┆ customers ┆ share_emailed ┆ naive_gap │
│ all customers   ┆ 4000      ┆ 0.32          ┆ 1.711     │
│ loyalty members ┆ 1194      ┆ 0.526         ┆ 1.62      │
│ non-members     ┆ 2806      ┆ 0.232         ┆ 1.466     │

Standardised over segments:   +1.512
True ATE:                     +0.319

Answer in docs/ESTIMANDS.md, under "Segments and Simpson":

1. Why is the pooled gap (1.71) larger than the gap in either segment?
2. Is loyalty a confounder or a mediator of the email's effect? So should we stratify?
3. Stratifying moves the estimate from 1.71 to 1.51. Why is it still nearly five times the truth?

19.6 Save your work with Git (15 min)

git add .
git commit -m "Day 19: Simpson's paradox, standardisation, NorthMart segments"
git push

Day 19 checklist

• I can explain how Simpson's paradox arises from group sizes and baselines
• I can compute a standardised effect and explain how the weights define the estimand
• I can decide, with a DAG, whether to report the pooled or the stratified result
• I can explain why stratifying on one confounder may not be enough