Potential Outcomes and the Fundamental Problem

11.1 Two worlds for every customer (30 min)

Concept. For each customer i and the email, imagine two numbers:

• Y₁(i): orders in weeks 9–16 if customer i receives the email.
• Y₀(i): orders in weeks 9–16 if customer i does not.

These are potential outcomes. Both exist before marketing decides anything; the decision only determines which one we get to see. The causal effect of the email on customer i is the difference:

τ(i) = Y₁(i) − Y₀(i)

This is the Rubin causal model, the framework behind every modern policy evaluation, from minimum-wage studies to drug trials. Economists know it as the logic of counterfactuals: "compared with what?"

Every causal question names the comparison. "Did the email work?" really means: did emailed customers buy more than they would have without the email? Not more than other customers; more than themselves, in the world that didn't happen.

11.2 Individual effects and the average effect (50 min)

Example 1 — the god's-eye view. Create day11/gods_view.py:

# day11/gods_view.py   (python -m day11.gods_view)
"""What only a simulation can show: both potential outcomes for the same customers."""
import polars as pl

po = pl.read_parquet("data/truth/potential_outcomes.parquet")

# 1. Six customers, both worlds side by side
six = po.filter(pl.col("customer_id").is_in(["C00004", "C00009", "C00010", "C00001", "C00002", "C00003"]))
print(six.select("customer_id", "treated", "y0", "y1", "tau").sort("customer_id"))

# 2. Individual effects vary: most customers would not change at all
print(po.group_by("tau").len().sort("tau"))

# 3. The average of the individual effects is the average treatment effect (ATE)
print(f"ATE = average of y1 - y0 = {po['tau'].mean():.3f} orders per customer over 8 weeks")

python -m day11.gods_view

│ customer_id ┆ treated ┆ y0  ┆ y1  ┆ tau │
│ C00001      ┆ 0       ┆ 3   ┆ 4   ┆ 1   │
│ C00002      ┆ 0       ┆ 4   ┆ 4   ┆ 0   │
│ C00003      ┆ 0       ┆ 3   ┆ 3   ┆ 0   │
│ C00004      ┆ 1       ┆ 2   ┆ 2   ┆ 0   │
│ C00009      ┆ 1       ┆ 2   ┆ 2   ┆ 0   │
│ C00010      ┆ 1       ┆ 3   ┆ 4   ┆ 1   │
...
│ 0   ┆ 2952 │
│ 1   ┆ 862  │
│ 2   ┆ 150  │
│ 3   ┆ 33   │
│ 4   ┆ 2    │
│ 5   ┆ 1    │
ATE = average of y1 - y0 = 0.319 orders per customer over 8 weeks

Read it carefully:

• Customer C00004 (Day 1: 7 orders before the email, 2 after) would have ordered 2 times either way. The email did nothing for them. Their drop from 7 to 2 was not caused by the email, which Day 1 could only guess.
• Customer C00001 was not emailed, but would have placed one extra order if they had been.
• Effects vary. For 2,952 of 4,000 customers the email changes nothing; for 862 it adds one order; a few gain more.
• The average treatment effect (ATE) is the mean of the individual effects: 0.319 extra orders per customer over eight weeks.

In the simulation, the email adds about 15% to each customer's expected orders. Because orders are counts, that shows up as "+1 order" for some customers and "0" for others. Real effects are lumpy like this too.

11.3 The fundamental problem of causal inference (45 min)

Concept. For every customer we observe one potential outcome, never both. This is called the fundamental problem of causal inference (Holland, 1986). It is not a data-quality problem that more data can fix: it's built into the question.

Create day11/fundamental_problem.py:

# day11/fundamental_problem.py   (python -m day11.fundamental_problem)
"""What the data really shows: one potential outcome per customer, never both."""
import polars as pl

po = pl.read_parquet("data/truth/potential_outcomes.parquet")

observed = po.select(
    "customer_id",
    "treated",
    y0=pl.when(pl.col("treated") == 0).then(pl.col("y0")),   # NULL when treated
    y1=pl.when(pl.col("treated") == 1).then(pl.col("y1")),   # NULL when not treated
)
print(observed.sort("customer_id").head(6))

missing = observed.select(pl.col("y0").is_null().sum() + pl.col("y1").is_null().sum()).item()
print(f"Missing potential outcomes: {missing:,} of {2 * po.height:,} ({missing / (2 * po.height):.0%})")
print("Individual effects we can compute from data alone:", 0)

│ customer_id ┆ treated ┆ y0   ┆ y1   │
│ C00001      ┆ 0       ┆ 3    ┆ null │
│ C00002      ┆ 0       ┆ 4    ┆ null │
│ C00003      ┆ 0       ┆ 3    ┆ null │
│ C00004      ┆ 1       ┆ null ┆ 2    │
...
Missing potential outcomes: 4,000 of 8,000 (50%)
Individual effects we can compute from data alone: 0

Causal inference is therefore a missing-data problem: exactly half the table is missing, and the missing half is the counterfactual. Every method in this course fills it in using other units or other periods, under assumptions. The assumptions are the whole game.

Try it. Take a pen and paper. For C00001, the data says 3 orders without the email. Write down three different ways you might guess their Y₁, and the assumption each guess relies on.

11.4 Assumptions that make potential outcomes meaningful (50 min)

Writing Y₁(i) and Y₀(i) already assumes two things. They're usually stated together as SUTVA (the stable unit treatment value assumption):

And one more, which links potential outcomes to data:

• Consistency: a customer who received the email shows Y₁ in the data (Y = Y₁ when T = 1, Y = Y₀ when T = 0). It fails when the recorded treatment isn't the one received: for example a send log that includes bounced emails.

Module 1's contracts protect consistency. assert_single_send_date and assert_treatment_is_fixed exist so that "treated" means one thing, at one time, for every customer.

Try it. For each scenario, say which assumption is threatened and how you would change the analysis.

1. NorthMart sends a referral email: "Give a friend $10".
2. 8% of SPRING26 emails bounced, but the send log lists all 1,279 customers.
3. The email had two subject lines, chosen at random per customer.

11.5 Day 11 mini-project: naive comparison = effect + selection bias (75 min)

Goal. Split NorthMart's naive comparison into its two parts, exactly. Add and subtract the never-observed quantity E[Y₀ | emailed]:

  E[Y₁ | T=1] − E[Y₀ | T=0]       naive difference (what the data shows)
= E[Y₁ − Y₀ | T=1]                ATT: the effect on the treated
+ E[Y₀ | T=1] − E[Y₀ | T=0]       selection bias

The first term is what we want. The second asks: without the email, would the emailed group have ordered more anyway? With targeted marketing, yes.

Create day11/decomposition.py:

# day11/decomposition.py   (python -m day11.decomposition)
"""Mini-project: split the naive comparison into the true effect and selection bias."""
import polars as pl

po = pl.read_parquet("data/truth/potential_outcomes.parquet")
t = po.filter(pl.col("treated") == 1)
c = po.filter(pl.col("treated") == 0)

naive = t["y1"].mean() - c["y0"].mean()       # what the data shows
att = t["y1"].mean() - t["y0"].mean()         # true effect on the emailed (needs the truth)
bias = t["y0"].mean() - c["y0"].mean()        # how different the groups were WITHOUT the email

print(f"E[Y1 | emailed]            {t['y1'].mean():6.3f}   (observed)")
print(f"E[Y0 | not emailed]        {c['y0'].mean():6.3f}   (observed)")
print(f"E[Y0 | emailed]            {t['y0'].mean():6.3f}   (NEVER observed: from the truth file)")
print()
print(f"Naive difference           {naive:+6.3f}")
print(f"  = true effect (ATT)      {att:+6.3f}")
print(f"  + selection bias         {bias:+6.3f}")
assert abs(naive - (att + bias)) < 1e-9

print(f"\nBias is {bias / naive:.0%} of the naive difference.")
print(f"In relative terms the true effect is {att / t['y0'].mean():+.1%}; the naive gap is {naive / c['y0'].mean():+.0%}.")

python -m day11.decomposition

E[Y1 | emailed]             3.332   (observed)
E[Y0 | not emailed]         1.621   (observed)
E[Y0 | emailed]             2.889   (NEVER observed: from the truth file)

Naive difference           +1.711
  = true effect (ATT)      +0.443
  + selection bias         +1.268

Bias is 74% of the naive difference.
In relative terms the true effect is +15.3%; the naive gap is +106%.

Three quarters of the naive gap is selection bias: the emailed customers would have ordered 2.889 times anyway, against 1.621 for the others. The assert line checks that the decomposition is an identity, not an approximation. It always holds; the only question is how big the bias term is.

Try it. Does comparing within a segment remove the bias? Repeat the decomposition separately for loyalty members and non-members.

import polars as pl

po = pl.read_parquet("data/truth/potential_outcomes.parquet")
for members in [True, False]:
    g = po.filter(pl.col("is_loyalty") == members)
    t, c = g.filter(pl.col("treated") == 1), g.filter(pl.col("treated") == 0)
    naive = t["y1"].mean() - c["y0"].mean()
    att = t["y1"].mean() - t["y0"].mean()
    bias = t["y0"].mean() - c["y0"].mean()
    label = "loyalty members" if members else "non-members"
    print(f"{label:<16} naive {naive:+.3f} = ATT {att:+.3f} + bias {bias:+.3f}")

loyalty members  naive +1.620 = ATT +0.482 + bias +1.137
non-members      naive +1.466 = ATT +0.404 + bias +1.062

11.6 Save your work with Git (20 min)

git add .
git status         # data/truth/ must NOT appear
git commit -m "Day 11: potential outcomes, truth file, bias decomposition"
git push

Day 11 checklist

• I can define potential outcomes and an individual causal effect for a business question
• I can explain the fundamental problem of causal inference as a missing-data problem
• I can name the parts of SUTVA and consistency, with a threat to each in my own data
• I can decompose a naive comparison into the ATT plus selection bias
• I use the truth file only to check methods, never as an input to them