Estimands: ATE, ATT, CATE, LATE

12.1 An estimand is a precise question (40 min)

Concept. Three words are easy to mix up:

Choosing the estimand comes first, because different decisions need different estimands, and their true values differ. A complete estimand states five things:

1. Unit: customer, household, store, region?
2. Treatment and comparison: emailed vs not emailed? vs a different email?
3. Outcome and horizon: orders, revenue, margin? Over 8 weeks? 12 months?
4. Population: all customers, the emailed ones, loyalty members?
5. Summary: an average difference, a ratio, a percentile?

The main population choices have names:

12.2 Same email, five true answers (50 min)

Create day12/estimands.py:

# day12/estimands.py   (python -m day12.estimands)
"""Same email, different questions, different true answers."""
import polars as pl

po = pl.read_parquet("data/truth/potential_outcomes.parquet")


def effect(df: pl.DataFrame) -> tuple[float, float]:
    """Average effect in orders, and relative to what the group would have done anyway."""
    return df["tau"].mean(), df["tau"].sum() / df["y0"].sum()


rows = []
for name, df in [
    ("ATE  (all customers)", po),
    ("ATT  (emailed customers)", po.filter(pl.col("treated") == 1)),
    ("ATC  (not emailed)", po.filter(pl.col("treated") == 0)),
    ("CATE (loyalty members)", po.filter(pl.col("is_loyalty"))),
    ("CATE (non-members)", po.filter(~pl.col("is_loyalty"))),
]:
    abs_eff, rel_eff = effect(df)
    rows.append({"estimand": name, "customers": df.height,
                 "orders_per_customer": round(abs_eff, 3), "relative": f"{rel_eff:+.1%}"})

with pl.Config(tbl_width_chars=100, fmt_str_lengths=40):
    print(pl.DataFrame(rows))

python -m day12.estimands

│ estimand                 ┆ customers ┆ orders_per_customer ┆ relative │
│ ATE  (all customers)     ┆ 4000      ┆ 0.319               ┆ +15.7%   │
│ ATT  (emailed customers) ┆ 1279      ┆ 0.443               ┆ +15.3%   │
│ ATC  (not emailed)       ┆ 2721      ┆ 0.26                ┆ +16.1%   │
│ CATE (loyalty members)   ┆ 1194      ┆ 0.429               ┆ +15.9%   │
│ CATE (non-members)       ┆ 2806      ┆ 0.272               ┆ +15.6%   │

The relative effect is about +15% for every group: that is how the simulation was built. The absolute effect differs by a factor of almost two between the emailed (0.443) and the rest (0.260), because emailed customers are heavier shoppers: 15% of more orders is more orders.

So "the effect of the email" has no single true value. A manager who wants to know what it would add to send the email to the 2,721 other customers needs the ATC, 0.26 orders each, not the ATT. Reporting the ATT for that decision would overstate the gain by 70%, with no estimation error at all.

Try it. NorthMart earns about $160 per order. Using the true values, how much extra revenue did the email bring in from the 1,279 customers who got it? How much would emailing the other 2,721 add?

12.3 Conditional effects (CATE) and where the effect comes from (45 min)

Concept. A CATE is the average effect for units with given characteristics X: CATE(x) = E[Y₁ − Y₀ | X = x]. CATEs drive targeting: email the segments with the largest effects. Module 6 estimates them with machine learning; today we look at the true ones.

Create day12/cate_by_region.py:

# day12/cate_by_region.py   (python -m day12.cate_by_region)
"""Conditional effects (CATE) by region, and a chart of where the effect comes from."""
import matplotlib

matplotlib.use("Agg")                      # write files; no screen needed
import matplotlib.pyplot as plt
import polars as pl

po = pl.read_parquet("data/truth/potential_outcomes.parquet")

by_region = (po.group_by("region")
             .agg(customers=pl.len(),
                  cate=pl.col("tau_expected").mean(),
                  baseline=pl.col("mu0").mean())
             .with_columns(relative=pl.col("cate") / pl.col("baseline"))
             .sort("cate", descending=True))
print(by_region)

fig, ax = plt.subplots(figsize=(7, 4.5))
ax.scatter(po["mu0"], po["tau_expected"], s=6, alpha=0.4)
ax.set_xlabel("Expected orders without the email, weeks 9-16 (mu0)")
ax.set_ylabel("Expected extra orders from the email (tau)")
ax.set_title("Same +15% for everyone, so bigger shoppers gain more orders")
fig.tight_layout()
fig.savefig("figures/day12_effect_vs_baseline.png", dpi=150)
print("Saved figures/day12_effect_vs_baseline.png")

│ region ┆ customers ┆ cate     ┆ baseline ┆ relative │
│ ATL    ┆ 485       ┆ 0.313129 ┆ 2.087529 ┆ 0.15     │
│ QC     ┆ 931       ┆ 0.311348 ┆ 2.075651 ┆ 0.15     │
│ AB     ┆ 453       ┆ 0.308103 ┆ 2.054023 ┆ 0.15     │
│ ON     ┆ 1513      ┆ 0.303939 ┆ 2.026258 ┆ 0.15     │
│ BC     ┆ 618       ┆ 0.303255 ┆ 2.021703 ┆ 0.15     │
Saved figures/day12_effect_vs_baseline.png

Open figures/day12_effect_vs_baseline.png in VS Code: a straight line. Each customer's expected extra orders is exactly 15% of their expected orders without the email. Region doesn't matter at all; what matters is how much a customer shops. These true CATEs use tau_expected (the effect without counting noise), which is why they look so smooth.

The truth file uses each customer's true region, so there's no UNKNOWN group here.

Targeting lesson. If marketing could only email 1,279 customers, the ones with the largest absolute CATE are the heaviest shoppers, which is roughly whom they chose. Targeting on expected effect is good business. It's also exactly what makes the naive comparison biased. Good targeting and easy evaluation pull in opposite directions; Module 3 shows how to get both.

12.4 When people don't comply: ITT and LATE (50 min)

Concept. Often we control the offer but not the take-up: we send an email (Z), but customers decide whether to open it (D). Two estimands follow:

• ITT (intention to treat): the effect of sending, whatever people do with it. Easy to estimate when sending is random.
• LATE (local average treatment effect): the effect of opening, for compliers: customers who open the email when they receive it.

Comparing openers with non-openers is badly biased: people who open marketing emails are different. But if sending was random, the LATE can be recovered with the Wald estimator:

LATE = ITT / first stage
     = (effect of sending on orders) / (effect of sending on opening)

Create day12/late_toy.py, a toy world built for this purpose (not NorthMart):

# day12/late_toy.py   (python -m day12.late_toy)
"""
A toy world for the LATE: the effect for customers who OPEN the email because it was sent.

Z = email sent (randomised)      D = email opened      Y = orders
Only 'compliers' open an email when they get one. 'Never-takers' never open it.
Heavy shoppers are more likely to be compliers: openers are not like non-openers.
"""
import numpy as np
import polars as pl

rng = np.random.default_rng(11)
n = 100_000

intensity = rng.lognormal(0, 0.5, n)                       # unobserved shopping intensity
complier = rng.random(n) < np.clip(0.2 + 0.25 * intensity, 0, 0.9)
z = rng.random(n) < 0.5                                     # random send
d = z & complier                                            # opened
effect = 0.5                                                # true effect of OPENING, for compliers
y = rng.poisson(2 * intensity + effect * d)

df = pl.DataFrame({"z": z, "d": d, "y": y, "complier": complier})
mean = lambda e: df.filter(e)["y"].mean()

itt = mean(pl.col("z")) - mean(~pl.col("z"))                 # effect of SENDING
first_stage = df.filter(pl.col("z"))["d"].mean() - df.filter(~pl.col("z"))["d"].mean()
late = itt / first_stage                                     # Wald estimator
naive_open = mean(pl.col("d")) - mean(~pl.col("d"))          # openers vs everyone else

print(f"Share of compliers              {complier.mean():.1%}")
print(f"ITT (effect of sending)         {itt:+.3f}")
print(f"First stage (opens if sent)     {first_stage:.3f}")
print(f"LATE = ITT / first stage        {late:+.3f}   (truth for compliers: +{effect})")
print(f"Naive: openers vs non-openers   {naive_open:+.3f}   (badly biased)")

Share of compliers              47.7%
ITT (effect of sending)         +0.234
First stage (opens if sent)     0.474
LATE = ITT / first stage        +0.493   (truth for compliers: +0.5)
Naive: openers vs non-openers   +0.947   (badly biased)

The ITT is diluted: half the people who were sent the email never opened it. Dividing by the share of compliers scales it back up to the effect per opener. This is instrumental variables in its simplest form. Module 4 develops it fully; for now, notice that the LATE is an effect for compliers only, which you can't list by name.

Try it. Make the effect larger for heavy shoppers: replace effect * d with 0.5 * intensity * d, and add two lines that print the true average effect for compliers, (0.5 * intensity[complier]).mean(), and for everyone, (0.5 * intensity).mean(). Does the LATE still describe all customers?

LATE = ITT / first stage        +0.662   (ignore the old "truth" label)
True effect for compliers       +0.652
True effect if everyone opened  +0.566

12.5 Day 12 mini-project: estimand cards for three decisions (70 min)

Goal. Before estimating anything, write down exactly what you want. Create docs/ESTIMANDS.md with one card per decision, using this template:

### Decision: <the business decision>
- **Unit:**
- **Treatment vs comparison:**
- **Outcome and horizon:**
- **Population:**
- **Summary measure:**
- **Name:** ATE / ATT / ATC / CATE / LATE
- **True value (from the truth file, for checking only):**
- **Data we would need to estimate it:**

Write cards for:

1. The CFO asks: "Did SPRING26 pay for itself?"
2. The CMO asks: "Next spring, should we email everyone?"
3. The CRM team asks: "Should loyalty members get the email first?"

### Decision: Did SPRING26 pay for itself?
- **Unit:** customer
- **Treatment vs comparison:** received SPRING26 vs the same customer not receiving it
- **Outcome and horizon:** completed orders (then margin) in weeks 9–16 of 2026
- **Population:** the 1,279 customers who were emailed
- **Summary measure:** average difference, then total (× 1,279)
- **Name:** ATT
- **True value:** +0.443 orders per customer (+15.3%)
- **Data we would need:** pre- and post-period orders for emailed and non-emailed customers,
  plus an assumption (e.g. parallel trends) to stand in for the missing Y₀ of the emailed

### Decision: Next spring, should we email everyone?
- **Population:** all 4,000 customers (or next year's base)        **Name:** ATE
- **True value:** +0.319 orders per customer (+15.7%)
- **Caveat:** next spring is a different period; the effect may not transfer

### Decision: Should loyalty members get the email first?
- **Population:** two subgroups: loyalty members and non-members   **Name:** CATE by loyalty
- **True values:** +0.429 (members) vs +0.272 (non-members) orders per customer
- **Note:** in relative terms both are ~+15.8%; members gain more orders only because they shop more

12.6 Save your work with Git (15 min)

git add .
git commit -m "Day 12: estimands, CATE, LATE, estimand cards"
git push

Day 12 checklist

• I can distinguish an estimand, an estimator and an estimate
• I can state an estimand completely: unit, treatment, outcome, horizon, population, summary
• I can choose between ATE, ATT, ATC and CATE for a given decision
• I can explain why the LATE is an effect for compliers only, and compute it with the Wald ratio
• I have written estimand cards for three NorthMart decisions