Capstone: The Loyalty Programme Effect

40.1 The brief (30 min)

NorthMart's CFO asks two questions about the loyalty programme:

1. Does the programme pay for itself, per member? Each member costs $45 a year in rewards. NorthMart keeps 30% of any extra spending as margin. So a member is worth 0.30 × effect − $45.
2. Should NorthMart keep sending the $20 sign-up invitations? Every invited customer who joins receives the $20 bonus, including those who would have joined anyway.

The brand team insists that "our fans join and our fans spend". You have one year of data on 20,000 customers, including last year's random invitation. Generate it:

python data/make_loyalty_scenario.py

"""
Capstone data for Module 4: NorthMart's loyalty programme. Provided.

    python data/make_loyalty_scenario.py

Writes data/scenario/loyalty.csv (what the analyst gets) and data/truth/loyalty_truth.json.
"""
import json
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from causalkit.loyalty import public, truth, world  # noqa: E402

full = world(seed=40, hidden=1.0)
root = Path(__file__).parent
(root / "scenario").mkdir(exist_ok=True)
(root / "truth").mkdir(exist_ok=True)
public(full).write_csv(root / "scenario" / "loyalty.csv")
(root / "truth" / "loyalty_truth.json").write_text(json.dumps(truth(full), indent=2))
print(f"Wrote {full.height:,} customers to data/scenario/loyalty.csv")

You don't know whether this is world A or world B. That's the point: the analysis has to be right in either. Don't open data/truth/ until your memo is committed.

40.2 Step 1: the DAG and the strategy (45 min)

Create day40/dag.py:

# day40/dag.py   (python -m day40.dag)
"""Capstone step 1: the DAG, and what it says about each strategy."""
import networkx as nx

from causalkit.dags import draw_dag, make_dag, minimal_adjustment_sets

X = ["pre_spend", "engaged", "tenure"]
edges = [(x, "joined") for x in X] + [(x, "spend") for x in X] + [
    ("tenure", "engaged"), ("tenure", "pre_spend"),
    ("affinity", "joined"), ("affinity", "spend"),      # the brand team's worry: loyal fans join AND spend
    ("invited", "joined"),                              # the random $20 invitation
    ("joined", "spend"),
]
dag = make_dag(edges, latent={"affinity"})

if __name__ == "__main__":
    print("Valid adjustment sets for joined -> spend:", minimal_adjustment_sets(dag, "joined", "spend") or "none")

    # Instrument checks, read off the graph:
    no_effect = dag.copy()
    no_effect.remove_edge("joined", "spend")
    print("invited independent of affinity:        ", nx.is_d_separator(dag, {"invited"}, {"affinity"}, set()))
    print("invited reaches spend only via joining: ", nx.is_d_separator(no_effect, {"invited"}, {"spend"}, set()))
    print("invited affects joining:                ", dag.has_edge("invited", "joined"))
    draw_dag(dag, "figures/day40_loyalty_dag.png", "Loyalty programme: assumed DAG", treatment="joined", outcome="spend")
    print("Saved figures/day40_loyalty_dag.png")

python -m day40.dag

Valid adjustment sets for joined -> spend: none
invited independent of affinity:         True
invited reaches spend only via joining:  True
invited affects joining:                 True
Saved figures/day40_loyalty_dag.png

If the brand team is right, no adjustment set exists: every Days 31–36 method is ruled out by the assumptions, before any data are touched. The invitation passes all three instrument checks on the graph. Remember, the graph shows what you assume: the exclusion check is only as good as the missing arrow from invited to spend (Day 38).

40.3 Step 2: estimates and diagnostics (55 min)

Run both strategies anyway: if they agree, the brand team's worry may be overblown; if not, the gap is informative. Create day40/estimates.py:

# day40/estimates.py   (python -m day40.estimates)
"""Capstone step 2: both strategies on the capstone data, with diagnostics."""
import pandas as pd
import polars as pl

from causalkit.estimators import aipw, two_sls

df = pd.read_csv("data/scenario/loyalty.csv")

a = aipw(df)
iv = two_sls(df)
first = df.groupby("invited").joined.mean()
fs = first[1] - first[0]

rows = [
    {"strategy": "AIPW (selection on observables)", "estimand": "ATE", "estimate": a["ATE"], "se": a["ATE_se"]},
    {"strategy": "AIPW (selection on observables)", "estimand": "ATT", "estimate": a["ATT"], "se": a["ATT_se"]},
    {"strategy": "2SLS, invitation as instrument", "estimand": "LATE", "estimate": iv.params["joined"],
     "se": iv.std_errors["joined"]},
]
table = pl.DataFrame(rows).with_columns(
    ci_low=(pl.col("estimate") - 1.96 * pl.col("se")).round(1),
    ci_high=(pl.col("estimate") + 1.96 * pl.col("se")).round(1),
    estimate=pl.col("estimate").round(1), se=pl.col("se").round(1))
with pl.Config(fmt_str_lengths=40, tbl_width_chars=110):
    print(table)
print(f"\nFirst stage: {first[1]:.1%} joined if invited vs {first[0]:.1%} if not ({fs:+.1%});"
      f" F = {iv.first_stage.diagnostics['f.stat'].iloc[0]:.0f}")
print(f"Joiners' average pre_spend ${df.pre_spend[df.joined == 1].mean():,.0f}; "
      f"compliers' (estimated) ${((df.pre_spend * df.joined)[df.invited == 1].mean() - (df.pre_spend * df.joined)[df.invited == 0].mean()) / fs:,.0f}")

python -m day40.estimates

│ strategy                        ┆ estimand ┆ estimate ┆ se   ┆ ci_low ┆ ci_high │
│ AIPW (selection on observables) ┆ ATE      ┆ 229.9    ┆ 2.4  ┆ 225.2  ┆ 234.5   │
│ AIPW (selection on observables) ┆ ATT      ┆ 256.4    ┆ 3.2  ┆ 250.0  ┆ 262.7   │
│ 2SLS, invitation as instrument  ┆ LATE     ┆ 146.3    ┆ 15.0 ┆ 116.9  ┆ 175.6   │

First stage: 55.7% joined if invited vs 41.1% if not (+14.6%); F = 513
Joiners' average pre_spend $881; compliers' (estimated) $696

• AIPW and 2SLS disagree by $84, and the 2SLS interval excludes AIPW's estimate. That's the world B pattern of Day 39.
• The instrument is strong (F = 513), so weak-instrument bias isn't the explanation.
• Compliers spent much less last year than joiners, so the LATE and the ATT describe different people. But in world A, where both strategies were valid, that difference was worth a few dollars (Day 37's Try it), not $84. Different populations can't explain a gap this size; a hidden confounder can.

40.4 Step 3: from effects to money (55 min)

Create day40/decision.py:

# day40/decision.py   (python -m day40.decision)
"""
Capstone step 3: turn effects into money.
  margin on extra spend: 30%      reward cost: $45 per member per year      sign-up bonus: $20 per invited joiner
Q1: does the programme pay for itself, per member?            value = 0.30 x effect - 45
Q2: should NorthMart keep sending $20 invitations?            per 1,000 invitations:
    compliers x (0.30 x LATE - 45)  -  $20 x (all invited customers who join, compliers or not)
"""
import pandas as pd

from causalkit.estimators import aipw, two_sls

MARGIN, REWARDS, BONUS = 0.30, 45.0, 20.0
df = pd.read_csv("data/scenario/loyalty.csv")
a, iv = aipw(df), two_sls(df)
late, late_se = iv.params["joined"], iv.std_errors["joined"]
first = df.groupby("invited").joined.mean()
compliers_per_1000 = 1000 * (first[1] - first[0])
joiners_per_1000_invited = 1000 * first[1]


def value(effect: float) -> float:
    return MARGIN * effect - REWARDS


print("Q1: value per member per year (0.30 x effect - $45)")
print(f"   using AIPW ATT ${a['ATT']:.0f}:  ${value(a['ATT']):+.1f}   <- biased upwards if fans self-select")
print(f"   using 2SLS LATE ${late:.0f}:  ${value(late):+.1f}  "
      f"[95% CI ${value(late - 1.96 * late_se):+.1f} to ${value(late + 1.96 * late_se):+.1f}]")


def invitations(effect: float) -> float:
    return compliers_per_1000 * value(effect) - BONUS * joiners_per_1000_invited


print("\nQ2: net value of 1,000 invitations")
print(f"   {compliers_per_1000:.0f} extra joiners, but {joiners_per_1000_invited:.0f} invited joiners receive the $20 bonus")
print(f"   using 2SLS LATE: ${invitations(late):+,.0f}  "
      f"[95% CI ${invitations(late - 1.96 * late_se):+,.0f} to ${invitations(late + 1.96 * late_se):+,.0f}]")
print(f"   using AIPW ATE (wrong population, biased): ${invitations(a['ATE']):+,.0f}")

python -m day40.decision

Q1: value per member per year (0.30 x effect - $45)
   using AIPW ATT $256:  $+31.9   <- biased upwards if fans self-select
   using 2SLS LATE $146:  $-1.1  [95% CI $-9.9 to $+7.7]

Q2: net value of 1,000 invitations
   146 extra joiners, but 557 invited joiners receive the $20 bonus
   using 2SLS LATE: $-11,303  [95% CI $-12,586 to $-10,021]
   using AIPW ATE (wrong population, biased): $-7,649

Q1. With AIPW, the programme looks clearly profitable: +$32 per member. With the LATE, it's break-even: −$1, with a CI from −$10 to +$8. The answer to "does it pay for itself?" depends entirely on which assumption you trust.

Q2. This one is clear whichever effect you use. Each 1,000 invitations create 146 extra members, but pay the $20 bonus to all 557 invited joiners, 411 of whom would have joined anyway. Even the optimistic estimate loses money. Note that the LATE is the right effect for Q2: the invitation's extra joiners are, by definition, its compliers.

40.5 The decision memo (55 min)

Write docs/LOYALTY_DECISION.md, one page, for the CFO:

1. Recommendation (two sentences, one per question).
2. The question and the data (one paragraph).
3. Why we didn't trust the simple comparison (the $593 gap; the fan worry; what the DAG says).
4. Results (a small table: strategy, estimand, estimate, CI, value per member).
5. Whose effect is this? (the complier profile, in plain language).
6. Risks and next steps (exclusion: could the invitation email itself raise spending? A follow-up that would settle Q1).

40.6 Answer key, final commit and Module 4 retrospective (30 min)

After the memo is committed, compare with the truth. Create day40/answer_key.py:

# day40/answer_key.py   (python -m day40.answer_key)
"""Instructor check, after the memo: the true effects and the decisions they imply."""
import json
from pathlib import Path

import pandas as pd

MARGIN, REWARDS, BONUS = 0.30, 45.0, 20.0
t = json.loads(Path("data/truth/loyalty_truth.json").read_text())
df = pd.read_csv("data/scenario/loyalty.csv")
joiners_per_1000_invited = 1000 * df[df.invited == 1].joined.mean()

print("True effects of joining on next-year spend:")
for k in ["ATE", "ATT", "ATC", "LATE"]:
    print(f"   {k:<5} ${t[k]:6.1f}   value per member (0.30 x effect - $45): ${MARGIN * t[k] - REWARDS:+.1f}")
compliers = 1000 * t["share_compliers"]
net = compliers * (MARGIN * t["LATE"] - REWARDS) - BONUS * joiners_per_1000_invited
print(f"True net value of 1,000 invitations: ${net:+,.0f} ({compliers:.0f} compliers)")

python -m day40.answer_key

True effects of joining on next-year spend:
   ATE   $ 158.1   value per member (0.30 x effect - $45): $+2.4
   ATT   $ 182.5   value per member (0.30 x effect - $45): $+9.8
   ATC   $ 135.2   value per member (0.30 x effect - $45): $-4.5
   LATE  $ 156.3   value per member (0.30 x effect - $45): $+1.9
True net value of 1,000 invitations: $-10,842 (157 compliers)

This was world B. The 2SLS interval contains the true LATE ($156), and the invitation decision is right: the true loss is $10,842 per 1,000 invitations, inside our interval. AIPW's ATT ($256) was $74 too high, and would have made the programme look worth $32 per member when it's really worth about $10 for current members. A director acting on AIPW would have expanded a programme that barely breaks even.

git add .
git commit -m "Day 40: loyalty programme capstone: DAG, estimates, decision memo"
git push

What you learned in Module 4.

Coming next: Module 5, Quasi-Experiments. IV needs a lucky random nudge. Module 5 uses the structure of time and rules instead: difference-in-differences, event studies, synthetic control and regression discontinuity, to evaluate rollouts and campaigns that were never randomised.

Day 40 checklist

• I chose a strategy from a DAG before looking at the estimates
• I ran both strategies and used their disagreement as evidence
• I reported the first-stage F and the complier profile with the IV estimate
• I matched each business question to the right estimand (LATE for the invitation decision)
• I wrote a decision memo with money, uncertainty and the assumptions it rests on