Capstone: The Targeting Policy

70.1 The brief (30 min)

NorthMart's CMO wants to replace the current coupon targeting with something better, and asks three questions:

1. Should we keep sending coupons at all?
2. If so, to whom? Marketing's current rule is "lapsing customers, or anyone who buys more than 25% of items on promotion".
3. Is a two-year programme worth building, and how do we limit the downside?

Economics as before: $5 per coupon, 30% margin, break-even uplift $16.67. You have two datasets:

• last quarter: 30,000 customers, coupons sent by the current targeting model (observational, and the targeting model used a browsing signal that wasn't stored);
• a pilot: 6,000 different customers, coupons by coin flip.

python data/make_targeting_scenario.py

"""
Capstone data for Module 7: NorthMart's coupon targeting. Provided.

    python data/make_targeting_scenario.py

Writes data/scenario/coupons_last_quarter.csv (30,000 customers, coupons sent by the current targeting model),
data/scenario/coupons_pilot.csv (6,000 different customers, coupons by coin flip), and the true effects
to data/truth/coupon_effects.csv.
"""
import sys
from pathlib import Path

import polars as pl

sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from causalkit.coupons import coupons, public  # noqa: E402

root = Path(__file__).parent
(root / "scenario").mkdir(exist_ok=True)
(root / "truth").mkdir(exist_ok=True)
obs = coupons(n=30_000, seed=70, hidden=0.3)
pilot = coupons(n=6_000, seed=71, randomized=True).with_columns(pl.col("customer_id").str.replace("C", "P"))
public(obs).write_csv(root / "scenario" / "coupons_last_quarter.csv")
public(pilot).write_csv(root / "scenario" / "coupons_pilot.csv")
pl.concat([obs.select("customer_id", "_tau"), pilot.select("customer_id", "_tau")]).rename(
    {"_tau": "true_effect"}).write_csv(root / "truth" / "coupon_effects.csv")
print("Wrote data/scenario/coupons_last_quarter.csv (30,000) and data/scenario/coupons_pilot.csv (6,000)")

Don't open data/truth/ until your memo is committed.

70.2 Step 1: what does last quarter say, and how robust is it? (50 min)

Create day70/sensitivity.py:

# day70/sensitivity.py   (python -m day70.sensitivity)
"""Capstone step 1: what does last quarter's data say about coupons, and how robust is it?"""
import doubleml as dml
import numpy as np
import pandas as pd
import polars as pl

from causalkit.coupons import X_COLS
from causalkit.learners import clf, reg
from causalkit.policy import BREAK_EVEN

np.random.seed(70)  # DoubleML draws its cross-fitting folds from numpy's global RNG
obs = pd.read_csv("data/scenario/coupons_last_quarter.csv")
pilot = pd.read_csv("data/scenario/coupons_pilot.csv")
irm = dml.DoubleMLIRM(dml.DoubleMLData(obs[X_COLS + ["coupon", "spend"]], "spend", "coupon", X_COLS),
                      ml_g=reg(), ml_m=clf(), n_folds=5, trimming_threshold=0.02).fit()
print(f"Last quarter (observational): ATE {irm.coef[0]:.2f}, 95% CI [{irm.confint().iloc[0, 0]:.2f}, "
      f"{irm.confint().iloc[0, 1]:.2f}]")
irm.sensitivity_analysis(null_hypothesis=BREAK_EVEN)
print(f"   robustness value for 'above break-even (${BREAK_EVEN:.2f})': {irm.sensitivity_params['rv'][0]:.1%}")
rows = []
for f in ["deal_share", "app_user", "pre_spend"]:
    b = irm.sensitivity_benchmark(benchmarking_set=[f]).iloc[0]
    rows.append({"confounder as strong as": f, "cf_y": round(b.cf_y, 3), "cf_d": round(b.cf_d, 3),
                 "moves the estimate by": round(b.delta_theta, 2)})
print(pl.DataFrame(rows))
diff = pilot.spend[pilot.coupon == 1].mean() - pilot.spend[pilot.coupon == 0].mean()
se = np.sqrt(pilot.spend[pilot.coupon == 1].var() / (pilot.coupon == 1).sum()
             + pilot.spend[pilot.coupon == 0].var() / (pilot.coupon == 0).sum())
print(f"Pilot (randomised): difference in means {diff:.2f}, 95% CI [{diff - 1.96 * se:.2f}, {diff + 1.96 * se:.2f}]")

python -m day70.sensitivity

Last quarter (observational): ATE 18.98, 95% CI [18.01, 19.95]
   robustness value for 'above break-even ($16.67)': 2.7%
│ confounder as strong as ┆ cf_y  ┆ cf_d  ┆ moves the estimate by │
│ deal_share              ┆ 0.045 ┆ 0.127 ┆ 2.92                  │
│ app_user                ┆ 0.185 ┆ 0.034 ┆ 5.34                  │
│ pre_spend               ┆ 1.0   ┆ 0.094 ┆ -20.54                │
Pilot (randomised): difference in means 16.13, 95% CI [13.00, 19.26]

Last quarter's data say coupons pay ($18.98, CI entirely above $16.67), but the robustness value for that conclusion is 2.7%: a hidden confounder far weaker than deal-seeking or app use could overturn it. The pilot, which needs no such assumption, says $16.13, with a CI from $13 to $19 that straddles break-even.

Answer to question 1: blanket coupons are, at best, break-even. Don't decide on last quarter's number.

70.3 Step 2: learn policies, judge them on the pilot (60 min)

Learn targeting policies from last quarter's 30,000 customers (more data, even if confounded), then judge every policy on the pilot, where coupons were random and the models never looked. Create day70/policies.py (about 20 seconds):

# day70/policies.py   (python -m day70.policies)
"""
Capstone step 2: learn targeting policies from last quarter's data, then judge them on the pilot, which the
models never saw and where coupons were random. Candidates:
  nobody | everyone | marketing's current rule (lapsing customers, or deal_share above 25%) |
  DR-learner: effect > $16.67 | policy tree, depth 2
Also check the DR-learner's calibration on the pilot (Day 57).
"""
import numpy as np
import pandas as pd
import polars as pl
from econml.dr import DRLearner
from econml.policy import DRPolicyTree
from econml.validate import DRTester
from sklearn.model_selection import KFold

from causalkit.coupons import X_COLS
from causalkit.learners import clf, reg
from causalkit.policy import BREAK_EVEN, dr_scores, net, policy_value, tree_rules


def fit_policies(obs):
    dr = DRLearner(model_propensity=clf(), model_regression=reg(),
                   model_final=reg(n_estimators=100, num_leaves=4, min_child_samples=300), cv=5,
                   min_propensity=0.02, random_state=0).fit(obs.spend, obs.coupon, X=obs[X_COLS])
    tree = DRPolicyTree(model_regression=reg(), model_propensity=clf(), max_depth=2, min_samples_leaf=500,
                        random_state=0).fit(net(obs.spend, obs.coupon), obs.coupon, X=obs[X_COLS])
    return dr, tree


def decisions(dr, tree, df):
    return {
        "nobody": np.zeros(len(df)),
        "everyone": np.ones(len(df)),
        "marketing's rule": ((df.lapsing == 1) | (df.deal_share > 0.25)).astype(int).to_numpy(),
        "DR-learner > $16.67": (dr.effect(df[X_COLS]) > BREAK_EVEN).astype(int),
        "policy tree": tree.predict(df[X_COLS]),
    }


def pilot_scores(pilot):
    """Cross-fitted doubly robust scores of the net outcome on the pilot (coupon probability 0.5)."""
    y = net(pilot.spend, pilot.coupon)
    mu0, mu1 = np.zeros(len(pilot)), np.zeros(len(pilot))
    for tr, te in KFold(5, shuffle=True, random_state=0).split(pilot):
        a = pilot.iloc[tr]
        for t, mu in [(0, mu0), (1, mu1)]:
            mu[te] = reg().fit(a[a.coupon == t][X_COLS], y[tr][a.coupon.to_numpy() == t]).predict(pilot.iloc[te][X_COLS])
    return dr_scores(y, pilot.coupon, 0.5, mu0, mu1)


if __name__ == "__main__":
    obs = pd.read_csv("data/scenario/coupons_last_quarter.csv")
    pilot = pd.read_csv("data/scenario/coupons_pilot.csv")
    dr, tree = fit_policies(obs)
    print("Policy tree rules:")
    for r in tree_rules(tree.export_graphviz(feature_names=X_COLS)):
        print("   " + r)
    s1, s0 = pilot_scores(pilot)
    rng = np.random.default_rng(0)
    boot = [rng.integers(0, len(pilot), len(pilot)) for _ in range(1000)]
    pols = decisions(dr, tree, pilot)
    rows = []
    for name, pol in pols.items():
        b = [policy_value(pol[i], s1[i], s0[i]) for i in boot]
        rows.append({"policy": name, "share couponed": f"{pol.mean():.0%}",
                     "net $ per customer (pilot)": round(policy_value(pol, s1, s0), 2),
                     "95% CI": f"[{np.percentile(b, 2.5):.2f}, {np.percentile(b, 97.5):.2f}]"})
    with pl.Config(tbl_width_chars=120, fmt_str_lengths=30):
        print(pl.DataFrame(rows))
    gain = pols["DR-learner > $16.67"] - pols["marketing's rule"]
    b = [policy_value(gain[i], s1[i], s0[i]) for i in boot]
    print(f"DR-learner minus marketing's rule: {policy_value(gain, s1, s0):+.2f} per customer, "
          f"95% CI [{np.percentile(b, 2.5):+.2f}, {np.percentile(b, 97.5):+.2f}]")
    pd.DataFrame({k: v for k, v in pols.items()}).assign(customer_id=pilot.customer_id).to_csv(
        "data/day70_pilot_decisions.csv", index=False)

    tester = DRTester(model_regression=reg(), model_propensity=clf(), cate=dr)
    tester.fit_nuisance(pilot[X_COLS].to_numpy(), pilot.coupon.to_numpy(), pilot.spend.to_numpy(),
                        obs[X_COLS].to_numpy(), obs.coupon.to_numpy(), obs.spend.to_numpy())
    s = tester.evaluate_all(pilot[X_COLS].to_numpy(), obs[X_COLS].to_numpy(), n_bootstrap=200).summary().iloc[0]
    print(f"DR-learner on the pilot: BLP slope {s.blp_est:.2f} (SE {s.blp_se:.2f}), AUTOC {s.autoc_est:.1f}, "
          f"calibration R2 {s.cal_r_squared:.2f}")

python -m day70.policies

Policy tree rules:
   deal_share <= 0.264 and basket <= 21.835  ->  no coupon  (558 training customers)
   deal_share <= 0.264 and basket > 21.835  ->  no coupon  (7099 training customers)
   deal_share > 0.264 and pre_spend <= 597.325  ->  coupon  (6657 training customers)
   deal_share > 0.264 and pre_spend > 597.325  ->  coupon  (686 training customers)
│ policy              ┆ share couponed ┆ net $ per customer (pilot) ┆ 95% CI        │
│ nobody              ┆ 0%             ┆ 0.0                        ┆ [0.00, 0.00]  │
│ everyone            ┆ 100%           ┆ -0.33                      ┆ [-0.92, 0.24] │
│ marketing's rule    ┆ 58%            ┆ 0.86                       ┆ [0.42, 1.29]  │
│ DR-learner > $16.67 ┆ 56%            ┆ 1.28                       ┆ [0.85, 1.70]  │
│ policy tree         ┆ 50%            ┆ 1.05                       ┆ [0.65, 1.46]  │
DR-learner minus marketing's rule: +0.42 per customer, 95% CI [+0.15, +0.70]
DR-learner on the pilot: BLP slope 0.96 (SE 0.12), AUTOC 9.2, calibration R2 0.65

• Every targeted policy beats blanket coupons, which lose $0.33 per customer on the pilot.
• The DR-learner beats marketing's rule by $0.42 per customer (95% CI +$0.15 to +$0.70): the comparison that matters, and it's clearly positive.
• The DR-learner is well calibrated on the pilot (BLP slope 0.96), even though it was trained on confounded data. The hidden browsing signal biases the level of effects (Step 1) more than their ranking across customers. Targeting depends on the ranking.
• The policy tree's rule ("coupon if deal_share > 26%") is marketing's rule without the lapsing clause, and earns $0.19 more than marketing's rule. If the CRM can't use a model score, it's the simple upgrade.

Answer to question 2: target with the DR-learner, coupon about 56% of customers.

70.4 Step 3: from campaign to programme (40 min)

Feed the pilot's measurements into Days 66–67's programme model. Create day70/programme.py:

# day70/programme.py   (python -m day70.programme)
"""
Capstone step 3: from one campaign to a two-year programme (Days 66-68). Inputs measured on the pilot:
each policy's extra spend per customer in the base, and the share it coupons. Everything else as on Day 66.
Run day70.policies first.
"""
import numpy as np
import pandas as pd
import polars as pl

from day66.cba import BASE, npv
from day67.options import draw_inputs
from day70.policies import pilot_scores

pilot = pd.read_csv("data/scenario/coupons_pilot.csv")
dec = pd.read_csv("data/day70_pilot_decisions.csv")
s1, s0 = pilot_scores(pilot)
extra_spend = ((s1 - s0) + 5.0) / 0.30                      # net score difference back to $ of extra spend
rows = []
for name in ["everyone", "marketing's rule", "DR-learner > $16.67"]:
    pol = dec[name].to_numpy()
    base = BASE | {"uplift": float(np.mean(pol * extra_spend)), "share": float(pol.mean())}
    for stop in [False, True]:
        x = np.array([npv(a, stop=stop)[0] for a in draw_inputs(base | {"uplift_everyone": base["uplift"]}, n=5000)]) / 1000
        rows.append({"policy": name, "stop rule": "yes" if stop else "no",
                     "expected NPV ($k)": round(x.mean()), "P(loss)": f"{(x < 0).mean():.0%}",
                     "5th percentile ($k)": round(np.percentile(x, 5))})
with pl.Config(tbl_width_chars=120, fmt_str_lengths=30):
    print(pl.DataFrame(rows))

python -m day70.programme

│ policy              ┆ stop rule ┆ expected NPV ($k) ┆ P(loss) ┆ 5th percentile ($k) │
│ everyone            ┆ no        ┆ -16086            ┆ 98%     ┆ -29187              │
│ everyone            ┆ yes       ┆ -110              ┆ 94%     ┆ -289                │
│ marketing's rule    ┆ no        ┆ -3447             ┆ 74%     ┆ -11989              │
│ marketing's rule    ┆ yes       ┆ 1055              ┆ 50%     ┆ -274                │
│ DR-learner > $16.67 ┆ no        ┆ -556              ┆ 56%     ┆ -9321               │
│ DR-learner > $16.67 ┆ yes       ┆ 2322              ┆ 28%     ┆ -249                │

With the pilot's more modest numbers (and pull-forward and fatigue on top), the programme is a close call:

• Without a stop rule, even the best policy loses money in expectation (−$0.6M).
• With a stop rule, the DR-learner programme is worth $2.3M in expectation, with a 28% chance of a small loss (at most about the build cost).
• Marketing's rule with a stop rule is worth less than half as much ($1.1M) and loses in half the scenarios.

Answer to question 3: build it only with a stop rule, and launch with a holdout (Day 68) so the stop rule has data to act on.

70.5 Step 4: the decision memo (70 min)

Write docs/TARGETING_POLICY_DECISION.md using Day 69's structure: recommendation first; money and probabilities; one chart; what would change the decision; next steps with a holdout and stop rule; methods and robustness in an appendix.

70.6 Answer key, final commit and Module 7 retrospective (35 min)

After the memo is committed, compare with the truth. Create day70/answer_key.py:

# day70/answer_key.py   (python -m day70.answer_key)
"""Instructor check, after the memo: each policy's TRUE net value per customer on the pilot customers."""
import pandas as pd
import polars as pl

from causalkit.policy import COST, MARGIN

truth = pd.read_csv("data/truth/coupon_effects.csv").set_index("customer_id").true_effect
dec = pd.read_csv("data/day70_pilot_decisions.csv").set_index("customer_id")
gain = MARGIN * truth.loc[dec.index] - COST
rows = [{"policy": c, "share couponed": f"{dec[c].mean():.0%}", "true net $ per customer": round((dec[c] * gain).mean(), 2)}
        for c in dec.columns]
rows.append({"policy": "oracle", "share couponed": f"{(gain > 0).mean():.0%}",
             "true net $ per customer": round(gain.clip(lower=0).mean(), 2)})
print(pl.DataFrame(rows))
print(f"True average effect, pilot customers: ${truth.loc[dec.index].mean():.2f}; "
      f"last-quarter customers: ${truth.drop(dec.index).mean():.2f} (break-even ${COST / MARGIN:.2f})")

python -m day70.answer_key

│ policy              ┆ share couponed ┆ true net $ per customer │
│ nobody              ┆ 0%             ┆ 0.0                     │
│ everyone            ┆ 100%           ┆ -0.06                   │
│ marketing's rule    ┆ 58%            ┆ 1.05                    │
│ DR-learner > $16.67 ┆ 56%            ┆ 1.58                    │
│ policy tree         ┆ 50%            ┆ 1.36                    │
│ oracle              ┆ 47%            ┆ 1.66                    │
True average effect, pilot customers: $16.45; last-quarter customers: $16.16 (break-even $16.67)

The pilot ranked the policies exactly as the truth does. The DR-learner earns $1.58 per customer per campaign in truth, 95% of the oracle's $1.66, and $0.53 more than marketing's rule. Couponing everyone loses a little, as the pilot said. The true average effect ($16.45 and $16.16) is below break-even, so last quarter's $18.98 would have justified blanket coupons that lose money. Over 400,000 customers a month, the switch from marketing's rule is worth about $210,000 a month.

git add .
git commit -m "Day 70: targeting policy capstone: sensitivity, policies on the pilot, programme, memo"
git push

What you learned in Module 7.

Coming next: Module 8, Decision Intelligence Delivery. The analyses are done. Module 8 makes them last: packaging the estimators as a tested library, scheduled pipelines that re-estimate effects as data arrive, monitoring for drift, dashboards for decision-makers, and the final course capstone.

Day 70 checklist

• I checked whether the observational conclusion survives plausible hidden confounding
• I learned policies on one dataset and judged them on a randomised one
• I compared policies with paired confidence intervals, not just separate ones
• I valued the programme with a stop rule, and recommended a holdout
• I wrote a memo a CMO can act on, and checked it against the truth