Capstone: Regional Campaign ROI

50.1 The brief (30 min)

NorthMart's marketing team ran an 8-week TV campaign in region R12, starting in week 80. It cost $100k. NorthMart keeps 30% of extra sales as margin, so the campaign paid off if it raised sales by more than 100 / 0.30 = $333k in total over the 25 post-period weeks (the campaign and the 17 weeks after it, when some of the effect lingers).

The marketing team's own dashboard compares R12 with the average region and reports a strong success. The CFO wants to know: did the campaign pay for itself, and should NorthMart run it again in other regions at the same cost?

Generate the data:

python data/make_campaign_scenario.py

"""
Capstone data for Module 5: NorthMart's TV campaign in region R12. Provided.

    python data/make_campaign_scenario.py

Writes data/scenario/regional_sales.csv (what the analyst gets) and data/truth/regional_truth.json.
"""
import json
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from causalkit.regions import public, regions, truth  # noqa: E402

full = regions(seed=52, treated="R12", treated_size=180, treated_urban=1.8, lift=25)
root = Path(__file__).parent
(root / "scenario").mkdir(exist_ok=True)
(root / "truth").mkdir(exist_ok=True)
public(full).write_csv(root / "scenario" / "regional_sales.csv")
(root / "truth" / "regional_truth.json").write_text(json.dumps(truth(full), indent=2))
print(f"Wrote {full.height:,} region-weeks to data/scenario/regional_sales.csv")

Don't open data/truth/ until your memo is committed.

50.2 Step 1: look before estimating (40 min)

Create day50/diagnostics.py:

# day50/diagnostics.py   (python -m day50.diagnostics)
"""Capstone step 1: look before estimating. Is R12 like the other regions, before the campaign?"""
import matplotlib.pyplot as plt
import polars as pl

df = pl.read_csv("data/scenario/regional_sales.csv")
pre = df.filter(pl.col("week") < 80)
size = pre.group_by("region").agg(pl.col("sales").mean()).sort("sales", descending=True)
print(f"R12 average weekly sales before the campaign: {size.filter(pl.col('region') == 'R12')['sales'][0]:.0f} $k; "
      f"rank {size['region'].to_list().index('R12') + 1} of {size.height}")
first_last = (pre.filter(pl.col("week").is_in([1, 79])).pivot(on="week", index="region", values="sales")
              .with_columns(growth=pl.col("79") - pl.col("1")).sort("growth", descending=True))
print(f"R12 sales growth, week 1 to week 79: {first_last.filter(pl.col('region') == 'R12')['growth'][0]:.0f} $k; "
      f"rank {first_last['region'].to_list().index('R12') + 1} of {first_last.height}; "
      f"median region {first_last['growth'].median():.0f}")

others = df.filter(pl.col("region") != "R12").group_by("week").agg(pl.col("sales").mean()).sort("week")
r12 = df.filter(pl.col("region") == "R12").sort("week")
fig, ax = plt.subplots(figsize=(9, 4))
ax.plot(r12["week"], r12["sales"], color="#dc2626", label="R12")
ax.plot(others["week"], others["sales"], color="#64748b", label="average of other regions")
ax.axvline(80, color="grey", ls="--", lw=0.8)
ax.set_xlabel("week")
ax.set_ylabel("sales ($k per week)")
ax.legend()
fig.tight_layout()
fig.savefig("figures/day50_diagnostics.png", dpi=120)
print("Saved figures/day50_diagnostics.png")

python -m day50.diagnostics

R12 average weekly sales before the campaign: 713 $k; rank 1 of 30
R12 sales growth, week 1 to week 79: 140 $k; rank 1 of 30; median region 95
Saved figures/day50_diagnostics.png

Two warnings before any estimate: R12 is the biggest region, and the fastest-growing. From Days 45–46: DiD will be biased by the growth gap, and plain synthetic control can't match the level.

50.3 Step 2: four estimators and a placebo test (60 min)

Create day50/estimates.py:

# day50/estimates.py   (python -m day50.estimates)
"""Capstone step 2: four estimators, their fit, and a placebo-in-space p-value for the one that fits."""
import polars as pl

from causalkit.synth import did, placebo_in_space, sdid, synthetic_control

df = pl.read_csv("data/scenario/regional_sales.csv")
WEEKS_POST = df.filter(pl.col("post") == 1)["week"].n_unique()
sc = synthetic_control(df, "R12", 80)
sci = synthetic_control(df, "R12", 80, intercept=True)
rows = [
    {"estimator": "DiD vs average region", "per week": did(df, "R12", 80)["effect"], "pre-period RMSPE": None},
    {"estimator": "synthetic control", "per week": sc["effect"], "pre-period RMSPE": sc["rmspe_pre"]},
    {"estimator": "SC + intercept", "per week": sci["effect"], "pre-period RMSPE": sci["rmspe_pre"]},
    {"estimator": "synthetic DiD", "per week": sdid(df, "R12", 80)["effect"], "pre-period RMSPE": None},
]
table = pl.DataFrame(rows).with_columns(cumulative=pl.col("per week") * WEEKS_POST).with_columns(
    pl.col("per week", "cumulative", "pre-period RMSPE").round(1))
with pl.Config(tbl_width_chars=110, fmt_str_lengths=30):
    print(table)
print("SC + intercept weights:", sci["weights"])

ranks = placebo_in_space(df, "R12", 80, intercept=True)
rank = ranks["region"].to_list().index("R12") + 1
print(f"Placebo in space (SC + intercept): R12 ranks {rank} of {ranks.height}, p = {rank / ranks.height:.3f}")

python -m day50.estimates

│ estimator             ┆ per week ┆ pre-period RMSPE ┆ cumulative │
│ DiD vs average region ┆ 26.9     ┆ null             ┆ 673.3      │
│ synthetic control     ┆ 75.9     ┆ 31.5             ┆ 1898.0     │
│ SC + intercept        ┆ 17.4     ┆ 5.0              ┆ 434.8      │
│ synthetic DiD         ┆ 11.6     ┆ null             ┆ 290.8      │
SC + intercept weights: {'R29': 0.533, 'R07': 0.264, 'R01': 0.119, 'R18': 0.071, 'R11': 0.012}
Placebo in space (SC + intercept): R12 ranks 1 of 30, p = 0.033

• DiD ($27k a week) is what the marketing dashboard reports. The diagnostics already told us why it's too high.
• Plain synthetic control misses the pre-period by $31.5k a week: discard it.
• SC with an intercept fits well ($5k a week) and passes the placebo test (p = 0.033): $17.4k a week, $435k in total.
• Synthetic DiD: $11.6k a week, $291k in total.

The two credible estimators disagree, and they fall on opposite sides of the $333k break-even line. That's the real finding of this step.

50.4 Step 3: a posterior for profit (50 min)

Create day50/causal_impact.py. It uses the SC + intercept donors as control series (Day 47):

# day50/causal_impact.py   (python -m day50.causal_impact)
"""Capstone step 3: a Bayesian estimate with a posterior for profit. Controls = the SC + intercept donors."""
import numpy as np
import polars as pl

from causalkit.synth import synthetic_control
from day47.causal_impact import causal_impact, summary

MARGIN, COST = 0.30, 100.0
df = pl.read_csv("data/scenario/regional_sales.csv")
controls = list(synthetic_control(df, "R12", 80, intercept=True)["weights"])
result, scale = causal_impact(df, "R12", 80, controls)
s = summary(result, scale)
print(f"Controls: {controls}")
print(f"Average lift {s['average']['mean']:.1f} $k/week [94% HDI {s['average']['low']:.1f}, {s['average']['high']:.1f}]")
print(f"Cumulative lift {s['cumulative']['mean']:.0f} $k [94% HDI {s['cumulative']['low']:.0f}, {s['cumulative']['high']:.0f}]")
profit = MARGIN * (result.post_impact.sum("obs_ind") * scale).values.ravel() - COST
lo, hi = np.percentile(profit, [3, 97])
print(f"Profit {profit.mean():+.1f} $k [94% interval {lo:+.1f}, {hi:+.1f}];  P(profit > 0) = {(profit > 0).mean():.2f}")

python -m day50.causal_impact

Controls: ['R29', 'R07', 'R01', 'R18', 'R11']
Average lift 11.7 $k/week [94% HDI 8.4, 14.9]
Cumulative lift 293 $k [94% HDI 210, 373]
Profit -12.0 $k [94% interval -35.0, +11.5];  P(profit > 0) = 0.16

The Bayesian model agrees with SDID: about $290k of extra sales, a loss of about $12k, and a 16% probability that the campaign paid off.

Then turn every estimate into money. Create day50/decision.py:

# day50/decision.py   (python -m day50.decision)
"""
Capstone step 4: from lift to money. The campaign cost $100k; NorthMart keeps 30% of extra sales.
Profit = 0.30 x cumulative lift - 100. Break-even needs a cumulative lift of 100 / 0.30 = 333 $k.
"""
import polars as pl

from causalkit.synth import did, sdid, synthetic_control

MARGIN, COST = 0.30, 100.0
df = pl.read_csv("data/scenario/regional_sales.csv")
weeks = df.filter(pl.col("post") == 1)["week"].n_unique()
estimates = {
    "DiD vs average region": did(df, "R12", 80)["effect"],
    "synthetic control (poor pre-fit)": synthetic_control(df, "R12", 80)["effect"],
    "SC + intercept": synthetic_control(df, "R12", 80, intercept=True)["effect"],
    "synthetic DiD": sdid(df, "R12", 80)["effect"],
}
rows = [{"estimator": k, "cumulative lift ($k)": round(v * weeks), "profit ($k)": round(MARGIN * v * weeks - COST, 1),
         "verdict": "pays off" if MARGIN * v * weeks > COST else "loses money"} for k, v in estimates.items()]
with pl.Config(tbl_width_chars=110, fmt_str_lengths=40):
    print(pl.DataFrame(rows))
print(f"Break-even cumulative lift: {COST / MARGIN:.0f} $k")

python -m day50.decision

│ estimator                        ┆ cumulative lift ($k) ┆ profit ($k) ┆ verdict     │
│ DiD vs average region            ┆ 673                  ┆ 102.0       ┆ pays off    │
│ synthetic control (poor pre-fit) ┆ 1898                 ┆ 469.4       ┆ pays off    │
│ SC + intercept                   ┆ 435                  ┆ 30.4        ┆ pays off    │
│ synthetic DiD                    ┆ 291                  ┆ -12.8       ┆ loses money │
Break-even cumulative lift: 333 $k

Why might SC + intercept be too high? It relies on five donors, two-thirds of the weight on R29 and R07, and R12 is the fastest-growing region, at the edge of what any mix of donors can reproduce. SDID spreads weight over more donors and puts its time weights on recent weeks, which matters when growth is accelerating (Day 46's Try it). Both pass their checks; neither is certain. A good memo says so.

50.5 Step 4: the decision memo (60 min)

Write docs/CAMPAIGN_ROI_DECISION.md, one page, for the CFO:

1. Recommendation (two or three sentences).
2. What happened (the campaign, its cost, the break-even lift).
3. Why the dashboard's number is wrong (R12's size and growth; one picture).
4. Results (the four estimators and the Bayesian model, with their checks, and profit for each).
5. How sure are we? (where the credible estimates disagree, and what that means for the decision).
6. Next steps (how to get a clean answer next time).

50.6 Answer key, final commit and Module 5 retrospective (30 min)

After the memo is committed, compare with the truth. Create day50/answer_key.py:

# day50/answer_key.py   (python -m day50.answer_key)
"""Instructor check, after the memo: the true lift and the true profit."""
import json
from pathlib import Path

MARGIN, COST = 0.30, 100.0
t = json.loads(Path("data/truth/regional_truth.json").read_text())
print(f"True lift: {t['avg_post_lift']:.1f} $k per post week; cumulative {t['cumulative_lift']:.0f} $k "
      f"({t['campaign_weeks_lift']:.0f} during the 8 campaign weeks)")
print(f"True profit: {MARGIN * t['cumulative_lift'] - COST:+.1f} $k")

python -m day50.answer_key

True lift: 11.9 $k per post week; cumulative 298 $k (200 during the 8 campaign weeks)
True profit: -10.7 $k

The campaign lost about $11k. Synthetic DiD ($291k) and the Bayesian model ($293k) were within $7k of the true cumulative lift; SC + intercept overstated it by 46%, and the dashboard's DiD by 126%. A memo that leaned towards the lower estimates, while admitting the disagreement, got the decision right.

git add .
git commit -m "Day 50: regional campaign ROI capstone: diagnostics, estimates, posterior, memo"
git push

What you learned in Module 5.

Coming next: Module 6, Causal Machine Learning. Modules 4 and 5 estimated one average effect. Module 6 asks for whom the effect is large: double machine learning, causal forests and uplift models, to target campaigns at the customers they actually move.

Day 50 checklist

• I ran diagnostics that predicted which estimators would fail, before estimating
• I discarded an estimator because of its pre-period fit, not its answer
• I ran a placebo test and reported a permutation p-value
• I turned every estimate into profit against a break-even line
• I wrote a memo that states where credible methods disagree, and what would settle it