Day 80: Final Capstone Defence · Applied Causal Inference & Decision Science

Written lesson

Time	Block
0:00 – 0:30	80.1 Format and rubric
0:30 – 2:30	80.2 Defences (10 minutes + 10 minutes of questions each; breaks between)
2:30 – 3:30	80.3 The answer key
3:30 – 3:45	Break
3:45 – 4:30	80.4 Module 8 and course retrospective
4:30 – 5:00	80.5 Final commit and what's next

80.1 Format and rubric (30 min)

Each student (or pair) presents the Day 79 deck to a "board" of instructor and peers, then takes questions. The board may ask for any number to be reproduced live from the repository.

Criterion	Weight	Excellent looks like
Decision first	15%	first slide: what to do, how much it earns, how sure
Diagnosis	20%	found the suburban drift; judged the urban alert as immaterial, with a reason
Re-estimation	20%	rebuilt on post-change weeks; reasoned about precision lost; committed and versioned
Honest uncertainty	20%	ranges in words; the correlated-error range; what the range leaves out
Engineering	15%	numbers from committed code; tests pass; API and app use the same parameters file
Answers under questioning	10%	concise, no new claims without evidence, "I don't know, here's how I'd find out"

80.2 Defences (120 min)

Questions the board will ask (prepare for all of them):

Your range for "5% everywhere" is $359k to $507k. What would have to be true for the real result to be outside it?
Urban premium alerted first. Why didn't you act on it? What would make you change your mind?
You used 26 weeks for suburban family and 104 for everyone else. What if the other segments changed too, just less?
The coupon curve comes from one experiment. How do you know it still holds next year?
A board member says: "just set the price that maximises profit". What do you say?
If we deploy your recommendation, how will we know in six months whether it worked?
What's the single assumption that, if wrong, would change your recommendation?

Good answers to 6 name a design, not a hope: a holdout of stores at the old prices, or a staggered rollout analysed as a difference-in-differences (Module 5).

80.3 The answer key (60 min)

After the defences, compare every projection with the simulator's truth. The key applies the true post-drift elasticities to this year's prices and volumes, and puts them next to three projections: the old file, the recent-only rebuild and the hybrid. Create day80/answer_key.py:

# day80/answer_key.py   (python -m day80.answer_key)   — instructor only
"""
After the defence: each price scenario's TRUE annual gross-profit change (simulator elasticities after the
drift, applied to this year's prices and volumes), next to what the old and the rebuilt parameters projected.
Coupons are left out: their truth did not drift, and Module 7's key already covers them.
"""
import json

import polars as pl

from engine import Scenario, load_parameters, simulate
from engine.presets import PRESETS

truth = json.load(open("data/truth/final_truth.json"))
old, new = load_parameters(), load_parameters("models/final_parameters.json")
recent = load_parameters("models/final_recent_only.json")
true_params = json.loads(json.dumps(new))
for c in true_params["pricing"]:
    c["elasticity"], c["se"] = truth["elasticity_now"][f"{c['segment']}/{c['category']}"], 0.0

scenarios = {n: s for n, s in PRESETS.items() if s.price_change and not s.coupon_share}
scenarios["5% outside suburban family"] = Scenario(price_change={"all": 0.05, "suburban family": 0.0})
scenarios["5% urban premium only"] = Scenario(price_change={"urban premium": 0.05})
def band(p, sc):
    g = simulate(p, sc)["change"]["gross_profit"]
    return g["p5"] / 1e3, g["p95"] / 1e3


rows = []
for name, sc in scenarios.items():
    sc = sc.model_copy(update={"coupon_share": 0.0})
    med = lambda p: simulate(p, sc)["change"]["gross_profit"]["median"] / 1e3   # noqa: E731
    rows.append({"scenario": name, "old projection $k": round(med(old)), "recent-only $k": round(med(recent)),
                 "final (hybrid) $k": round(med(new)), "final 90% range": "{:.0f} to {:.0f}".format(*band(new, sc)),
                 "TRUE $k": round(med(true_params))})
print(f"Drift: {truth['drift']}")
with pl.Config(tbl_width_chars=160, fmt_str_lengths=30):
    print(pl.DataFrame(rows))
print("""Grade the defence on: noticing the suburban-family drift (and that the urban-premium alert is
statistically real but economically small), rebuilding on post-drift weeks only, keeping projections
inside valid_change, and stating the uncertainty in words the board can use. Note that TRUE can fall outside
the 90% range: the bands cover sampling noise, not the small attenuation bias every estimate shares.""")

python -m day80.answer_key

Drift: {'segment': 'suburban family', 'week': 105, 'factor': 1.25}
│ scenario                   ┆ old projection $k ┆ recent-only $k ┆ final (hybrid) $k ┆ final 90% range ┆ TRUE $k │
│ 5% everywhere              ┆ 459               ┆ 508            ┆ 431               ┆ 359 to 507      ┆ 341     │
│ 5% in the 'raise' cells    ┆ 517               ┆ 603            ┆ 576               ┆ 518 to 632      ┆ 546     │
│ 5% outside suburban family ┆ 324               ┆ 486            ┆ 411               ┆ 363 to 461      ┆ 356     │
│ 5% urban premium only      ┆ 356               ┆ 489            ┆ 467               ┆ 440 to 493      ┆ 459     │

What the key says:

The hybrid beats the recent-only rebuild everywhere. Re-estimating every cell on 26 weeks overstated every price scenario, by $30k to $167k. Short samples didn't just add noise: they made every elasticity a little less negative, and in a near-break-even calculation that is worth a lot.
The recommended scenario holds. The 'raise' cells earn $546k in truth; the hybrid said $576k with a range of $518k–$632k. The decision was right, and the number was close.
"5% everywhere" is the warning. True $341k lies below the hybrid's range. Every estimate in the file shares a small attenuation bias (Day 73's correlated errors, made real). The range covers sampling noise, not shared bias, and a scenario that sums 20 cells near break-even magnifies it. Students who quoted the ρ = 0.5 range were closer to honest.
Coupons are not in the key: their truth didn't drift, and Module 7's key covers them. The coupon programme is most of the plan's value, so the holdout matters more than any pricing refinement.

80.4 Module 8 and course retrospective (45 min)

What you learned in Module 8.

Day	Idea	The simulator lesson
71	Serving causal models	Estimation once (minutes), simulation per request (10 ms); a versioned parameters file in between
72	Scenario design	Revenue fell and profit rose; the optimiser left the data in 13 of 20 cells
73	Communicating uncertainty	Rounded medians, ranges in words, calibrated probabilities; correlated errors made the range 4× wider
74	The front end	Inputs are decisions; the first screen is the counterfactual; scenarios are links
75	Counterfactual charts	Zero-based axes, grey counterfactual, ranges always; decompositions that add up
76	Testing and deployment	Tests of economic properties; CI; a container per parameters version; a smoke test
77	Monitoring	Rolling 2SLS found the student drift in 16 weeks, CUSUM in 11; a tight SE turns method gaps into alarms
78	Re-run decisions	The drift flipped no decision; a 10% test beats a 5% one; the test wasn't worth its $51k
79–80	Capstone	The hybrid rebuild got the decision right; shared bias is what ranges miss

The whole course, in one table.

Module	Question it answers	The habit it leaves
1 Causal-ready data	Can I trust the data's history?	contracts, lineage, reproducible pipelines
2 Causal reasoning	What exactly am I estimating, and what must I assume?	estimands first, DAGs before regressions
3 Experiments	How do I learn by intervening?	power before launch, pre-registration, variance reduction
4 Observational methods	What if I can't randomise?	overlap, doubly robust estimators, instruments with honest first stages
5 Quasi-experiments	What if the world randomised for me?	parallel trends, synthetic controls, discontinuities, placebos
6 Causal ML	What if the confounding is complicated, and effects differ?	cross-fitting, heterogeneity that's tested, not mined
7 Decisions	What should we do, given what we don't know?	sensitivity, policy value, value of information
8 Delivery	How do people use it, and when is it out of date?	versioned parameters, honest interfaces, monitoring and re-run rules

The through-line: every number is a comparison with a counterfactual, every counterfactual rests on an assumption, and every assumption has a price. Your job is to make that chain visible to the people who decide.

80.5 Final commit and what's next (30 min)

python -m pytest -q
git add .
git commit -m "Day 80: final capstone defence and answer key. Course complete."
git tag v1.0
git push --tags

Your repository now contains a reusable causal toolkit (causalkit), a parameters builder, a tested scenario API and app, monitoring scripts, and 80 days of worked analyses. It's a portfolio: pin it on your GitHub profile, and write a short README that links the capstones of Modules 3, 5, 7 and 8.

Where to go next:

Read: Cunningham, Causal Inference: The Mixtape; Huntington-Klein, The Effect; Chernozhukov et al., Applied Causal Inference Powered by ML and AI; Hernán & Robins, Causal Inference: What If.
Practise: reproduce a published study's main table from its replication files, then run its sensitivity analysis.
Build: add one estimator you met only in passing (synthetic DiD, Bayesian structural time series, off-policy evaluation for bandits) to causalkit, with tests.

Day 80 checklist

I defended a recommendation under questioning, reproducing numbers live
I compared my projections with the truth and can explain every gap
I know which uncertainty my ranges covered and which they didn't
My repository is tagged v1.0 and its tests pass
Course complete

Final Capstone Defence

What you'll learn today

Today's plan