Sensitivity, Robustness & Decisions
Stress-testing estimates and turning them into targeting policies and choices under uncertainty.
What you'll learn
- 1Days 61–63Sensitivity to hidden confounding, robustness values, E-values, refutation tests and specification curves.
By the end: Students can quantify how strong a hidden confounder would have to be to change a conclusion (DoubleML sensitivity bounds, robustness values, E-values), make that strength meaningful by benchmarking against observed covariates, and show how much an answer depends on analysis choices with refutation tests and a specification curve.
- 2Days 64–66Uplift modelling, policy learning and cost-benefit analysis.
By the end: Students can build and evaluate uplift models on a randomised campaign with Qini and uplift curves written from scratch, learn a cost-aware targeting policy and value it with doubly robust scores, and scale a campaign result into a programme-level cost-benefit analysis with NPV, a tornado chart and Monte Carlo uncertainty.
- 3Days 67–70Decisions under uncertainty, value of information and the decision memo.
By the end: Students can compare options under uncertainty with expected value, downside risk, regret and risk attitude; value flexibility (stop rules) and information (EVPI, EVSI, holdouts); write a decision memo whose numbers come from code; and deliver a complete targeting decision with robustness checks.
Tools
Mini-capstone
Targeting policy: a learned coupon policy versus treat-all and the current rule, judged on a randomised pilot, with sensitivity checks, a programme NPV and a stop rule.
The ten days
Before you start: module setup and project structure
Course: Applied Causal Inference & Decision Science Audience: Economists and analysts who completed Modules 1–6 Environment: GitHub Codespaces (VS Code in the browser) Time per day: ~5 hours (including breaks and exercises) Goal by end of Day 63: Students can quantify how strong a hidden confounder would have to be to change a conclusion (DoubleML sensitivity bounds, robustness values, E-values), make that strength meaningful by benchmarking against observed covariates, and show how much an answer depends on analysis choices with refutation tests and a specification curve.
Where we are
Every module so far ended with an assumption the data couldn't check: no hidden confounders (Modules 4 and 6), parallel trends (Module 5), exclusion (Module 4). Module 7 stops treating those assumptions as yes/no and asks the two questions a decision-maker actually has:
- How wrong would the assumption have to be to change the decision? (Days 61–63)
- Given everything we know and don't know, what should we do? (Days 64–70)
| Days | Question | Tools |
|---|---|---|
| 61–63 | How fragile is the estimate? | DoubleML sensitivity, robustness values, E-values, specification curves |
| 64–66 | Whom should we treat, and is the programme worth it? | uplift curves, EconML policy trees, cost-benefit analysis |
| 67–70 | How do we decide under uncertainty, and when should we learn more first? | decision criteria, value of information, the decision memo |
The running example is Module 6's coupon programme, now with real money attached: a coupon costs $5, NorthMart keeps 30% of extra spend, so a coupon pays for itself if it raises spending by more than $5 / 0.30 = $16.67. The true average effect is $16.51, just below that line. Every method in this module is tested on how well it handles a close call.
Small changes to earlier files
Two simulators gain a hidden column so we can measure the real confounder later (an analyst wouldn't have it): _browsing in causalkit/coupons.py and _affinity in causalkit/loyalty.py. Add "_browsing": browsing to the dictionary at the end of coupons(), and "_affinity": affinity next to "_ps": ps in world(). Nothing else changes; every earlier result is identical.
mkdir -p day61 day62 day63 day64 day65 day66 day67 day68 day69 day70
No new packages this module. (The usual uplift package, scikit-uplift, no longer runs with current numpy and scikit-learn; on Day 64 we write the 40 lines we need instead.)