Writing the Decision Memo

69.1 What a decision memo is for (40 min)

You've written a decision memo at the end of every module. Today makes the format explicit, because it's where most analyses succeed or fail: a decision-maker reads one page, often only the first paragraph.

Four rules that cover most of it:

1. Lead with the decision, not the analysis.
2. Money and probabilities, not coefficients and p-values. "86% chance it pays" beats "p < 0.001".
3. State what would change your mind, with a number. "If the coupon costs more than $6, coupon the top 35% instead."
4. Every number traceable to code. If the memo says $4.98M, a script printed $4.98M.

69.2 Numbers from code, not from memory (40 min)

Collect every number the memo uses in one script that writes them to a file, so the memo and the chart can't drift apart. Create day69/memo_numbers.py (about 20 seconds):

# day69/memo_numbers.py   (python -m day69.memo_numbers)
"""
A memo's numbers should come from code, not from copying out of six notebooks. Collect every number the
coupon-programme memo needs in one place, with its source, and save them for the memo and the chart.
Run day66.cba first.
"""
import json

import numpy as np
from econml.dr import DRLearner

from causalkit.coupons import X_COLS
from causalkit.learners import clf, reg
from causalkit.policy import BREAK_EVEN, dr_scores, policy_value
from day64.uplift_models import experiment
from day65.policy_value import outcome_models
from day67.options import option_npvs

train, test = experiment()
dr = DRLearner(model_propensity=clf(), model_regression=reg(),
               model_final=reg(n_estimators=100, num_leaves=4, min_child_samples=300), cv=5,
               min_propensity=0.02, random_state=0).fit(train.spend, train.coupon, X=train[X_COLS])
pol = (dr.effect(test[X_COLS]) > BREAK_EVEN).astype(int)
y, mu0, mu1 = outcome_models(test)
s1, s0 = dr_scores(y, test.coupon, 0.5, mu0, mu1)
rng = np.random.default_rng(0)
boot = [policy_value(pol[i], s1[i], s0[i]) for i in (rng.integers(0, len(test), len(test)) for _ in range(500))]
v = option_npvs(json.load(open("data/day66_inputs.json")))

numbers = {
    "experiment customers": 40_000,
    "share couponed by the targeted policy": round(float(pol.mean()), 3),
    "targeted: net $ per customer per campaign": round(policy_value(pol, s1, s0), 2),
    "targeted: 95% CI": [round(float(np.percentile(boot, 2.5)), 2), round(float(np.percentile(boot, 97.5)), 2)],
    "everyone: net $ per customer per campaign": round(policy_value(np.ones(len(test)), s1, s0), 2),
    "programme NPV, targeted + stop rule ($k, expected)": round(float(v["targeted + stop rule"].mean())),
    "programme P(loss), targeted + stop rule": round(float((v["targeted + stop rule"] < 0).mean()), 3),
    "programme NPV, targeted without stop rule ($k, expected)": round(float(v["targeted"].mean())),
    "programme P(loss), targeted without stop rule": round(float((v["targeted"] < 0).mean()), 3),
    "programme NPV, coupons for everyone ($k, expected)": round(float(v["coupons for everyone"].mean())),
}
json.dump(numbers, open("data/day69_memo_numbers.json", "w"), indent=2)
for k, val in numbers.items():
    print(f"{k:<58} {val}")

python -m day69.memo_numbers

experiment customers                                       40000
share couponed by the targeted policy                      0.484
targeted: net $ per customer per campaign                  1.84
targeted: 95% CI                                           [1.63, 2.05]
everyone: net $ per customer per campaign                  0.18
programme NPV, targeted + stop rule ($k, expected)         4976
programme P(loss), targeted + stop rule                    0.063
programme NPV, targeted without stop rule ($k, expected)   3809
programme P(loss), targeted without stop rule              0.26
programme NPV, coupons for everyone ($k, expected)         -13007

In a real team, data/day69_memo_numbers.json is what gets reviewed alongside the memo. A reviewer who wants to check "$5.0M" opens one file, not six notebooks.

69.3 One chart (40 min)

A memo gets one chart. For a decision under uncertainty, show each option's range of outcomes and its chance of a loss, not a bar of the expected value alone. Create day69/one_chart.py:

# day69/one_chart.py   (python -m day69.one_chart)
"""
One chart for the memo: for each option, the distribution of two-year NPV (median, 5th-95th percentile),
with the probability of a loss written next to it. Decision-makers read ranges and odds, not p-values.
Run day66.cba first.
"""
import json

import matplotlib.pyplot as plt
import numpy as np

from day67.options import option_npvs

v = option_npvs(json.load(open("data/day66_inputs.json")))
fig, ax = plt.subplots(figsize=(8, 3.6))
names = list(v)[::-1]
for i, name in enumerate(names):
    x = v[name] / 1000
    lo, med, hi = np.percentile(x, [5, 50, 95])
    ax.plot([lo, hi], [i, i], color="#94a3b8", lw=6, solid_capstyle="round")
    ax.plot(med, i, "o", color="#1e293b")
    ax.text(max(hi, 0) + 1.5, i, f"P(loss) {np.mean(x < 0):.0%}", va="center", fontsize=9)
ax.axvline(0, color="#dc2626", lw=1)
ax.set_yticks(range(len(names)), names)
ax.set_xlabel("two-year NPV, $ million (dot: median; bar: 90% range)")
ax.set_xlim(-45, 30)
fig.tight_layout()
fig.savefig("figures/day69_memo_chart.png", dpi=150)
print("Saved figures/day69_memo_chart.png")

python -m day69.one_chart

Saved figures/day69_memo_chart.png

Open it: four horizontal bars (90% range of NPV), a dot for the median, a red line at zero, and P(loss) written beside each. A reader sees in five seconds that blanket coupons are a disaster, the plain targeted programme is a gamble, and the stop rule is the safe winner.

69.4 Anatomy of a bad memo (55 min)

Here's a memo, written in good faith, about the same programme. Read it as the CFO would, then list everything wrong with it before reading the critique.

Subject: Coupon programme analysis results

We analysed last quarter's coupon campaign using double/debiased machine learning (DML) with LightGBM nuisance learners and 5-fold cross-fitting. The interactive regression model gives an ATE of 24.61 (SE 0.64, p < 0.001), which is highly significant. We also fitted a causal forest, which found significant heterogeneity, with deal_share being the most important feature (importance 0.887). A sensitivity analysis gives a robustness value of 24.7%, so the result is robust to unobserved confounding.

Given the strong and robust effect, we recommend rolling out coupons to all customers. The expected incremental revenue is $24.61 × 400,000 × 12 = $118M per year.

Further research could explore heterogeneity in more detail.

69.5 Day 69 mini-project: rewrite it (75 min)

Write docs/COUPON_PROGRAMME_MEMO.md, one page, using only numbers from data/day69_memo_numbers.json and Days 61–68's outputs, and the chart from 69.3.

69.6 Save your work with Git (20 min)

git add .
git commit -m "Day 69: memo numbers from code, decision chart, memo rewrite"
git push

Day 69 checklist

• I lead with the decision, in money and probabilities
• Every number in my memo comes from a script that prints it
• I use one chart that shows ranges and the chance of a loss
• I state what would change the recommendation, with numbers
• My memo ends with a concrete plan: monitoring, stop rule, next test