Cost-Benefit Analysis

66.1 From a campaign to a programme (40 min)

Day 65 valued one campaign. The decision on the table is a programme: build a targeting system and run monthly campaigns for two years. That brings in things an experiment doesn't measure:

The experiment pins down two inputs. The others are assumptions, some of them big. A cost-benefit analysis (CBA) makes them explicit and shows which ones the decision depends on.

66.2 A two-year NPV (60 min)

Create day66/cba.py. It measures the two experimental inputs and saves all inputs for the rest of the module:

# day66/cba.py   (python -m day66.cba)
"""
Cost-benefit analysis: is the targeted coupon PROGRAMME worth building? Day 65 valued one campaign.
A programme runs monthly for two years, costs money to build and run, and its effect may fade.
  per customer per month:  margin x (extra spend among couponed customers, per customer in the base)
                           x (1 - pull-forward) x fatigue^month  -  coupon cost x share couponed
  pull-forward: part of the extra spend is purchases customers would have made next month anyway
  fatigue:      the coupon's effect shrinks a little every month customers keep receiving them
The two numbers that come from the experiment (Day 65's DR-learner policy, measured on held-out customers):
  uplift: extra spend per customer in the base;  share: share of customers couponed.
"""
import json

import numpy as np
from econml.dr import DRLearner

from causalkit.coupons import X_COLS
from causalkit.learners import clf, reg
from causalkit.policy import BREAK_EVEN, dr_scores
from day64.uplift_models import experiment
from day65.policy_value import outcome_models


def npv(a: dict, months: int = 24, stop: bool = False) -> tuple[float, int | None]:
    """Net present value ($) of the programme, and the first month cumulative cash flow turns positive.
    stop=True: shut the programme down in the first month its monthly cash flow would be negative."""
    rate = (1 + a["discount_rate"]) ** (1 / 12) - 1
    cum, total, payback = -a["build_cost"], -a["build_cost"], None
    for m in range(1, months + 1):
        per_customer = (a["margin"] * a["uplift"] * (1 - a["pull_forward"]) * a["fatigue"] ** (m - 1)
                        - a["coupon_cost"] * a["share"])
        flow = a["customers"] * per_customer - a["running_cost"]
        if stop and flow < 0:
            break
        total += flow / (1 + rate) ** m
        cum += flow
        if payback is None and cum > 0:
            payback = m
    return total, payback


BASE = {"customers": 400_000, "margin": 0.30, "coupon_cost": 5.0, "pull_forward": 0.15, "fatigue": 0.99,
        "build_cost": 150_000, "running_cost": 25_000, "discount_rate": 0.10}

if __name__ == "__main__":
    train, test = experiment()
    dr = DRLearner(model_propensity=clf(), model_regression=reg(),
                   model_final=reg(n_estimators=100, num_leaves=4, min_child_samples=300), cv=5,
                   min_propensity=0.02, random_state=0).fit(train.spend, train.coupon, X=train[X_COLS])
    pol = (dr.effect(test[X_COLS]) > BREAK_EVEN).astype(int)
    y, mu0, mu1 = outcome_models(test, cost=0.0)            # cost 0 and margin 1/0.3: models of spend x 0.3
    s1, s0 = dr_scores(y / 0.30, test.coupon, 0.5, mu0 / 0.30, mu1 / 0.30)
    inputs = BASE | {"uplift": float(np.mean(pol * (s1 - s0))), "share": float(pol.mean()),
                     "uplift_everyone": float(np.mean(s1 - s0))}
    json.dump(inputs, open("data/day66_inputs.json", "w"), indent=2)
    print("Inputs:", {k: round(v, 3) for k, v in inputs.items()})
    total, payback = npv(inputs)
    first = (inputs["margin"] * inputs["uplift"] * (1 - inputs["pull_forward"]) - inputs["coupon_cost"] * inputs["share"])
    print(f"Month 1: net ${first:.2f} per customer x {inputs['customers']:,} customers "
          f"= ${first * inputs['customers']:,.0f}, minus ${inputs['running_cost']:,} running cost")
    print(f"Two-year NPV: ${total:,.0f};  payback in month {payback}")
    print(f"Same programme with coupons for EVERYONE: ${npv(inputs | {'uplift': inputs['uplift_everyone'], 'share': 1.0})[0]:,.0f}")

python -m day66.cba

Inputs: {'customers': 400000, 'margin': 0.3, 'coupon_cost': 5.0, 'pull_forward': 0.15, 'fatigue': 0.99, 'build_cost': 150000, 'running_cost': 25000, 'discount_rate': 0.1, 'uplift': 14.192, 'share': 0.484, 'uplift_everyone': 17.283}
Month 1: net $1.20 per customer x 400,000 customers = $479,545, minus $25,000 running cost
Two-year NPV: $6,479,697;  payback in month 1
Same programme with coupons for EVERYONE: $-9,829,495

At the base assumptions, the targeted programme is worth about $6.5M over two years and pays back its build cost in the first month. The same programme without targeting loses $9.8M, even though its average uplift is higher ($17.28 vs $14.19 per customer): it coupons everyone, and pull-forward pushes the effective uplift below break-even.

The month-1 net value ($1.20 per customer) is lower than Day 65's $1.84 because of pull-forward: 15% of the extra spend would have happened next month anyway.

66.3 Which assumption matters most? The tornado chart (50 min)

Move each input to a plausible low and high value, one at a time; sort by the swing in NPV. Create day66/tornado.py:

# day66/tornado.py   (python -m day66.tornado)
"""
Which assumption matters most? Move each input to a plausible low and high value, one at a time, and
record the NPV. Sorted by the size of the swing, that's a tornado chart. Run day66.cba first.
"""
import json

import matplotlib.pyplot as plt
import polars as pl

from day66.cba import npv

base = json.load(open("data/day66_inputs.json"))
ranges = {
    "fatigue (monthly retention of the effect)": (0.96, 1.00),
    "pull-forward share": (0.30, 0.0),
    "uplift (95% CI of the experiment)": (base["uplift"] * 0.87, base["uplift"] * 1.13),
    "margin": (0.25, 0.35),
    "coupon cost ($)": (6.0, 4.0),
    "customers reached": (300_000, 500_000),
    "running cost ($/month)": (50_000, 15_000),
    "build cost ($)": (300_000, 100_000),
}
keys = {"fatigue (monthly retention of the effect)": "fatigue", "pull-forward share": "pull_forward",
        "uplift (95% CI of the experiment)": "uplift", "margin": "margin", "coupon cost ($)": "coupon_cost",
        "customers reached": "customers", "running cost ($/month)": "running_cost", "build cost ($)": "build_cost"}
base_npv = npv(base)[0]
rows = []
for label, (lo, hi) in ranges.items():
    k = keys[label]
    rows.append({"assumption": label, "low input": lo, "high input": hi,
                 "NPV low ($k)": round(npv(base | {k: lo})[0] / 1000), "NPV high ($k)": round(npv(base | {k: hi})[0] / 1000)})
table = pl.DataFrame(rows).with_columns(swing=(pl.col("NPV high ($k)") - pl.col("NPV low ($k)")).abs()).sort("swing")
with pl.Config(tbl_width_chars=130, fmt_str_lengths=45, float_precision=2):
    print(table.sort("swing", descending=True))
print(f"Base NPV: ${base_npv / 1000:,.0f}k")

fig, ax = plt.subplots(figsize=(9, 4.5))
for i, r in enumerate(table.iter_rows(named=True)):
    ax.barh(i, r["NPV low ($k)"] - base_npv / 1000, left=base_npv / 1000, color="#dc2626", alpha=0.8)
    ax.barh(i, r["NPV high ($k)"] - base_npv / 1000, left=base_npv / 1000, color="#16a34a", alpha=0.8)
ax.set_yticks(range(table.height), table["assumption"], fontsize=8)
ax.axvline(base_npv / 1000, color="k", lw=0.8)
ax.axvline(0, color="grey", ls="--", lw=0.8)
ax.set_xlabel("two-year NPV ($k)  (red: low input, green: high input)")
fig.tight_layout()
fig.savefig("figures/day66_tornado.png", dpi=120)
print("Saved figures/day66_tornado.png")

python -m day66.tornado

│ assumption                                ┆ low input ┆ high input ┆ NPV low ($k) ┆ NPV high ($k) ┆ swing │
│ fatigue (monthly retention of the effect) ┆ 0.96      ┆ 1.00       ┆ -950         ┆ 9743          ┆ 10693 │
│ pull-forward share                        ┆ 0.30      ┆ 0.00       ┆ 1496         ┆ 11464         ┆ 9968  │
│ margin                                    ┆ 0.25      ┆ 0.35       ┆ 1773         ┆ 11187         ┆ 9414  │
│ coupon cost ($)                           ┆ 6.00      ┆ 4.00       ┆ 2266         ┆ 10693         ┆ 8427  │
│ uplift (95% CI of the experiment)         ┆ 12.35     ┆ 16.04      ┆ 2808         ┆ 10151         ┆ 7343  │
│ customers reached                         ┆ 300000.00 ┆ 500000.00  ┆ 4686         ┆ 8273          ┆ 3587  │
│ running cost ($/month)                    ┆ 50000.00  ┆ 15000.00   ┆ 5936         ┆ 6697          ┆ 761   │
│ build cost ($)                            ┆ 300000.00 ┆ 100000.00  ┆ 6330         ┆ 6530          ┆ 200   │
Base NPV: $6,480k

Open figures/day66_tornado.png. The two biggest bars are assumptions nobody has measured: fatigue and pull-forward. The experiment's uncertainty (uplift) comes fifth. IT's costs, which usually get the most scrutiny in a budget meeting, barely matter.

The tornado tells you what to learn next. A one-month experiment can't measure fatigue; a holdout kept for six months can. Day 68 puts a dollar value on that.

66.4 How much fatigue can it take? (35 min)

Try it. Find the fatigue value at which the NPV is exactly zero (bisection: try the midpoint, keep the half where the sign changes). Create day66/break_even_fatigue.py.

# day66/break_even_fatigue.py   (python -m day66.break_even_fatigue)
import json
import math

from day66.cba import npv

base = json.load(open("data/day66_inputs.json"))
lo, hi = 0.90, 1.0
for _ in range(40):
    mid = (lo + hi) / 2
    lo, hi = (mid, hi) if npv(base | {"fatigue": mid})[0] < 0 else (lo, mid)
print(f"NPV is zero when the effect keeps {hi:.4f} of its size each month "
      f"(it halves every {math.log(0.5) / math.log(hi):.0f} months)")

NPV is zero when the effect keeps 0.9646 of its size each month (it halves every 19 months)

66.5 Day 66 mini-project: everything uncertain at once (65 min)

A tornado moves one input at a time. Draw them all together, 10,000 times. Create day66/npv_monte_carlo.py:

# day66/npv_monte_carlo.py   (python -m day66.npv_monte_carlo)
"""
Mini-project: a tornado moves one input at a time. Reality moves them all at once. Draw every uncertain
input from a distribution (uniform over the tornado's range; the uplift from its sampling distribution),
10,000 times, and look at the distribution of NPV. Run day66.cba first.
"""
import json

import numpy as np

from day66.cba import npv

base = json.load(open("data/day66_inputs.json"))
rng = np.random.default_rng(66)
draws = []
for _ in range(10_000):
    a = base | {"fatigue": rng.uniform(0.96, 1.00), "pull_forward": rng.uniform(0.0, 0.30),
                "uplift": rng.normal(base["uplift"], base["uplift"] * 0.066), "margin": rng.uniform(0.25, 0.35),
                "coupon_cost": rng.uniform(4.0, 6.0), "customers": rng.uniform(300_000, 500_000),
                "running_cost": rng.uniform(15_000, 50_000), "build_cost": rng.uniform(100_000, 300_000)}
    draws.append(npv(a)[0])
d = np.array(draws) / 1000
print(f"NPV ($k): mean {d.mean():,.0f}, median {np.median(d):,.0f}, 5th percentile {np.percentile(d, 5):,.0f}, "
      f"95th percentile {np.percentile(d, 95):,.0f}")
print(f"Probability the programme loses money over two years: {(d < 0).mean():.1%}")

python -m day66.npv_monte_carlo

NPV ($k): mean 3,618, median 3,128, 5th percentile -4,900, 95th percentile 13,725
Probability the programme loses money over two years: 27.4%

The expected NPV ($3.6M) is well below the base case ($6.5M). Nothing is wrong: NPV is not a straight-line function of fatigue (effects compound month after month), so averaging over uncertainty isn't the same as plugging in the average. And there's a 27% chance of losing money. A base-case NPV alone would have hidden both facts.

Tomorrow: what to do about that 27%.

66.6 Save your work with Git (20 min)

git add .
git commit -m "Day 66: programme cost-benefit analysis, NPV, tornado chart, Monte Carlo"
git push

Day 66 checklist

• I can turn a campaign result into a multi-period programme NPV, with pull-forward and fatigue
• I separate measured inputs from assumptions, and give every assumption a range
• I can build a tornado chart and use it to decide what to learn next
• I can find and explain a break-even value for an assumption
• I know why the expected NPV differs from the base-case NPV