The Value of Information

68.1 Information is worth what it changes (40 min)

A study is worth running only if its result could change the decision, and only by how much the better decision earns. If you'd launch whatever the study found, it's worth nothing to this decision, however interesting.

The decision today is Day 61's: NorthMart's marketing team, working from last quarter's observational data, wants to send a $5 coupon to all 400,000 customers every month for a year. The DML estimate ($24.6) is comfortably above the $16.67 break-even. Should they launch, test first, or something in between?

Two quantities:

• EVPI, the expected value of perfect information: how much better, on average, we'd decide if we knew the true effect. The ceiling on what any study can be worth.
• EVSI, the expected value of sample information: the same for a real study with a given size and noise.

68.2 A prior that includes the sensitivity analysis (40 min)

Value of information needs a belief about the effect, uncertainty included. The sampling SE ($0.64) is the smallest part of the uncertainty. Day 61's benchmarks said a hidden confounder as strong as app use would inflate the estimate by about $5, and we can't rule out something stronger. So the belief: the true effect = $24.6 − (a bias somewhere between $0 and $10) ± sampling error.

Create day68/prior.py:

# day68/prior.py   (python -m day68.prior)
"""
The value of information starts from what you believe now, uncertainty included. The decision: send a $5
coupon to all 400,000 customers every month for a year? It pays if the average effect exceeds $16.67.
What we know (Day 61): observational DML says $24.6 (SE 0.6), but a hidden confounder could inflate it.
Benchmarks said a confounder as strong as app use would move it by about $5, one as strong as deal-seeking
by about $2.5; we can't rule out something stronger. Belief: bias somewhere between $0 and $10, all values
equally plausible. Prior for the true effect = 24.6 - bias + sampling error.
"""
import numpy as np

ESTIMATE, SE, BIAS_MAX = 24.6, 0.64, 10.0
BREAK_EVEN = 5 / 0.30
CUSTOMER_MONTHS = 400_000 * 12
DOLLARS_PER_DOLLAR_OF_EFFECT = 0.30 * CUSTOMER_MONTHS     # NPV changes by this much per $1 of average effect


def prior_draws(n=200_000, seed=68):
    rng = np.random.default_rng(seed)
    return ESTIMATE - rng.uniform(0, BIAS_MAX, n) + rng.normal(0, SE, n)


def launch_value(effect):
    """Net value ($) of a year of coupons for everyone, if the true average effect is `effect`."""
    return DOLLARS_PER_DOLLAR_OF_EFFECT * (effect - BREAK_EVEN)


if __name__ == "__main__":
    theta = prior_draws()
    print(f"Prior for the true effect: mean {theta.mean():.1f}, SD {theta.std():.1f}, "
          f"90% range {np.percentile(theta, 5):.1f} to {np.percentile(theta, 95):.1f}")
    print(f"P(effect above break-even ${BREAK_EVEN:.2f}): {(theta > BREAK_EVEN).mean():.0%}")
    v = launch_value(theta)
    print(f"Launch: expected value ${v.mean() / 1e6:.1f}M, P(loss) {(v < 0).mean():.0%}, "
          f"expected loss when it loses ${-v[v < 0].mean() / 1e6:.1f}M")

python -m day68.prior

Prior for the true effect: mean 19.6, SD 3.0, 90% range 15.0 to 24.2
P(effect above break-even $16.67): 79%
Launch: expected value $4.2M, P(loss) 21%, expected loss when it loses $1.6M

This is the step that connects Days 61–63 to the decision: the sensitivity analysis becomes a probability distribution, and the "fragile" robustness value becomes a 21% chance that the launch loses money.

68.3 EVPI: the ceiling (35 min)

With perfect information we'd launch only when it pays: E[max(value, 0)]. Without it, we choose the better option on average: max(E[value], 0). Create day68/evpi.py:

# day68/evpi.py   (python -m day68.evpi)
"""
Expected value of perfect information (EVPI): how much better would we do, on average, if we KNEW the true
effect before deciding? With perfect information we'd launch only when it pays: E[max(value, 0)].
Without it we pick the better option on average: max(E[value], 0). The difference caps what ANY study is worth.
"""
import numpy as np

from day68.prior import launch_value, prior_draws

v = launch_value(prior_draws())
with_info = np.maximum(v, 0).mean()
without = max(v.mean(), 0)
print(f"Decide now:                     ${without / 1e6:.2f}M  ({'launch' if v.mean() > 0 else 'do nothing'})")
print(f"Decide knowing the true effect: ${with_info / 1e6:.2f}M")
print(f"EVPI:                           ${(with_info - without) / 1e6:.2f}M")

python -m day68.evpi

Decide now:                     $4.23M  (launch)
Decide knowing the true effect: $4.57M
EVPI:                           $0.34M

Perfect information about the effect is worth at most $340,000 for this decision. Any study costing more isn't worth it. The EVPI is small relative to the stakes because we'd probably launch anyway (79%); information only pays in the 21% of worlds where it would stop us.

68.4 EVSI: test first, or launch and learn? (60 min)

Two ways to learn:

• B. Test first: a one-month experiment with n customers per arm, then decide for the remaining 11 months. Learning costs a month of the programme.
• C. Launch with a holdout: launch to everyone except a random holdout of n customers; after month 1, compare; stop if the evidence says it loses money. Learning costs only the holdout's month.

The math (normal approximation): a test with standard error s moves the belief to a posterior mean m′; before the test, m′ is itself uncertain, with variance = prior variance − posterior variance. The value of deciding after the test is E[max(0, k(m′ − break-even))], which has a closed form. Create day68/evsi.py:

# day68/evsi.py   (python -m day68.evsi)
"""
Expected value of SAMPLE information: a real study is smaller than perfect information. Three ways to act:
  A. launch to everyone now, for 12 months
  B. test first: a 1-month experiment with n customers per arm, then decide for the remaining 11 months
  C. launch now, but keep a random holdout of n customers for the first month; after month 1, compare,
     and stop the programme for the remaining 11 months if the evidence says it loses money
Normal approximation (the prior is approximately normal): a test with standard error s moves our belief
to a posterior mean m'. Before the test, m' is uncertain: normal around today's mean with variance
(prior variance - posterior variance). Value of deciding after the test = E[max(0, k x (m' - break-even))].
"""
import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import norm

from day68.prior import BREAK_EVEN, DOLLARS_PER_DOLLAR_OF_EFFECT, prior_draws

RESIDUAL_SD = 38.8        # SD of spend after adjusting for the features (an experiment's analysis, Day 24)
CUSTOMERS = 400_000
FIXED_TEST_COST = 30_000  # setting up and analysing a test

theta = prior_draws()
mu0, sd0 = theta.mean(), theta.std()
per_month = DOLLARS_PER_DOLLAR_OF_EFFECT / 12              # $ per $1 of effect, per month, all customers


def expected_max0(mean, sd):
    """E[max(0, X)] for X ~ Normal(mean, sd)."""
    if sd == 0:
        return max(mean, 0.0)
    z = mean / sd
    return mean * norm.cdf(z) + sd * norm.pdf(z)


def value_after_learning(test_se, months):
    """Expected value of deciding, for `months` months, after a test with standard error test_se."""
    post_var = 1 / (1 / sd0**2 + 1 / test_se**2)
    spread = np.sqrt(sd0**2 - post_var)                     # SD of the posterior mean, before the test
    k = per_month * months
    return expected_max0(k * (mu0 - BREAK_EVEN), k * spread)


if __name__ == "__main__":
    A = per_month * 12 * (mu0 - BREAK_EVEN)
    rows, ns = [], [500, 1000, 2000, 5000, 10000, 20000, 50000]
    for n in ns:
        test_first = value_after_learning(RESIDUAL_SD * np.sqrt(2 / n), 11) - FIXED_TEST_COST
        holdout = (per_month * (mu0 - BREAK_EVEN) * (1 - n / CUSTOMERS)
                   + value_after_learning(RESIDUAL_SD * np.sqrt(1 / n + 1 / (CUSTOMERS - n)), 11) - FIXED_TEST_COST)
        rows.append((n, test_first, holdout))
    print(f"Prior: mean {mu0:.1f}, SD {sd0:.1f}.   A. launch now (12 months): ${A / 1e6:.2f}M")
    print(f"{'n per arm':>10} {'B. test first':>15} {'C. launch with holdout':>24}")
    for n, b, c in rows:
        print(f"{n:>10,} {b / 1e6:>14.2f}M {c / 1e6:>23.2f}M")
    best = max(rows, key=lambda r: r[2])
    print(f"Best: launch with a holdout of {best[0]:,} customers: ${best[2] / 1e6:.2f}M, "
          f"${(best[2] - A) / 1e6:.2f}M more than launching blind")

    fig, ax = plt.subplots(figsize=(7, 4))
    ax.plot(ns, [r[1] / 1e6 for r in rows], marker="o", label="B. test first, then decide")
    ax.plot(ns, [r[2] / 1e6 for r in rows], marker="o", label="C. launch with a holdout")
    ax.axhline(A / 1e6, color="#64748b", ls="--", label="A. launch now")
    ax.set_xscale("log")
    ax.set_xlabel("customers per arm in the test")
    ax.set_ylabel("expected value of the year ($M)")
    ax.legend()
    fig.tight_layout()
    fig.savefig("figures/day68_evsi.png", dpi=120)
    print("Saved figures/day68_evsi.png")

python -m day68.evsi

Prior: mean 19.6, SD 3.0.   A. launch now (12 months): $4.23M
 n per arm   B. test first   C. launch with holdout
       500           3.99M                    4.41M
     1,000           4.05M                    4.46M
     2,000           4.11M                    4.49M
     5,000           4.14M                    4.51M
    10,000           4.16M                    4.51M
    20,000           4.17M                    4.51M
    50,000           4.17M                    4.48M
Best: launch with a holdout of 10,000 customers: $4.51M, $0.28M more than launching blind
Saved figures/day68_evsi.png

• Testing first is worse than launching now, at every size. The month of delay costs about $350,000 in expected value (a twelfth of $4.2M), more than the information is worth.
• Launching with a holdout beats both. It captures $0.28M of the $0.34M EVPI, for the cost of not couponing 5,000–20,000 customers for one month.
• Beyond about 10,000 customers, the holdout stops helping: the test is already precise enough, and more holdout customers just forgo profitable coupons.

"Test or launch?" is often a false choice. Launch with a holdout and a stop rule combines Day 67's flexibility with Day 68's information.

Try it. How does the EVPI change if you trust the observational estimate more: a possible bias of up to $5, or up to $2? Create day68/more_trust.py.

# day68/more_trust.py   (python -m day68.more_trust)
import numpy as np

import day68.prior as prior
from day68.prior import launch_value

for bias_max in [10.0, 5.0, 2.0]:
    prior.BIAS_MAX = bias_max
    v = launch_value(prior.prior_draws())
    print(f"bias up to ${bias_max:>4.1f}: P(loss) {(v < 0).mean():.0%}, EVPI ${(np.maximum(v, 0).mean() - max(v.mean(), 0)) / 1e6:.2f}M")

bias up to $10.0: P(loss) 21%, EVPI $0.34M
bias up to $ 5.0: P(loss) 0%, EVPI $0.00M
bias up to $ 2.0: P(loss) 0%, EVPI $0.00M

68.5 Day 68 mini-project: what would actually have happened (65 min)

Check the formula by simulation (with the real, not-quite-normal prior), then run the holdout plan once against NorthMart's real effect from the simulator. Create day68/simulate_holdout.py:

# day68/simulate_holdout.py   (python -m day68.simulate_holdout)
"""
Mini-project: check the formula by simulation, then ask what would ACTUALLY happen at NorthMart.
For each of 200,000 draws of the true effect from the (not quite normal) prior: run the month-1 holdout
(10,000 customers), update the belief, continue only if the posterior mean is above break-even, and record
the value actually earned. Then do the same once with the simulator's real effect.
"""
import numpy as np

from causalkit.coupons import coupons, truth
from day68.evsi import CUSTOMERS, FIXED_TEST_COST, RESIDUAL_SD, mu0, per_month, sd0
from day68.prior import BREAK_EVEN, prior_draws

N = 10_000
se = RESIDUAL_SD * np.sqrt(1 / N + 1 / (CUSTOMERS - N))


def year_with_holdout(theta, rng):
    estimate = theta + rng.normal(0, se, np.shape(theta))
    post_mean = (mu0 / sd0**2 + estimate / se**2) / (1 / sd0**2 + 1 / se**2)
    month1 = per_month * (theta - BREAK_EVEN) * (1 - N / CUSTOMERS)
    rest = np.where(post_mean > BREAK_EVEN, per_month * 11 * (theta - BREAK_EVEN), 0.0)
    return month1 + rest - FIXED_TEST_COST, post_mean > BREAK_EVEN


rng = np.random.default_rng(1)
theta = prior_draws()
value, cont = year_with_holdout(theta, rng)
blind = per_month * 12 * (theta - BREAK_EVEN)
print(f"Simulated over the prior: launch blind ${blind.mean() / 1e6:.2f}M; launch with holdout ${value.mean() / 1e6:.2f}M "
      f"(continues in {cont.mean():.0%} of draws)")
true_effect = truth(coupons())["ATE"]
v, c = year_with_holdout(np.array([true_effect]), np.random.default_rng(2))
print(f"NorthMart's real average effect: ${true_effect:.2f} (break-even ${BREAK_EVEN:.2f})")
print(f"   launch blind:        ${per_month * 12 * (true_effect - BREAK_EVEN) / 1e6:+.2f}M")
print(f"   launch with holdout: ${v[0] / 1e6:+.2f}M  ({'continued' if c[0] else 'stopped after month 1'})")

python -m day68.simulate_holdout

Simulated over the prior: launch blind $4.23M; launch with holdout $4.49M (continues in 80% of draws)
NorthMart's real average effect: $16.51 (break-even $16.67)
   launch blind:        $-0.23M
   launch with holdout: $-0.05M  (stopped after month 1)

The simulation matches the formula ($4.49M vs $4.51M). And in NorthMart's actual world, where the observational estimate was inflated by the browsing signal and the true effect is just below break-even, launching blind would have lost $230,000; launching with a 10,000-customer holdout loses $50,000 (one month, plus the test's cost) and stops.

That's the whole module in one line: the estimate was wrong, the sensitivity analysis said it might be, the value-of-information analysis said how to find out cheaply, and the plan that followed lost a fifth as much.

68.6 Save your work with Git (20 min)

git add .
git commit -m "Day 68: prior from sensitivity analysis, EVPI, EVSI, launch with a holdout"
git push

Day 68 checklist

• I can build a prior that includes sensitivity to hidden confounding, not just sampling error
• I can compute EVPI and explain why it caps the value of any study
• I can compute EVSI for a test of a given size, with a normal approximation
• I can compare "test first" with "launch with a holdout and a stop rule"
• I can explain why the value of a test depends on the sensitivity analysis