When to Re-Run the Experiment

78.1 Rebuild, re-test, or carry on (40 min)

When the monitor fires, there are three responses, in increasing cost:

Module 7's value-of-information logic (Day 68) decides between them: do it if the expected improvement in decisions is worth more than it costs.

The demand() simulator can now run a price test: price_test={"segment": "student", "start": 105, "weeks": 8, "change": 0.10} raises prices by 10% in a random half of student stores for 8 weeks, and adds a price_test column.

78.2 What stale estimates cost (50 min)

Suppose every student elasticity is now 1.3 times what the parameters file says (Day 77's finding). For each student cell, compare a +5% decision under the stale and the updated values. Create day78/stale_cost.py:

# day78/stale_cost.py   (python -m day78.stale_cost)
"""
What does acting on stale estimates cost? The monitoring says students became more price-sensitive.
Suppose (as the simulator knows) every student elasticity is now 1.3 times what the parameters file says.
For each student cell, compare the +5% price decision under the stale and the updated elasticities.
"""
import copy

import polars as pl

from engine import Scenario, load_parameters, simulate

stale = load_parameters()
updated = copy.deepcopy(stale)
for c in updated["pricing"]:
    if c["segment"] == "student":
        c["elasticity"] = round(c["elasticity"] * 1.3, 3)

rows = []
for c in [c for c in stale["pricing"] if c["segment"] == "student"]:
    key = f"student/{c['category']}"
    sc = Scenario(price_change={key: 0.05}, seed=1)
    old = next(x for x in simulate(stale, sc)["cells"] if (x["segment"], x["category"]) == ("student", c["category"]))
    new = next(x for x in simulate(updated, sc)["cells"] if (x["segment"], x["category"]) == ("student", c["category"]))
    rows.append({"cell": key, "stale elasticity": c["elasticity"], "updated": round(c["elasticity"] * 1.3, 2),
                 "+5% profit, stale view ($k/yr)": round(old["profit_change_median"] / 1e3, 1),
                 "+5% profit, updated ($k/yr)": round(new["profit_change_median"] / 1e3, 1),
                 "decision flips?": "YES" if (old["profit_change_median"] > 0) != (new["profit_change_median"] > 0) else ""})
with pl.Config(tbl_width_chars=140, fmt_str_lengths=30):
    print(pl.DataFrame(rows))
plan = Scenario(price_change={"student/pantry": 0.05})
print(f"Day 60's plan raises student pantry prices. Stale view: "
      f"{simulate(stale, plan)['change']['gross_profit']['median'] / 1e3:+.1f} $k/yr; "
      f"updated: {simulate(updated, plan)['change']['gross_profit']['median'] / 1e3:+.1f} $k/yr")

python -m day78.stale_cost

│ cell                  ┆ stale elasticity ┆ updated ┆ +5% profit, stale view ($k/yr) ┆ +5% profit, updated ($k/yr) ┆ decision flips? │
│ student/fresh         ┆ -3.522           ┆ -4.58   ┆ -3.5                           ┆ -26.1                       ┆                 │
│ student/household     ┆ -3.811           ┆ -4.95   ┆ -36.4                          ┆ -97.4                       ┆                 │
│ student/pantry        ┆ -3.114           ┆ -4.05   ┆ 18.0                           ┆ 2.5                         ┆                 │
│ student/personal care ┆ -2.828           ┆ -3.68   ┆ -5.5                           ┆ -57.2                       ┆                 │
│ student/snacks        ┆ -4.37            ┆ -5.68   ┆ -23.3                          ┆ -47.1                       ┆                 │
Day 60's plan raises student pantry prices. Stale view: +18.0 $k/yr; updated: +2.4 $k/yr

No decision flips: pantry still gains, the others still lose, just by more. The plan's value falls by 87%, but its sign holds. Drift matters for decisions only when it moves a cell across break-even. That's why "the monitor fired" isn't by itself a reason for an expensive re-test.

78.3 Designing the re-test (45 min)

Concept. A randomised price test identifies the elasticity with no assumptions about the pricing team: the random assignment is the instrument (Module 4). The design choices are the size of the change, the length and the stores. Simulate before you run it. Create day78/test_design.py:

# day78/test_design.py   (python -m day78.test_design)
"""
Designing the re-test: raise prices in a random half of student stores for k weeks, and estimate the
student elasticity from the test alone (the random price change is the instrument: no assumptions about
the pricing team needed). The window includes the 8 weeks before the test, so each store is compared with
itself; cost_index soaks up supplier-cost noise in prices. Which design is precise enough?
Simulated 12 times per design; every design sees the same 12 markets (common random numbers),
so the differences between rows come from the design, not from luck.
"""
import numpy as np
import polars as pl
import pyfixest as pf

from causalkit.demand import demand

rows = []
for change in [0.05, 0.10]:
    for weeks in [4, 8, 13]:
        est, ses, fs_t, truth = [], [], [], []
        for s in range(12):
            df = demand(seed=7800 + s, weeks=117, drift_week=95, drift_segment="student", drift_factor=1.3,
                        price_test={"segment": "student", "start": 105, "weeks": weeks, "change": change})
            win = df.filter((pl.col("segment") == "student") & pl.col("week").is_between(97, 104 + weeks)).to_pandas()
            fe = "store_id^category + category^week"
            fs = pf.feols(f"log_price ~ price_test + cost_index + promo | {fe}", win, vcov={"CRV1": "store_id"})
            fit = pf.feols(f"log_units ~ promo + cost_index | {fe} | log_price ~ price_test", win,
                           vcov={"CRV1": "store_id"})
            fs_t.append(fs.tstat()["price_test"])
            est.append(fit.coef()["log_price"])
            ses.append(fit.se()["log_price"])
            truth.append(win._elasticity.mean())
        err = np.array(est) - np.array(truth)
        rows.append({"price change": f"{change:.0%}", "weeks": weeks, "first-stage t": round(np.median(fs_t), 1),
                     "average error": round(err.mean(), 2), "SD of estimates": round(err.std(), 2),
                     "median SE": round(np.median(ses), 2)})
print(pl.DataFrame(rows))
print("true student elasticity (average):", round(np.mean(truth), 2))

python -m day78.test_design

│ price change ┆ weeks ┆ first-stage t ┆ average error ┆ SD of estimates ┆ median SE │
│ 5%           ┆ 4     ┆ 2.0           ┆ -0.26         ┆ 0.93            ┆ 0.7       │
│ 5%           ┆ 8     ┆ 3.4           ┆ 0.05          ┆ 0.27            ┆ 0.53      │
│ 5%           ┆ 13    ┆ 2.8           ┆ -0.7          ┆ 1.11            ┆ 0.76      │
│ 10%          ┆ 4     ┆ 4.5           ┆ 0.03          ┆ 0.3             ┆ 0.34      │
│ 10%          ┆ 8     ┆ 4.9           ┆ -0.02         ┆ 0.32            ┆ 0.36      │
│ 10%          ┆ 13    ┆ 5.4           ┆ -0.14         ┆ 0.35            ┆ 0.31      │
true student elasticity (average): -4.73

(The analysis window starts 8 weeks before the test so each store is compared with itself. A first version that used only the test weeks failed with a singular matrix: with store × category and category × week fixed effects, a treatment that never varies within a store in the window is absorbed completely.)

78.4 Why size beats length (40 min)

Read the table by the first-stage t-statistic (Module 4's weak-instrument diagnostic):

• At 5%, the test barely moves prices relative to their normal week-to-week noise: t of 2 to 3.4, a weak instrument. Estimates are erratic (SD up to 1.1) and longer doesn't help: 13 weeks is worse than 8 in these 12 markets.
• At 10%, t is 4.5 or more at every length and the estimates are tight (SD about 0.3) from 4 weeks on.

Length adds weeks from the same 20 stores, and with errors clustered by store, the effective sample is the number of stores, not store-weeks. Doubling the price change doubles the signal in every one of them. Recommend: 10% for 4–8 weeks. (Cost: Day 78.5.)

78.5 Day 78 mini-project: value of information and re-run rules (75 min)

Is the test worth running? Our belief after Day 77: student elasticities are k times the file, with k ~ Normal(1.3, 0.15). Compare acting now with acting after perfect information (EVPI, an upper bound) and after the realistic test (SE 0.35 from 78.3), against what the test costs. Create day78/rerun_rules.py:

# day78/rerun_rules.py   (python -m day78.rerun_rules)   — Day 78 mini-project
"""
Is a new student price test worth running? Value of information, in dollars.

Decision: raise each student cell's price 5%, or leave it. Our belief about the drift (from Day 77's
monitor): student elasticities are now k times the parameters file, with k ~ Normal(1.3, 0.15).
  acting now     = pick, per cell, the action with the best EXPECTED profit under that belief
  perfect info   = learn k first, then pick the best action for that k
  EVPI           = E[perfect info] - acting now   (an upper bound on what any test is worth)
A real 8-week, 10% test does not reveal k exactly: Day 78's design gives SE about 0.35 on the elasticity,
so we also compute the value of THAT (imperfect) information. Then compare with what the test costs.
"""
import numpy as np

from engine import load_parameters

params = load_parameters()
cells = [c for c in params["pricing"] if c["segment"] == "student"]
HORIZON_YEARS = 1.0


def profit_change(c, d, k):
    """Annual gross-profit change of price change d in cell c if the elasticity is k * estimate."""
    e = c["elasticity"] * k
    base = c["margin"] * c["price"] * c["weekly_units"] * 52
    return (c["margin"] + d) * c["price"] * c["weekly_units"] * (1 + d) ** e * 52 - base


if __name__ == "__main__":
    rng = np.random.default_rng(78)
    k = rng.normal(1.3, 0.15, 20_000)             # belief about the drift factor
    now = perfect = sample = 0.0
    for c in cells:
        gain = profit_change(c, 0.05, k)              # profit of raising vs leaving, one value per belief draw
        now += max(gain.mean(), 0)                    # act on the expected value
        perfect += np.maximum(gain, 0).mean()         # act knowing k
        # imperfect test: observe the cell elasticity with noise SE 0.35, update by Bayes on a grid of k
        e_hat = c["elasticity"] * k + rng.normal(0, 0.35, k.size)
        grid = np.linspace(0.7, 1.9, 241)
        prior = np.exp(-0.5 * ((grid - 1.3) / 0.15) ** 2)
        like = np.exp(-0.5 * ((e_hat[:, None] - c["elasticity"] * grid[None, :]) / 0.35) ** 2)
        post = like * prior
        post /= post.sum(axis=1, keepdims=True)
        exp_gain = post @ profit_change(c, 0.05, grid)    # expected gain after seeing the test
        sample += np.where(exp_gain > 0, gain, 0).mean()

    test_cost = 15_000 + sum(                         # fixed cost + profit lost while half the student stores pay +10%
        max(-profit_change(c, 0.10, 1.3), 0) / 52 * 8 / 2 for c in cells)
    print(f"Expected profit change, acting now:        {now / 1e3:8.1f} $k/yr")
    print(f"With perfect information:                  {perfect / 1e3:8.1f} $k/yr   -> EVPI {(perfect - now) / 1e3:.1f} $k/yr")
    print(f"With an 8-week 10% test (SE 0.35):         {sample / 1e3:8.1f} $k/yr   -> value {(sample - now) / 1e3:.1f} $k/yr")
    print(f"Cost of the test:                          {test_cost / 1e3:8.1f} $k")
    print("Run the test" if (sample - now) * HORIZON_YEARS > test_cost else "Do not run the test: act on current beliefs")

    print("""
    Re-run rules for NorthMart (agreed with the pricing team):
      1. The monitor alerts on a segment (3 consecutive windows outside +-3 SE) or CUSUM alarms.
      2. A decision recommended by the app would flip under the monitor's latest estimate.
      3. The value of a test (as above) exceeds its cost over the decision horizon.
      4. Anyway, every 12 months: assumptions age even when no alarm fires.
      Rule 1 or 4 -> rebuild models/causal_parameters.json on recent data; rule 2 + 3 -> run the experiment.""")

python -m day78.rerun_rules

Expected profit change, acting now:             2.5 $k/yr
With perfect information:                       4.6 $k/yr   -> EVPI 2.2 $k/yr
With an 8-week 10% test (SE 0.35):              3.9 $k/yr   -> value 1.4 $k/yr
Cost of the test:                              50.9 $k
Do not run the test: act on current beliefs

The test would cost about $51k (mostly profit lost while half the student stores charge 10% more) to improve a decision worth $2.5k a year. Even perfect information is worth $2.2k a year. Don't test. Rebuild on recent data instead (free), and raise student pantry only.

The script also prints NorthMart's four re-run rules. Adopt them in writing: when the monitor alerts or a year has passed, rebuild; when a decision would flip and the test is worth more than it costs, re-test.

Try it. How does EVPI depend on how unsure we are about the drift? Recompute it for belief SDs of 0.05, 0.15 and 0.30. Create day78/try_voi.py (reuse cells and profit_change).

# day78/try_voi.py   (python -m day78.try_voi)
"""How does the value of perfect information change with how unsure we are about the drift?"""
import numpy as np

from day78.rerun_rules import cells, profit_change

rng = np.random.default_rng(1)
for sd in [0.05, 0.15, 0.30]:
    k = rng.normal(1.3, sd, 20_000)
    now = sum(max(profit_change(c, 0.05, k).mean(), 0) for c in cells)
    perfect = sum(np.maximum(profit_change(c, 0.05, k), 0).mean() for c in cells)
    print(f"belief SD {sd:.2f}: EVPI {(perfect - now) / 1e3:5.1f} $k/yr")

belief SD 0.05: EVPI   0.2 $k/yr
belief SD 0.15: EVPI   2.1 $k/yr
belief SD 0.30: EVPI  11.9 $k/yr

78.6 Save your work with Git (20 min)

git add day78 causalkit/demand.py
git commit -m "Day 78: cost of stale estimates, price-test design, value of information and re-run rules"
git push

Day 78 checklist

• I distinguish carrying on, rebuilding and re-testing, and know what each costs
• I checked whether the drift flips any decision before recommending action
• I simulated the re-test's precision before proposing it, and read its first stage
• I compared the value of information with the test's cost
• NorthMart has written re-run rules