Monitoring Estimates over Time

77.1 Causal estimates go stale (40 min)

An elasticity is a fact about customers at a time. Competitors open stores, incomes change, a segment's mix shifts. Prediction models are monitored for data drift (inputs change) and performance drift (accuracy falls). Causal parameters need a third kind: effect drift. The same price change now does something different, and nothing in the inputs warns you.

You can't monitor an effect by comparing predictions with outcomes week by week: that confuses effects with everything else that moves sales. You monitor it the way you estimated it: re-run the identification strategy on recent data, and compare.

The weekly feed after the simulator went live. Create data/make_monitoring_data.py:

"""
Module 8 monitoring data: the weekly feed after the simulator went live. Provided.

    python data/make_monitoring_data.py

Writes data/scenario/weekly_feed.csv: 130 weeks of store-category sales. Something changes partway through;
the monitoring on Day 77 should find what and when, without being told.
"""
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from causalkit.demand import demand, public  # noqa: E402

root = Path(__file__).parent
(root / "scenario").mkdir(exist_ok=True)
public(demand(seed=77, weeks=130, drift_week=105, drift_segment="student", drift_factor=1.3)).write_csv(
    root / "scenario" / "weekly_feed.csv")
print("Wrote data/scenario/weekly_feed.csv (130 weeks)")

python data/make_monitoring_data.py

Wrote data/scenario/weekly_feed.csv (130 weeks)

Something changes partway through. Today's job is to find what and when, without looking at the generator's arguments.

77.2 A rolling re-estimate (50 min)

Every two weeks, re-estimate each segment's elasticity on the last 13 weeks, with Module 6's 2SLS (supplier cost as the instrument). 2SLS rather than DML: it's fast, it has an honest standard error, and in a short window flexible ML has too little data. Create day77/monitor.py:

# day77/monitor.py   (python -m day77.monitor)
"""
Monitoring causal estimates: re-estimate each segment's elasticity on a rolling 13-week window of the
weekly feed (2SLS with the supplier-cost instrument, Module 6), and compare it with what the simulator
is using. Alert when the estimate falls outside the reference value +/- 3 standard errors in three windows
in a row (six weeks): single excursions happen, because neighbouring windows share most of their data.
"""
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import polars as pl
import pyfixest as pf

from engine import load_parameters

WINDOW, STEP = 13, 2


def rolling_estimates(feed: pd.DataFrame) -> pd.DataFrame:
    rows = []
    for end in range(WINDOW, feed.week.max() + 1, STEP):
        win = feed[(feed.week > end - WINDOW) & (feed.week <= end)]
        for seg, s in win.groupby("segment"):
            fit = pf.feols("log_units ~ promo | store_id^category + week | log_price ~ cost_index", s,
                           vcov={"CRV1": "store_id"})
            rows.append({"segment": seg, "end_week": end, "estimate": fit.coef()["log_price"],
                         "se": fit.se()["log_price"]})
    return pd.DataFrame(rows)


def reference(params: dict) -> dict:
    """The simulator's view of each segment: the average of its cells' elasticities."""
    cells = pd.DataFrame(params["pricing"])
    return cells.groupby("segment").elasticity.mean().to_dict()


def alerts(est: pd.DataFrame, ref: dict) -> pd.DataFrame:
    est = est.assign(reference=est.segment.map(ref))
    est["outside"] = (est.estimate - est.reference).abs() > 3 * est.se
    est["alert"] = est.groupby("segment").outside.transform(
        lambda s: s & s.shift(1, fill_value=False) & s.shift(2, fill_value=False))
    return est


if __name__ == "__main__":
    feed = pd.read_csv("data/scenario/weekly_feed.csv")
    est = alerts(rolling_estimates(feed), reference(load_parameters()))
    est.to_csv("data/day77_rolling.csv", index=False)
    first = est[est.alert].groupby("segment").end_week.min()
    print("Typical standard error of a 13-week segment estimate:", f"{est.se.median():.2f}")
    summary = est.groupby("segment").agg(reference=("reference", "first"), windows=("outside", "size"),
                                         windows_outside=("outside", "sum")).reset_index()
    summary["first_alert_week"] = summary.segment.map(first).astype("Int64")
    print(pl.from_pandas(summary).with_columns(pl.col("reference").round(2)))
    late = est[est.end_week >= 120].groupby("segment").estimate.mean().round(2)
    print("Average estimate in windows ending week 120 or later:", late.to_dict())

    fig, axes = plt.subplots(2, 2, figsize=(11, 6), sharex=True)
    for ax, (seg, s) in zip(axes.ravel(), est.groupby("segment")):
        ax.fill_between(s.end_week, s.reference - 3 * s.se, s.reference + 3 * s.se, color="#cbd5e1", alpha=0.6)
        ax.plot(s.end_week, s.estimate, color="#2563eb", marker="o", ms=3)
        ax.plot(s.end_week[s.alert], s.estimate[s.alert], "o", color="#dc2626", ms=6)
        ax.axhline(s.reference.iloc[0], color="#334155", lw=1)
        ax.set_title(seg)
    for ax in axes[1]:
        ax.set_xlabel("last week of the 13-week window")
    fig.tight_layout()
    fig.savefig("figures/day77_monitor.png", dpi=120)
    print("Saved figures/day77_monitor.png")

python -m day77.monitor

Typical standard error of a 13-week segment estimate: 0.21
│ segment         ┆ reference ┆ windows ┆ windows_outside ┆ first_alert_week │
│ rural value     ┆ -3.15     ┆ 59      ┆ 2               ┆ null             │
│ student         ┆ -3.53     ┆ 59      ┆ 7               ┆ 121              │
│ suburban family ┆ -2.54     ┆ 59      ┆ 3               ┆ null             │
│ urban premium   ┆ -1.38     ┆ 59      ┆ 2               ┆ null             │
Average estimate in windows ending week 120 or later: {'rural value': -3.78, 'student': -5.19, 'suburban family': -2.8, 'urban premium': -1.31}
Saved figures/day77_monitor.png

Open the figure. Student stores drift below the band from about week 110 and stay there: their elasticity went from about −3.5 to about −5.2, roughly 1.3 to 1.5 times as price-sensitive. The first alert comes at week 121. (The simulator's truth: ×1.3 from week 105. The monitor needed about 16 weeks to be sure.)

77.3 Alert rules: sensitivity vs false alarms (45 min)

Every alert rule trades detection speed for false alarms:

Rural value's late average (−3.78 vs −3.15) looks alarming in the table, but only 2 of its 59 windows were outside the band and never three in a row: it's noise. The alert rule is the thing to agree with the business before the first alert, so nobody argues about it under pressure.

Try it. Change the window to 8 and to 26 weeks (set day77.monitor.WINDOW before calling the functions). What happens to the standard errors and the alerts? Create day77/try_window.py.

# day77/try_window.py   (python -m day77.try_window)
import pandas as pd

import day77.monitor as mon
from engine import load_parameters

feed = pd.read_csv("data/scenario/weekly_feed.csv")
for w in [8, 13, 26]:
    mon.WINDOW = w
    est = mon.alerts(mon.rolling_estimates(feed), mon.reference(load_parameters()))
    first = est[est.alert].groupby("segment").end_week.min().to_dict()
    print(f"window {w:2} weeks: median SE {est.se.median():.2f}, first alerts {first}")

window  8 weeks: median SE 0.27, first alerts {'student': 120}
window 13 weeks: median SE 0.21, first alerts {'student': 121}
window 26 weeks: median SE 0.15, first alerts {'rural value': 40, 'student': 54, 'suburban family': 40, 'urban premium': 38}

77.4 What a monitor can't see (40 min)

• Drift in the instrument. If supplier costs stop moving prices (a long-term contract), 2SLS loses its first stage and the monitor goes blind. Track the first-stage F alongside the estimate.
• Drift inside a segment. A segment average can stay put while two categories move in opposite directions. Monitor cells when you have the data for it; here cells are too noisy at 13 weeks.
• Drift in coupons. The coupon curve came from one experiment. Nothing in weekly sales data re-identifies it; only a holdout (Module 7, Day 68) does. Keep a 5% holdout and monitor its uplift instead.

77.5 Day 77 mini-project: CUSUM (75 min)

Rolling windows overlap, so their alerts are correlated and hard to calibrate. Page's CUSUM (1954) uses non-overlapping blocks and accumulates evidence: S = max(0, S + z − k), alarm when S > h. A persistent shift of size k or more builds up; noise drains away. Create day77/cusum.py:

# day77/cusum.py   (python -m day77.cusum)
"""
Mini-project: a CUSUM chart (Page, 1954) for elasticities. Estimate each segment's elasticity on
NON-overlapping 4-week blocks (independent, but noisy), standardise the gap from the reference, and add
it up: S = max(0, S_previous + z - k). A small persistent shift accumulates; noise doesn't. Alert at S > h.
Here z measures 'more price-sensitive than the reference' (the direction a competitor would push).
"""
import pandas as pd
import polars as pl
import pyfixest as pf

from day77.monitor import reference
from engine import load_parameters

BLOCK, K, H = 4, 0.5, 5.0
feed = pd.read_csv("data/scenario/weekly_feed.csv")
ref = reference(load_parameters())
rows = []
for seg, s in feed.groupby("segment"):
    cusum, alarm = 0.0, None
    for start in range(1, feed.week.max() + 1, BLOCK):
        block = s[(s.week >= start) & (s.week < start + BLOCK)]
        fit = pf.feols("log_units ~ promo | store_id^category + week | log_price ~ cost_index", block,
                       vcov={"CRV1": "store_id"})
        z = (ref[seg] - fit.coef()["log_price"]) / fit.se()["log_price"]       # > 0: more elastic than reference
        cusum = max(0.0, cusum + z - K)
        if alarm is None and cusum > H:
            alarm = start + BLOCK - 1
    rows.append({"segment": seg, "alarm at week": alarm, "final CUSUM": round(cusum, 1)})
print(pl.DataFrame(rows))
print(f"Blocks of {BLOCK} weeks, k = {K}, h = {H}.")

python -m day77.cusum

│ segment         ┆ alarm at week ┆ final CUSUM │
│ rural value     ┆ null          ┆ 3.5         │
│ student         ┆ 116           ┆ 13.8        │
│ suburban family ┆ null          ┆ 0.5         │
│ urban premium   ┆ null          ┆ 0.0         │
Blocks of 4 weeks, k = 0.5, h = 5.0.

Student alarms at week 116, five weeks before the rolling monitor, with no false alarms elsewhere. Rural's 3.5 is worth watching. CUSUM is designed for exactly this: small, persistent, one-directional shifts. Use it as the primary alarm and the rolling chart as the picture you show.

77.6 Save your work with Git (20 min)

git add data/make_monitoring_data.py day77
git commit -m "Day 77: rolling 2SLS monitor and CUSUM for elasticity drift"
git push

Day 77 checklist

• I monitor effects by re-running the identification strategy, not by tracking forecasts
• I can explain my alert rule's trade-off between speed and false alarms
• I know that a monitor compares two noisy estimates
• CUSUM found the drift earlier than the rolling window, without false alarms