Segment-Level Elasticities

59.1 One elasticity is not a pricing strategy (30 min)

NorthMart's four shopper segments differ: urban premium shoppers barely notice a price change, while students and rural value shoppers switch to cheaper stores or brands. Categories differ too: snacks are easy to skip, personal care isn't. The decision on Day 60 is made cell by cell (segment × category), so the elasticity must be too.

This is Days 55–57's heterogeneity question with a continuous treatment. Two approaches today:

1. Split and estimate: run Day 58's DML separately in each segment and each cell.
2. One model: EconML's LinearDML makes the elasticity a linear function of segment and category features, θ(x) = x′β, with ML nuisance models for everything else.

59.2 Elasticities by segment and by cell (60 min)

Create day59/segment_elasticities.py (about a minute):

# day59/segment_elasticities.py   (python -m day59.segment_elasticities)
"""
One elasticity for all of NorthMart hides what pricing needs: students and rural shoppers react very
differently from urban premium shoppers. Run Day 58's DML separately for each segment, then for each
segment x category cell (20 cells of about 2,000 store-weeks each).
"""
import polars as pl

from causalkit.demand import demand, public, truth
from day58.dml_elasticity import dml_elasticity, prepare

full = demand()
df = prepare(public(full).to_pandas())
true = {(r["segment"], r["category"]): r["elasticity"] for r in truth(full).iter_rows(named=True)}

rows = []
for seg, s in df.groupby("segment"):
    fit = dml_elasticity(s, drivers=["promo", "temperature", "holiday", "local_event", "competitor_index", "week", "cat"])
    rows.append({"segment": seg, "DML": round(fit.coef[0], 2), "SE": round(fit.se[0], 2),
                 "truth": round(sum(v for (sg, _), v in true.items() if sg == seg) / 5, 2)})
print(pl.DataFrame(rows))

cells = []
for (seg, cat), s in df.groupby(["segment", "category"]):
    fit = dml_elasticity(s, drivers=["promo", "temperature", "holiday", "local_event", "competitor_index", "week"])
    cells.append({"segment": seg, "category": cat, "DML": round(fit.coef[0], 2), "SE": round(fit.se[0], 2),
                  "truth": true[seg, cat]})
cells = pl.DataFrame(cells).with_columns(error=(pl.col("DML") - pl.col("truth")).round(2))
cells.write_csv("data/day59_cell_elasticities.csv")
with pl.Config(tbl_rows=20):
    print(cells)
print(f"Cells whose 95% CI contains the truth: {((cells['error'].abs()) <= 1.96 * cells['SE']).sum()} of 20")

python -m day59.segment_elasticities

│ segment         ┆ DML   ┆ SE   ┆ truth │
│ rural value     ┆ -3.17 ┆ 0.02 ┆ -3.23 │
│ student         ┆ -3.53 ┆ 0.02 ┆ -3.64 │
│ suburban family ┆ -2.57 ┆ 0.02 ┆ -2.63 │
│ urban premium   ┆ -1.33 ┆ 0.01 ┆ -1.41 │
│ segment         ┆ category      ┆ DML   ┆ SE   ┆ truth ┆ error │
│ rural value     ┆ fresh         ┆ -3.07 ┆ 0.03 ┆ -3.2  ┆ 0.13  │
│ rural value     ┆ household     ┆ -3.43 ┆ 0.03 ┆ -3.52 ┆ 0.09  │
...
│ urban premium   ┆ snacks        ┆ -1.62 ┆ 0.02 ┆ -1.75 ┆ 0.13  │
Cells whose 95% CI contains the truth: 4 of 20

The segments are clearly different (−1.3 to −3.5), and the cells add more: student snacks −4.4, urban personal care −1.1. Every estimate is within 0.15 of the truth.

But look at the last line: only 4 of 20 confidence intervals contain the truth. Every error has the same sign: DML is slightly too close to zero everywhere (by 0.08 on average, about 3%). The standard errors (0.02–0.03) only measure sampling noise. They don't include the small bias left by imperfect nuisance models: errors in the price model act like measurement error in price, and measurement error pulls slopes towards zero. It's small here because the data include a lot of price variation the drivers don't explain. With less, it would be bigger.

Lesson: a precise DML estimate can still be slightly biased, and its confidence interval won't tell you. Tomorrow's decision rule takes that into account.

59.3 One model for all cells: LinearDML (55 min)

Create day59/linear_dml.py:

# day59/linear_dml.py   (python -m day59.linear_dml)
"""
Twenty separate DML fits are fine for 20 cells, but not for 2,000. EconML's LinearDML fits ONE model:
the elasticity is a linear function of segment and category features, theta(x) = x' beta,
while the nuisance models still use all the demand drivers (W). Here x = segment and category dummies,
so the elasticity is 'segment effect + category effect', with confidence intervals for every cell.
"""
import numpy as np
import pandas as pd
import polars as pl
from econml.dml import LinearDML

from causalkit.demand import demand, public, truth
from causalkit.learners import reg
from day58.dml_elasticity import prepare

full = demand()
df = prepare(public(full).to_pandas())
X = pd.get_dummies(df[["segment", "category"]], drop_first=True).astype(float)
W = df[["promo", "temperature", "holiday", "local_event", "competitor_index", "week", "cat", "seg"]]
learner = reg(n_estimators=500, num_leaves=31, min_child_samples=20)
est = LinearDML(model_y=learner, model_t=learner, cv=5, random_state=0)
est.fit(df.y, df.d, X=X, W=W, groups=df.store)              # groups: folds keep each store together
print(est.summary(feature_names=list(X.columns)).tables[0])

cells = truth(full).to_pandas()
Xc = pd.get_dummies(cells[["segment", "category"]], drop_first=True).astype(float)[X.columns]
pt = est.effect(Xc)
lo, hi = est.effect_interval(Xc, alpha=0.05)
out = pl.DataFrame({"segment": cells.segment, "category": cells.category, "LinearDML": np.round(pt, 2),
                    "95% CI": [f"[{a:.2f}, {b:.2f}]" for a, b in zip(lo, hi)], "truth": cells.elasticity})
with pl.Config(tbl_rows=20, tbl_width_chars=100, fmt_str_lengths=30):
    print(out)
print(f"Mean absolute error {np.mean(np.abs(pt - cells.elasticity)):.2f}")

python -m day59.linear_dml

                        point_estimate stderr  zstat  pvalue ci_lower ci_upper
segment_student                 -0.374  0.019 -20.103    0.0    -0.41   -0.337
segment_suburban family          0.586  0.017   33.94    0.0    0.552     0.62
segment_urban premium            1.826  0.017 105.117    0.0    1.792     1.86
category_household              -0.295  0.019 -15.791    0.0   -0.332   -0.259
category_pantry                   0.25  0.019  13.296    0.0    0.213    0.287
category_personal care           0.491  0.019  26.011    0.0    0.454    0.528
category_snacks                 -0.654  0.021 -31.507    0.0   -0.695   -0.613
│ segment         ┆ category      ┆ LinearDML ┆ 95% CI         ┆ truth │
│ urban premium   ┆ personal care ┆ -0.81     ┆ [-0.85, -0.78] ┆ -1.12 │
│ urban premium   ┆ snacks        ┆ -1.96     ┆ [-2.00, -1.92] ┆ -1.75 │
...
Mean absolute error 0.11

The coefficients read as differences from the base cell (rural value, fresh): urban premium shoppers are 1.83 less elastic, snacks 0.65 more. One model, with confidence intervals for every cell, and it scales to thousands of cells, where splitting wouldn't. groups=df.store keeps each store's weeks in the same cross-fitting fold.

But the error is bigger than splitting (0.11 vs about 0.08), and the worst cells are off by 0.3: urban premium personal care is estimated at −0.81 against a true −1.12.

59.4 Additive or interacted? (40 min)

Concept. The model above is additive: segment effect + category effect. But true elasticities are multiplicative: a segment's sensitivity scales every category's (students are 2.6 times as elastic as urban premium shoppers in every category). An additive model can't represent that, however much data it has. The CATE model's functional form matters, just like the outcome model's did on Day 31.

Try it. Refit LinearDML with one dummy per segment × category cell (19 dummies plus the base cell) instead of separate segment and category dummies. Create day59/interactions.py.

# day59/interactions.py   (python -m day59.interactions)
import numpy as np
import pandas as pd
from econml.dml import LinearDML

from causalkit.demand import demand, public, truth
from causalkit.learners import reg
from day58.dml_elasticity import prepare

full = demand()
df = prepare(public(full).to_pandas())
cell = df.segment + " / " + df.category
X = pd.get_dummies(cell, drop_first=True).astype(float)
W = df[["promo", "temperature", "holiday", "local_event", "competitor_index", "week", "cat", "seg"]]
learner = reg(n_estimators=500, num_leaves=31, min_child_samples=20)
est = LinearDML(model_y=learner, model_t=learner, cv=5, random_state=0).fit(df.y, df.d, X=X, W=W, groups=df.store)
cells = truth(full).to_pandas()
Xc = pd.get_dummies(cells.segment + " / " + cells.category).astype(float).reindex(columns=X.columns, fill_value=0)
print(f"LinearDML with a separate elasticity per cell: mean absolute error "
      f"{np.mean(np.abs(est.effect(Xc) - cells.elasticity)):.2f}")

LinearDML with a separate elasticity per cell: mean absolute error 0.06

59.5 Day 59 mini-project: two estimates per cell (65 min)

DML assumes the pricing team only reacted to recorded drivers. The cost instrument assumes supplier costs affect sales only through price. Different assumptions, so if both estimates agree in a cell, the answer doesn't rest on one assumption alone. Create day59/iv_crosscheck.py:

# day59/iv_crosscheck.py   (python -m day59.iv_crosscheck)
"""
Mini-project: two estimates per cell that rest on DIFFERENT assumptions.
  DML (Day 59):  the pricing team only reacted to recorded demand drivers (selection on observables).
  2SLS (Day 58): supplier costs affect units only through price (exclusion).
If they agree, neither assumption is doing all the work alone. Run day59.segment_elasticities first.
"""
import numpy as np
import polars as pl
import pyfixest as pf

from causalkit.demand import demand, public

full = demand()
df = public(full).to_pandas()
dml_cells = pl.read_csv("data/day59_cell_elasticities.csv")
rows = []
for r in dml_cells.iter_rows(named=True):
    s = df[(df.segment == r["segment"]) & (df.category == r["category"])]
    iv = pf.feols("log_units ~ promo | store_id + week | log_price ~ cost_index", s, vcov={"CRV1": "store_id"})
    rows.append({"segment": r["segment"], "category": r["category"], "DML": r["DML"],
                 "2SLS": round(iv.coef()["log_price"], 2), "2SLS SE": round(iv.se()["log_price"], 2),
                 "truth": r["truth"]})
out = pl.DataFrame(rows)
with pl.Config(tbl_rows=20):
    print(out)
iv_err = out["2SLS"] - out["truth"]
dml_err = out["DML"] - out["truth"]
print(f"Average error: DML {dml_err.mean():+.3f}, 2SLS {iv_err.mean():+.3f}")
print(f"Typical size of error: DML {np.sqrt((dml_err**2).mean()):.3f}, 2SLS {np.sqrt((iv_err**2).mean()):.3f}")
print(f"2SLS 95% CIs containing the truth: {(iv_err.abs() <= 1.96 * out['2SLS SE']).sum()} of 20")

python -m day59.iv_crosscheck

│ segment         ┆ category      ┆ DML   ┆ 2SLS  ┆ 2SLS SE ┆ truth │
│ rural value     ┆ fresh         ┆ -3.07 ┆ -3.32 ┆ 0.13    ┆ -3.2  │
│ student         ┆ snacks        ┆ -4.36 ┆ -4.86 ┆ 0.23    ┆ -4.5  │
│ suburban family ┆ personal care ┆ -2.03 ┆ -2.51 ┆ 0.19    ┆ -2.08 │
...
Average error: DML +0.077, 2SLS -0.036
Typical size of error: DML 0.086, 2SLS 0.167
2SLS 95% CIs containing the truth: 19 of 20

The two estimators fail in opposite ways:

DML is usually closer; 2SLS is honest about its uncertainty. Day 60 uses both: DML as the point estimate, the 2SLS interval as the range of plausible values.

59.6 Save your work with Git (20 min)

git add .
git commit -m "Day 59: segment and cell elasticities, LinearDML, IV cross-check"
git push

Day 59 checklist

• I can estimate elasticities by segment and by segment × category cell with DML
• I know that DML's confidence intervals don't include the small bias left by its nuisance models
• I can fit LinearDML with segment and category features, and read its coefficients
• I can explain why an additive CATE model misses multiplicative heterogeneity
• I cross-check cell estimates with an estimator that rests on a different assumption