Price Elasticity Estimation

58.1 Why prices are endogenous (40 min)

The price elasticity of demand is the % change in units sold for a 1% change in price. With elasticity −2.5, a 1% price rise loses 2.5% of units. In a regression of log units on log price, it's the slope.

The trouble: NorthMart's prices aren't random. The pricing team raises prices in weeks it expects to be busy (warm weather for snacks, holidays, local events, competitors charging more) and passes on supplier cost changes. So high prices and high demand show up together, and a naive regression mistakes "busy weeks get higher prices" for "higher prices don't hurt sales".

Create causalkit/demand.py (provided; read only the docstrings):

"""
NorthMart weekly demand (simulated). Provided for Module 6: read the docstrings, not the code,
until the Day 60 capstone is done.

80 stores in four shopper segments, five categories, 104 weeks. The pricing team sets each week's
category price from supplier costs AND from its own demand forecast: when it expects a busy week
(warm weather, a holiday, a local event, competitors charging more) it raises prices. That makes
prices and demand move together for reasons that have nothing to do with how customers respond to price.

    from causalkit.demand import demand, truth, SEGMENTS, CATEGORIES

    df = demand()                      # 41,600 store-category-weeks
    df = demand(hidden=1.0)            # the pricing team also reacts to signals that were never recorded
    truth(df)                          # true elasticity for each segment x category (for checking only)

Columns an analyst has:
    store_id, segment, category, week, price ($), units, log_price, log_units,
    cost_index (supplier cost, 1.0 = normal), promo (1 = on promotion),
    temperature (deg C), holiday (1 = holiday week), local_event (1 = event near the store),
    competitor_index (competitors' price level, 1.0 = normal), margin (gross margin share at the base price)
Columns starting with '_' are the simulator's hidden truth: _elasticity.
"""
import numpy as np
import polars as pl

SEGMENTS = ["urban premium", "suburban family", "student", "rural value"]
CATEGORIES = ["fresh", "pantry", "household", "personal care", "snacks"]
SEG_BASE = {"urban premium": -1.4, "suburban family": -2.6, "student": -3.6, "rural value": -3.2}
CAT_MULT = {"fresh": 1.0, "pantry": 0.9, "household": 1.1, "personal care": 0.8, "snacks": 1.25}
MARGIN = {"fresh": 0.28, "pantry": 0.22, "household": 0.30, "personal care": 0.35, "snacks": 0.30}
BASE_PRICE = {"fresh": 6.0, "pantry": 4.0, "household": 9.0, "personal care": 8.0, "snacks": 3.5}


def elasticity(segment: str, category: str) -> float:
    return round(SEG_BASE[segment] * CAT_MULT[category], 3)


def demand(n_stores: int = 80, weeks: int = 104, seed: int = 58, hidden: float = 0.0) -> pl.DataFrame:
    rng = np.random.default_rng(seed)
    seg = np.array(SEGMENTS)[np.arange(n_stores) % 4]
    size = rng.lognormal(0, 0.3, n_stores)
    t = np.arange(1, weeks + 1)
    temp_wk = 10 + 12 * np.sin(2 * np.pi * (t - 15) / 52) + rng.normal(0, 3, weeks)
    holiday_wk = np.isin(t % 52, [0, 1, 13, 26, 35, 47, 51]).astype(float)
    rows = []
    for c_i, cat in enumerate(CATEGORIES):
        cost_wk = np.exp(np.cumsum(rng.normal(0, 0.02, weeks)) - 0.0)      # supplier cost drifts
        comp_wk = np.exp(rng.normal(0, 0.04, weeks))
        for s in range(n_stores):
            e = elasticity(seg[s], cat)
            temp = temp_wk + rng.normal(0, 1.5, weeks)
            event = (rng.random(weeks) < 0.15).astype(float)
            promo = (rng.random(weeks) < 0.12).astype(float)
            cost = cost_wk * np.exp(rng.normal(0, 0.05, weeks))
            comp = comp_wk * np.exp(rng.normal(0, 0.06, weeks))
            unrecorded = rng.normal(0, 1, weeks)
            # demand drivers, nonlinear: what the pricing team forecasts from
            shifter = (0.25 * np.tanh((temp - 12) / 6) * (1 if cat in ("snacks", "fresh") else 0.3)
                       + 0.30 * holiday_wk + 0.40 * event + 1.5 * np.log(comp) * (1 + (seg[s] == "student"))
                       + 0.1 * np.sin(2 * np.pi * t / 52 + c_i))
            forecast = shifter + 0.15 * hidden * unrecorded
            log_p = (np.log(BASE_PRICE[cat]) + 0.7 * np.log(cost) + 0.6 * forecast + 0.15 * forecast**2
                     - 0.20 * promo + rng.normal(0, 0.12, weeks))
            log_q = (np.log(400 * size[s]) + shifter + 0.15 * hidden * unrecorded
                     + e * (log_p - np.log(BASE_PRICE[cat])) + 0.10 * promo + rng.normal(0, 0.12, weeks))
            units = rng.poisson(np.exp(log_q))
            rows.append(pl.DataFrame({
                "store_id": f"T{s + 1:02d}", "segment": seg[s], "category": cat, "week": t,
                "price": np.round(np.exp(log_p), 2), "units": units,
                "cost_index": np.round(cost, 4), "promo": promo.astype(np.int64),
                "temperature": np.round(temp, 1), "holiday": holiday_wk.astype(np.int64),
                "local_event": event.astype(np.int64), "competitor_index": np.round(comp, 4),
                "margin": MARGIN[cat], "_elasticity": e,
            }))
    df = pl.concat(rows).filter(pl.col("units") > 0)
    return df.with_columns(log_price=pl.col("price").log(), log_units=pl.col("units").log()).select(
        "store_id", "segment", "category", "week", "price", "units", "log_price", "log_units", "cost_index", "promo",
        "temperature", "holiday", "local_event", "competitor_index", "margin", "_elasticity")


def truth(df: pl.DataFrame) -> pl.DataFrame:
    return (df.group_by("segment", "category").agg(elasticity=pl.col("_elasticity").first(),
                                                    margin=pl.col("margin").first())
            .sort("segment", "category"))


def public(df: pl.DataFrame) -> pl.DataFrame:
    return df.select([c for c in df.columns if not c.startswith("_")])

The confounders here are the demand drivers: temperature, holidays, local events, competitors' prices. They're all recorded. Supplier cost is a second lever: it moves prices but not demand.

58.2 Regressions that get it wrong (50 min)

Create day58/naive_elasticity.py:

# day58/naive_elasticity.py   (python -m day58.naive_elasticity)
"""
Price elasticity: the % change in units sold for a 1% change in price. In a log-log regression of
units on price it's the slope. But NorthMart's pricing team raises prices in weeks it expects to be busy,
so high prices and high demand go together for reasons that have nothing to do with customers' response.
"""
import pyfixest as pf

from causalkit.demand import demand, public

full = demand()
df = public(full).to_pandas()
print(f"{len(df):,} store-category-weeks: {df.store_id.nunique()} stores, {df.category.nunique()} categories, "
      f"{df.week.nunique()} weeks")
print(df.head(3).to_string(index=False))

for name, fml in [("pooled, no controls", "log_units ~ log_price"),
                  ("store x category fixed effects", "log_units ~ log_price | store_id^category"),
                  ("+ week fixed effects", "log_units ~ log_price | store_id^category + week"),
                  ("+ linear demand controls", "log_units ~ log_price + promo + temperature + holiday + local_event"
                                               " + competitor_index | store_id^category + week")]:
    fit = pf.feols(fml, df, vcov={"CRV1": "store_id"})
    print(f"{name:<34} elasticity {fit.coef()['log_price']:+.2f}  (SE {fit.se()['log_price']:.2f})")
print(f"Truth: average elasticity {full['_elasticity'].mean():.2f} (from {full['_elasticity'].min():.2f} "
      f"to {full['_elasticity'].max():.2f} across segments and categories)")

python -m day58.naive_elasticity

41,600 store-category-weeks: 80 stores, 5 categories, 104 weeks
...
pooled, no controls                elasticity -0.38  (SE 0.03)
store x category fixed effects     elasticity -2.04  (SE 0.09)
+ week fixed effects               elasticity -2.14  (SE 0.09)
+ linear demand controls           elasticity -2.51  (SE 0.09)
Truth: average elasticity -2.73 (from -4.50 to -1.12 across segments and categories)

• The pooled regression is nonsense: it mostly compares expensive categories with cheap ones.
• Store × category fixed effects compare each store-category with itself over time: much better, but still 25% too small in magnitude, because busy weeks get higher prices.
• Week fixed effects remove what's common to every store (national holidays, national weather), and linear controls remove more. But the pricing rule is nonlinear (it reacts more than proportionally to very busy weeks) and differs by segment, so a linear adjustment leaves bias behind.

An elasticity of −2.1 instead of −2.7 is not a detail: on Day 60 it decides whether a price rise looks profitable.

58.3 Supplier costs as an instrument (45 min)

Concept. Supplier costs shift prices (relevance) and plausibly affect units sold only through the price (exclusion: customers don't see supplier costs). That's Day 37's instrument, now for a continuous treatment. Create day58/iv_elasticity.py:

# day58/iv_elasticity.py   (python -m day58.iv_elasticity)
"""
Supplier costs move prices but don't move customers' demand: a classic instrument (Day 37) for price.
"""
import pyfixest as pf

from causalkit.demand import demand, public

full = demand()
df = public(full).to_pandas()
first = pf.feols("log_price ~ cost_index + promo | store_id^category + week", df, vcov={"CRV1": "store_id"})
print(f"First stage: a 10% higher supplier cost raises the price by "
      f"{first.coef()['cost_index'] * 10:.1f}% (cost_index is 1.0 at normal cost); "
      f"t = {first.tstat()['cost_index']:.0f}")
iv = pf.feols("log_units ~ promo | store_id^category + week | log_price ~ cost_index", df, vcov={"CRV1": "store_id"})
print(f"2SLS elasticity: {iv.coef()['log_price']:+.2f}  (SE {iv.se()['log_price']:.2f})")
print(f"Truth: average elasticity {full['_elasticity'].mean():.2f}")

python -m day58.iv_elasticity

First stage: a 10% higher supplier cost raises the price by 6.2% (cost_index is 1.0 at normal cost); t = 75
2SLS elasticity: -2.82  (SE 0.10)
Truth: average elasticity -2.73

A strong first stage (t = 75) and an estimate within one standard error of the truth. The instrument doesn't need the demand drivers at all; its assumption is the exclusion restriction instead. (Could a supplier cost rise also signal a shortage that customers notice? That's the kind of argument Day 38 asked for.)

58.4 DML for elasticities (50 min)

Concept. The demand drivers are recorded, so selection on observables holds: DML's partially linear model applies with log price as a continuous treatment. Residualise log units and log price on the drivers with LightGBM, regress residual on residual. Two panel details:

• Remove each store × category's average first: a store's normal sales level isn't something the ML model should have to learn.
• Cluster standard errors by store (Day 41): weeks of the same store aren't independent. DoubleML does this with cluster_cols.

Create day58/dml_elasticity.py. Its prepare and dml_elasticity functions are reused on Days 59–60:

# day58/dml_elasticity.py   (python -m day58.dml_elasticity)
"""
The pricing team's rule depends on things that ARE recorded (temperature, holidays, local events,
competitors' prices), in nonlinear ways. That's selection on observables, so DML's partially linear
model applies: residualise log units and log price on the demand drivers with ML, then regress
residual on residual. Store x category averages are removed first (a store's normal sales level), and
standard errors are clustered by store: weeks of the same store are not independent (Day 41).
"""
import doubleml as dml
import numpy as np
from sklearn.linear_model import LassoCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

from causalkit.demand import demand, public
from causalkit.learners import reg

DRIVERS = ["promo", "temperature", "holiday", "local_event", "competitor_index", "week", "cat", "seg"]


def prepare(df):
    """Subtract each store x category's average log units and log price; code segment and category."""
    g = df.groupby(["store_id", "category"])
    return df.assign(y=df.log_units - g.log_units.transform("mean"), d=df.log_price - g.log_price.transform("mean"),
                     cat=df.category.astype("category").cat.codes, seg=df.segment.astype("category").cat.codes,
                     store=df.store_id.str[1:].astype(int))


def dml_elasticity(df, learner=None, drivers=DRIVERS):
    np.random.seed(58)                                   # DoubleML draws its folds from numpy's global RNG
    data = dml.DoubleMLData(df[drivers + ["y", "d", "store"]], "y", "d", drivers, cluster_cols="store")
    learner = learner or reg(n_estimators=500, num_leaves=31, min_child_samples=20)
    return dml.DoubleMLPLR(data, ml_l=learner, ml_m=learner, n_folds=5).fit()


if __name__ == "__main__":
    full = demand()
    df = prepare(public(full).to_pandas())
    for name, learner in [("lasso", make_pipeline(StandardScaler(), LassoCV(cv=3))), ("LightGBM", None)]:
        fit = dml_elasticity(df, learner)
        print(f"DML with {name:<8} elasticity {fit.coef[0]:+.2f}  (SE {fit.se[0]:.2f})")
    print(f"Truth: average elasticity {full['_elasticity'].mean():.2f}")

python -m day58.dml_elasticity

DML with lasso    elasticity -2.49  (SE 0.10)
DML with LightGBM elasticity -2.68  (SE 0.09)
Truth: average elasticity -2.73

With lasso (linear) nuisance models, DML is no better than the linear-controls regression: the nuisance models can't capture the pricing rule. With LightGBM, it's within one standard error of the truth. DML is only as good as its nuisance models.

58.5 Day 58 mini-project: when the drivers aren't recorded (65 min)

demand(hidden=1.0): the pricing team also reacts to signals it never wrote down (a store manager's call about a local road closure, say). Compare DML with 2SLS. Create day58/hidden_drivers.py.

# day58/hidden_drivers.py   (python -m day58.hidden_drivers)
import pyfixest as pf

from causalkit.demand import demand, public
from day58.dml_elasticity import dml_elasticity, prepare

full = demand(hidden=1.0)
df = public(full).to_pandas()
fit = dml_elasticity(prepare(df))
iv = pf.feols("log_units ~ promo | store_id^category + week | log_price ~ cost_index", df, vcov={"CRV1": "store_id"})
print(f"DML  {fit.coef[0]:+.2f} (SE {fit.se[0]:.2f})")
print(f"2SLS {iv.coef()['log_price']:+.2f} (SE {iv.se()['log_price']:.2f})")
print(f"Truth {full['_elasticity'].mean():.2f}")

DML  -2.13 (SE 0.09)
2SLS -2.64 (SE 0.09)
Truth -2.73

Ask in day58/NOTES.md: what would you need to know about NorthMart's pricing process to trust the DML estimate? Who would you ask?

58.6 Save your work with Git (20 min)

git add .
git commit -m "Day 58: demand simulator, endogenous prices, cost instrument, DML elasticity"
git push

Day 58 checklist

• I can explain why prices set by a pricing team are endogenous
• I can estimate an elasticity with fixed effects, and say what they do and don't remove
• I can estimate an elasticity with a cost instrument and check the first stage
• I can estimate an elasticity with DML, with store-clustered standard errors
• I can say which assumption each estimator rests on, and when each one fails