Policy Learning

65.1 From effects to decisions (40 min)

A policy says, for each customer, coupon or no coupon. Its value is what it earns: each couponed customer adds margin × (their effect) − cost. With a 30% margin and a $5 coupon:

value of couponing customer i = 0.30 × τᵢ − 5      →  coupon if τᵢ > $16.67

Two ways to get a policy:

1. Plug-in: estimate τ(x) (Days 55–57, 64), coupon if τ̂(x) > $16.67.
2. Policy learning (Athey & Wager, 2021): search directly for the rule, among simple ones, that maximises the estimated value. You don't need τ(x) to be accurate everywhere, only to know which side of $16.67 each customer is on.

Both are evaluated the same way: with doubly robust scores (Day 36), which give an unbiased value for any policy from one experiment (or from observational data, with the usual assumptions). This is off-policy evaluation: judging a policy nobody has run yet.

Create causalkit/policy.py:

"""Policy evaluation helpers (Days 65-70). A policy says, for each customer, coupon (1) or not (0).

Net value of a coupon for one customer: MARGIN x (extra spend) - COST.
"""
import re

import numpy as np

MARGIN, COST = 0.30, 5.0
BREAK_EVEN = COST / MARGIN            # a coupon pays if it raises spend by more than $16.67


def net(spend, coupon, margin=MARGIN, cost=COST):
    """Each customer's net outcome: margin on what they spent, minus the cost if they got a coupon."""
    return margin * np.asarray(spend) - cost * np.asarray(coupon)


def dr_scores(y, t, p, mu0, mu1):
    """Doubly robust (AIPW) scores for E[y(1)] and E[y(0)], one per customer."""
    y, t = np.asarray(y, float), np.asarray(t, float)
    s1 = mu1 + t * (y - mu1) / p
    s0 = mu0 + (1 - t) * (y - mu0) / (1 - p)
    return s1, s0


def policy_value(policy, s1, s0):
    """Average net value per customer of a policy, relative to coupons for nobody."""
    return float(np.mean(np.asarray(policy) * (s1 - s0)))


def tree_rules(graphviz: str) -> list[str]:
    """Turn EconML's export_graphviz text into one readable rule per leaf."""
    labels = dict(re.findall(r'^(\d+) \[label="([^"]+)"', graphviz, flags=re.M))
    edges = re.findall(r"^(\d+) -> (\d+)", graphviz, flags=re.M)
    children = {}
    for a, b in edges:
        children.setdefault(a, []).append(b)
    rules = []

    def walk(node, conds):
        lab = labels[node].split("\\n")
        if node not in children:
            treat = [x for x in lab if x.startswith("Treatment")][0].split("= ")[1]
            samples = [x for x in lab if x.startswith("samples")][0].split("= ")[1]
            rules.append(f"{' and '.join(conds) or 'everyone'}  ->  {'coupon' if treat != 'None' else 'no coupon'}"
                         f"  ({samples} training customers)")
            return
        cond = lab[0]
        left, right = children[node]
        walk(left, conds + [cond])
        walk(right, conds + [cond.replace("<=", ">")])

    walk("0", [])
    return rules

65.2 Policy trees (50 min)

EconML's DRPolicyTree grows a shallow tree whose leaves are decisions, choosing splits to maximise the doubly robust value of the net outcome (0.30 × spend − $5 if couponed). Create day65/policy_tree.py:

# day65/policy_tree.py   (python -m day65.policy_tree)
"""
Policy learning (Athey & Wager, 2021): instead of estimating effects and then thresholding them, search
directly for the simple rule that maximises the net value. EconML's DRPolicyTree uses doubly robust scores,
so it works on observational data too. Net outcome = 0.30 x spend - $5 if couponed.
"""
from econml.policy import DRPolicyTree

from causalkit.coupons import X_COLS
from causalkit.learners import clf, reg
from causalkit.policy import net, tree_rules
from day64.uplift_models import experiment

if __name__ == "__main__":
    train, test = experiment()
    for depth in [1, 2]:
        tree = DRPolicyTree(model_regression=reg(), model_propensity=clf(), max_depth=depth, min_samples_leaf=500,
                            random_state=0).fit(net(train.spend, train.coupon), train.coupon, X=train[X_COLS])
        print(f"Policy tree, depth {depth}:")
        for rule in tree_rules(tree.export_graphviz(feature_names=X_COLS)):
            print("   " + rule)

python -m day65.policy_tree

Policy tree, depth 1:
   deal_share <= 0.322  ->  no coupon  (6290 training customers)
   deal_share > 0.322  ->  coupon  (3710 training customers)
Policy tree, depth 2:
   deal_share <= 0.322 and z2 <= -1.411  ->  no coupon  (502 training customers)
   deal_share <= 0.322 and z2 > -1.411  ->  no coupon  (5788 training customers)
   deal_share > 0.322 and region_west <= 0.5  ->  coupon  (2986 training customers)
   deal_share > 0.322 and region_west > 0.5  ->  coupon  (724 training customers)

One rule: coupon customers who buy more than 32% of their items on promotion. The depth-2 tree adds splits on a noise column (z2) and on region, but both sides of each split get the same decision, so they don't change the policy. A rule a store manager can apply, and audit, from one number.

65.3 Which policy makes the most money? (55 min)

Create day65/policy_value.py. Its outcome_models function is reused through Day 70:

# day65/policy_value.py   (python -m day65.policy_value)
"""
Which policy makes the most money? Evaluate five policies on the held-out half of the experiment with
doubly robust scores (in an experiment the propensity is known: 0.5), with bootstrap CIs. Value is net $
per customer in the base, relative to sending nobody a coupon.
"""
import numpy as np
import polars as pl
from econml.dr import DRLearner
from econml.policy import DRPolicyTree

from causalkit.coupons import X_COLS
from causalkit.learners import clf, reg
from causalkit.policy import BREAK_EVEN, MARGIN, COST, dr_scores, net, policy_value
from day64.uplift_models import experiment


def policies(train, test):
    """Fit the candidate policies on train; return each one's decisions for the test customers."""
    dr = DRLearner(model_propensity=clf(), model_regression=reg(),
                   model_final=reg(n_estimators=100, num_leaves=4, min_child_samples=300), cv=5,
                   min_propensity=0.02, random_state=0).fit(train.spend, train.coupon, X=train[X_COLS])
    tree = DRPolicyTree(model_regression=reg(), model_propensity=clf(), max_depth=2, min_samples_leaf=500,
                        random_state=0).fit(net(train.spend, train.coupon), train.coupon, X=train[X_COLS])
    n = len(test)
    return {
        "coupons for nobody": np.zeros(n),
        "coupons for everyone": np.ones(n),
        "policy tree (depth 2)": tree.predict(test[X_COLS]),
        "DR-learner: effect > $16.67": (dr.effect(test[X_COLS]) > BREAK_EVEN).astype(int),
        "oracle (true effects)": (test._tau.to_numpy() > BREAK_EVEN).astype(int),
    }


def outcome_models(test, cost=COST):
    """Cross-fitted models of the net outcome for the evaluation half (for doubly robust scores)."""
    from sklearn.model_selection import KFold
    mu0, mu1 = np.zeros(len(test)), np.zeros(len(test))
    y = net(test.spend, test.coupon, cost=cost)
    for tr, te in KFold(5, shuffle=True, random_state=0).split(test):
        a = test.iloc[tr]
        for t, mu in [(0, mu0), (1, mu1)]:
            mu[te] = reg().fit(a[a.coupon == t][X_COLS], y[tr][a.coupon.to_numpy() == t]).predict(test.iloc[te][X_COLS])
    return y, mu0, mu1


if __name__ == "__main__":
    train, test = experiment()
    pols = policies(train, test)
    y, mu0, mu1 = outcome_models(test)
    s1, s0 = dr_scores(y, test.coupon, 0.5, mu0, mu1)
    true_gain = MARGIN * test._tau.to_numpy() - COST
    rng = np.random.default_rng(0)
    boot = [rng.integers(0, len(test), len(test)) for _ in range(500)]
    rows = []
    for name, pol in pols.items():
        v = policy_value(pol, s1, s0)
        b = [policy_value(pol[i], s1[i], s0[i]) for i in boot]
        rows.append({"policy": name, "share couponed": f"{np.mean(pol):.0%}", "net $ per customer": round(v, 2),
                     "95% CI": f"[{np.percentile(b, 2.5):.2f}, {np.percentile(b, 97.5):.2f}]",
                     "true net $ per customer": round(float(np.mean(pol * true_gain)), 2)})
    with pl.Config(tbl_width_chars=130, fmt_str_lengths=40):
        print(pl.DataFrame(rows))

python -m day65.policy_value

│ policy                      ┆ share couponed ┆ net $ per customer ┆ 95% CI        ┆ true net $ per customer │
│ coupons for nobody          ┆ 0%             ┆ 0.0                ┆ [0.00, 0.00]  ┆ 0.0                     │
│ coupons for everyone        ┆ 100%           ┆ 0.18               ┆ [-0.10, 0.46] ┆ -0.01                   │
│ policy tree (depth 2)       ┆ 38%            ┆ 1.58               ┆ [1.40, 1.75]  ┆ 1.33                    │
│ DR-learner: effect > $16.67 ┆ 48%            ┆ 1.84               ┆ [1.63, 2.05]  ┆ 1.61                    │
│ oracle (true effects)       ┆ 48%            ┆ 1.9                ┆ [1.68, 2.10]  ┆ 1.65                    │

• Coupons for everyone break even (true −$0.01 per customer): the average effect is just below $16.67, so the gains on persuadables are cancelled by the losses on everyone else.
• Targeting earns $1.3–1.6 per customer per campaign in truth. Over NorthMart's 400,000 customers, that's about $600,000 a month against zero for blanket coupons.
• The plug-in DR-learner policy beats the policy tree here and comes within 3% of the oracle: with an accurate CATE model, thresholding it is hard to beat. The tree gives up $0.28 per customer for a one-line rule. Whether that's worth it is a business question (can the CRM system apply a model score? does the team need to explain the rule?).
• The DR estimates are about $0.25 above the truth for every targeted policy, consistently: the experiment's evaluation half happened to be a little favourable. That's sampling noise, inside the CIs, and it shifts all the policies together; the comparisons between them are much more precise than the levels.

65.4 How deep should the tree be? (40 min)

Try it. Fit policy trees of depth 1 to 4 (with smaller minimum leaves for deeper trees) and evaluate each on the experiment. Create day65/deeper_tree.py.

# day65/deeper_tree.py   (python -m day65.deeper_tree)
from econml.policy import DRPolicyTree

from causalkit.coupons import X_COLS
from causalkit.learners import clf, reg
from causalkit.policy import dr_scores, net, policy_value
from day64.uplift_models import experiment
from day65.policy_value import outcome_models

train, test = experiment()
y, mu0, mu1 = outcome_models(test)
s1, s0 = dr_scores(y, test.coupon, 0.5, mu0, mu1)
for depth, leaf in [(1, 500), (2, 500), (3, 200), (4, 50)]:
    tree = DRPolicyTree(model_regression=reg(), model_propensity=clf(), max_depth=depth, min_samples_leaf=leaf,
                        random_state=0).fit(net(train.spend, train.coupon), train.coupon, X=train[X_COLS])
    print(f"depth {depth}, min leaf {leaf:>3}: net $ per customer {policy_value(tree.predict(test[X_COLS]), s1, s0):.2f}")

depth 1, min leaf 500: net $ per customer 1.58
depth 2, min leaf 500: net $ per customer 1.58
depth 3, min leaf 200: net $ per customer 1.58
depth 4, min leaf  50: net $ per customer 1.56

65.5 Day 65 mini-project: when coupons cost more (65 min)

The best policy depends on the cost: the threshold is cost / margin. Create day65/cost_sensitivity.py:

# day65/cost_sensitivity.py   (python -m day65.cost_sensitivity)
"""
Mini-project: the policy depends on the coupon's cost. For costs from $2 to $10, threshold the DR-learner's
effects at cost / margin and evaluate on the experiment, against coupons for everyone.
"""
import numpy as np
import polars as pl
from econml.dr import DRLearner

from causalkit.coupons import X_COLS
from causalkit.learners import clf, reg
from causalkit.policy import MARGIN, dr_scores, policy_value
from day64.uplift_models import experiment
from day65.policy_value import outcome_models

train, test = experiment()
dr = DRLearner(model_propensity=clf(), model_regression=reg(),
               model_final=reg(n_estimators=100, num_leaves=4, min_child_samples=300), cv=5,
               min_propensity=0.02, random_state=0).fit(train.spend, train.coupon, X=train[X_COLS])
tau_hat = dr.effect(test[X_COLS])
rows = []
for cost in [2, 4, 5, 6, 8, 10]:
    y, mu0, mu1 = outcome_models(test, cost=cost)
    s1, s0 = dr_scores(y, test.coupon, 0.5, mu0, mu1)
    pol = (tau_hat > cost / MARGIN).astype(int)
    rows.append({"coupon cost": cost, "break-even effect": round(cost / MARGIN, 1), "share couponed": f"{pol.mean():.0%}",
                 "targeted: net $ per customer": round(policy_value(pol, s1, s0), 2),
                 "everyone: net $ per customer": round(policy_value(np.ones(len(test)), s1, s0), 2)})
with pl.Config(tbl_width_chars=130):
    print(pl.DataFrame(rows))

python -m day65.cost_sensitivity

│ coupon cost ┆ break-even effect ┆ share couponed ┆ targeted: net $ per customer ┆ everyone: net $ per customer │
│ 2           ┆ 6.7               ┆ 76%            ┆ 3.66                         ┆ 3.18                         │
│ 4           ┆ 13.3              ┆ 58%            ┆ 2.38                         ┆ 1.18                         │
│ 5           ┆ 16.7              ┆ 48%            ┆ 1.84                         ┆ 0.18                         │
│ 6           ┆ 20.0              ┆ 40%            ┆ 1.44                         ┆ -0.82                        │
│ 8           ┆ 26.7              ┆ 24%            ┆ 0.73                         ┆ -2.82                        │
│ 10          ┆ 33.3              ┆ 10%            ┆ 0.28                         ┆ -4.82                        │

With cheap coupons ($2), targeting barely beats blanket coupons (most customers are worth couponing). With expensive ones ($10), blanket coupons lose $4.82 per customer while targeting still earns a little from the top 10%. The value of a targeting model rises with the cost of the treatment. That's the business case for building one, in a single table.

65.6 Save your work with Git (20 min)

git add .
git commit -m "Day 65: policy value, off-policy evaluation, policy trees, cost sensitivity"
git push

Day 65 checklist

• I can define a policy's value with margins and costs, and its break-even effect
• I can evaluate any policy with doubly robust scores and bootstrap CIs
• I can learn a policy tree with EconML and read its rules
• I can compare plug-in and policy-learning approaches, and explain the trade-off
• I can show how the optimal policy changes with costs