Capstone: Defending the DAG

20.1 The brief and how it is assessed (30 min)

The brief. NorthMart's pricing lead writes:

"For the last month our pricing algorithm has offered personalised discounts to about a third of customers. Finance looked at the numbers: discounted customers spent about the same as everyone else, so they want to switch the programme off. I think the comparison is unfair. Before we decide, I need someone to tell me (a) what the discount actually did to spending, (b) which comparison is the fair one, and (c) why I should trust your answer over Finance's. Two pages, please."

Deliverables.

Assessment rubric.

20.2 Read the scenario and the data (45 min)

Generate the data (the script is provided; don't read its code before you've drawn your DAG, it contains the answer):

python data/make_pricing_scenario.py

Wrote 6,000 customers to data/scenario/pricing.csv

What we know about the business (from interviews with the pricing, CRM and support teams):

1. The pricing algorithm decides who gets a discount. Its documentation lists three inputs: the customer's cart abandonment rate, their loyalty status (members get fewer discounts: "they buy anyway"), and whether they live in a high-traffic region.
2. Cart abandonment reflects how price-sensitive a customer is. Price-sensitive customers also spend less in general. Price sensitivity itself is not measured anywhere.
3. Loyalty membership is more common among long-tenured customers. Members spend more.
4. A separate spring email was sent mostly to loyalty members. Emails raise spending.
5. High-traffic regions have higher spending (more stores nearby, faster delivery).
6. A discount is delivered as a coupon; only customers offered a discount can use one. The pricing team believes the discount works by getting people to redeem the coupon.
7. Customer support contacts happen after purchases. Support sees more contacts from discounted customers (coupon questions) and from big spenders.

The data (data/scenario/pricing.csv), one row per customer, 30-day window:

Before any code: write the estimand card (Day 12 template) for the pricing lead's question (a). Then sketch the DAG on paper.

### Decision: keep or stop the personalised discount programme
- **Unit:** customer
- **Treatment vs comparison:** offered a personalised discount vs not offered one
- **Outcome and horizon:** spending in the 30-day window, in dollars
- **Population:** customers the algorithm chose (ATT), since the decision is about this programme
- **Summary measure:** average difference in dollars per customer
- **Note:** the effect in the simulation is the same for everyone, so ATT = ATE here;
  in general, keep them apart.

20.3 Draw the DAG and find the adjustment set (45 min)

Translate each numbered fact from 20.2 into arrows. Each arrow needs a comment pointing to its fact; each missing arrow is a claim too (for example: does the email affect who gets a discount? The documentation says the algorithm has three inputs, so no).

Create day20/my_dag.py. This model answer follows the brief point by point:

# day20/my_dag.py   (python -m day20.my_dag)
"""Capstone step 1: the DAG, written from the brief. One comment per arrow = one assumption."""
from causalkit.dags import (backdoor_paths, describe_path, draw_dag, make_dag,
                            minimal_adjustment_sets)

T, Y = "discount", "spend_30d"

dag = make_dag([
    ("tenure_years", "loyalty"),            # longer-tenured customers join the programme
    ("tenure_years", "spend_30d"),          # ...and spend more
    ("loyalty", "email"),                   # the spring email targeted loyalty members
    ("loyalty", "discount"),                # the pricing algorithm uses loyalty (documented)
    ("loyalty", "spend_30d"),
    ("high_traffic_region", "discount"),    # ...and regional traffic (documented)
    ("high_traffic_region", "spend_30d"),
    ("price_sensitivity", "cart_abandon_rate"),   # sensitive shoppers abandon carts
    ("price_sensitivity", "spend_30d"),           # ...and spend less
    ("cart_abandon_rate", "discount"),      # ...and the algorithm targets abandoners (documented)
    ("email", "spend_30d"),
    ("discount", "coupon_used"),            # a coupon exists only if a discount was offered
    ("coupon_used", "spend_30d"),           # the discount works through the coupon
    ("discount", "support_contact"),        # discounted customers contact support more
    ("spend_30d", "support_contact"),       # ...and so do big spenders (contact happens AFTER buying)
], latent={"price_sensitivity"})

if __name__ == "__main__":
    print("Backdoor paths from discount to spend:")
    for p in backdoor_paths(dag, T, Y):
        print("  ", describe_path(dag, p))
    print("\nMinimal adjustment sets:", minimal_adjustment_sets(dag, T, Y))
    draw_dag(dag, "figures/day20_pricing_dag.png", "Discount programme: assumed DAG", treatment=T, outcome=Y)
    print("Saved figures/day20_pricing_dag.png")

python -m day20.my_dag

Backdoor paths from discount to spend:
   discount <- loyalty <- tenure_years -> spend_30d
   discount <- loyalty -> email -> spend_30d
   discount <- loyalty -> spend_30d
   discount <- high_traffic_region -> spend_30d
   discount <- cart_abandon_rate <- price_sensitivity -> spend_30d

Minimal adjustment sets: [{'cart_abandon_rate', 'high_traffic_region', 'loyalty'}]
Saved figures/day20_pricing_dag.png

Every backdoor path goes through one of the algorithm's three documented inputs. That's Day 15's lesson in action: because the assignment rule is known and its inputs are recorded, the effect is identifiable even though price sensitivity is latent. Compare with NorthMart's email, where marketing's judgement of "heavy shoppers" was never recorded.

Note what's not in the set: coupon_used (a mediator), support_contact (a collider and descendant of the treatment), email and tenure_years (blocked through loyalty; allowed but not needed).

20.4 Confront the DAG with the data (35 min)

Create day20/test_my_dag.py:

# day20/test_my_dag.py   (python -m day20.test_my_dag)
"""Capstone step 2: confront the DAG with the data. Every rejected claim is an arrow to rethink."""
import polars as pl
import statsmodels.formula.api as smf

from causalkit.dags import implied_independencies
from day20.my_dag import dag

df = pl.read_csv("data/scenario/pricing.csv").to_pandas()

claims = implied_independencies(dag, max_conditioning=2)
print(f"The DAG implies {len(claims)} testable independences. Testing each (alpha = 0.01):\n")
rejected = 0
for a, b, z in claims:
    fit = smf.ols(f"{b} ~ {a}" + "".join(f" + {v}" for v in sorted(z)), df).fit(cov_type="HC2")
    p = fit.pvalues[a]
    rejected += p < 0.01
    claim = f"{a} _||_ {b}" + (f" | {', '.join(sorted(z))}" if z else "")
    print(f"  {claim:<70} p = {p:.3f} {'REJECTED' if p < 0.01 else ''}")
print(f"\n{rejected} of {len(claims)} claims rejected.")

python -m day20.test_my_dag

The DAG implies 21 testable independences. Testing each (alpha = 0.01):

  cart_abandon_rate _||_ coupon_used | discount                          p = 0.979
  cart_abandon_rate _||_ email                                           p = 0.790
  ...
  discount _||_ email | loyalty                                          p = 0.194
  discount _||_ tenure_years | loyalty                                   p = 0.270
  ...
  support_contact _||_ tenure_years | discount, spend_30d                p = 0.691

0 of 21 claims rejected.

Two of these deserve a sentence each in your memo:

• discount _||_ email | loyalty: the data agree that the algorithm doesn't use the email (once loyalty is accounted for). That supports leaving email out of the adjustment set.
• discount _||_ tenure_years | loyalty: tenure matters for discounts only through loyalty, as the documentation implies.

"Not rejected" means the data don't contradict the DAG. It doesn't prove it: a DAG with extra arrows would pass too, and the latent confounder's arrows can't be tested at all.

Try it. A colleague insists "the algorithm ignores loyalty". Remove the arrow loyalty → discount and test only the new claims that version makes. Is the colleague's DAG consistent with the data?

# day20/colleague_dag.py   (python -m day20.colleague_dag)
import polars as pl
import statsmodels.formula.api as smf

from causalkit.dags import implied_independencies
from day20.my_dag import dag

colleague = dag.copy()
colleague.remove_edge("loyalty", "discount")      # "the algorithm doesn't look at loyalty"

df = pl.read_csv("data/scenario/pricing.csv").to_pandas()
new_claims = set(implied_independencies(colleague, 2)) - set(implied_independencies(dag, 2))
for a, b, z in sorted(new_claims):
    fit = smf.ols(f"{b} ~ {a}" + "".join(f" + {v}" for v in sorted(z)), df).fit(cov_type="HC2")
    claim = f"{a} _||_ {b}" + (f" | {', '.join(sorted(z))}" if z else "")
    print(f"{claim:<55} p = {fit.pvalues[a]:.4f}")

coupon_used _||_ email                                  p = 0.3385
coupon_used _||_ loyalty                                p = 0.0000
coupon_used _||_ tenure_years                           p = 0.0000
discount _||_ email                                     p = 0.0051
discount _||_ loyalty                                   p = 0.0000
discount _||_ tenure_years                              p = 0.0000

20.5 Estimate, and show what the alternatives do (40 min)

Create day20/estimate.py:

# day20/estimate.py   (python -m day20.estimate)
"""Capstone step 3: estimate the effect with the DAG's adjustment set, and show what the alternatives do."""
import polars as pl
import statsmodels.formula.api as smf

df = pl.read_csv("data/scenario/pricing.csv").to_pandas()
VALID = "cart_abandon_rate + loyalty + high_traffic_region"

specs = {
    "naive (no controls)": "spend_30d ~ discount",
    "DAG adjustment set": f"spend_30d ~ discount + {VALID}",
    "DAG set + precision controls": f"spend_30d ~ discount + {VALID} + email + tenure_years",
    "forgot cart_abandon_rate": "spend_30d ~ discount + loyalty + high_traffic_region",
    "+ coupon_used (mediator)": f"spend_30d ~ discount + {VALID} + coupon_used",
    "+ support_contact (collider)": f"spend_30d ~ discount + {VALID} + support_contact",
}
rows = []
for name, f in specs.items():
    fit = smf.ols(f, df).fit(cov_type="HC2")
    lo, hi = fit.conf_int().loc["discount"]
    rows.append({"specification": name, "effect ($)": round(fit.params["discount"], 2),
                 "SE": round(fit.bse["discount"], 2), "95% CI": f"[{lo:.1f}, {hi:.1f}]"})
with pl.Config(fmt_str_lengths=40, tbl_width_chars=110):
    print(pl.DataFrame(rows))

python -m day20.estimate

│ specification                ┆ effect ($) ┆ SE   ┆ 95% CI       │
│ naive (no controls)          ┆ -0.36      ┆ 0.72 ┆ [-1.8, 1.1]  │
│ DAG adjustment set           ┆ 12.83      ┆ 0.73 ┆ [11.4, 14.3] │
│ DAG set + precision controls ┆ 12.84      ┆ 0.69 ┆ [11.5, 14.2] │
│ forgot cart_abandon_rate     ┆ 2.37       ┆ 0.67 ┆ [1.1, 3.7]   │
│ + coupon_used (mediator)     ┆ 0.69       ┆ 0.97 ┆ [-1.2, 2.6]  │
│ + support_contact (collider) ┆ 9.31       ┆ 0.71 ┆ [7.9, 10.7]  │

Every row is a plausible analysis someone might present:

• Naive (Finance's number): −$0.36, "discounts do nothing". The algorithm sends discounts to price-sensitive, non-member customers who spend less anyway.
• The DAG's set: +$12.83 [11.4, 14.3]. Adding email and tenure_years (outcome predictors, allowed by the DAG) narrows the interval slightly without moving the estimate.
• Forgetting one input: +$2.37. Omitting cart abandonment leaves the price-sensitivity path open.
• Controlling for the coupon: +$0.69. The coupon is the mechanism; controlling for it removes the effect.
• Controlling for support contacts: +$9.31. A collider: biased, and confidently so.

Extension: confirm with DoWhy. Hand the same DAG to DoWhy (Day 18) and check that it finds the same adjustment set and estimate.

# day18/dowhy_pricing.py   (python -m day18.dowhy_pricing)
import warnings

import polars as pl
from dowhy import CausalModel

from day20.my_dag import dag

warnings.filterwarnings("ignore")
df = pl.read_csv("data/scenario/pricing.csv").drop("customer_id").to_pandas()
m = CausalModel(data=df, treatment="discount", outcome="spend_30d", graph=dag)
estimand = m.identify_effect()
print("Backdoor variables:", sorted(estimand.get_backdoor_variables()))
est = m.estimate_effect(estimand, method_name="backdoor.linear_regression")
print(f"Effect: {est.value:+.2f}")

Backdoor variables: ['cart_abandon_rate', 'high_traffic_region', 'loyalty']
Effect: +12.84

20.6 Write the memo (55 min)

Write docs/DAG_MEMO.md for the pricing lead: two pages at most, plain language, numbers rounded. Use these headings.

1. The answer. One paragraph: the effect, with its interval, and the recommendation.
2. Why Finance's comparison misleads. Who the algorithm targets, and why those customers would spend less anyway.
3. The fair comparison. The DAG (insert figures/day20_pricing_dag.png) and, in words, the three variables we compare like-for-like on, and why.
4. What we deliberately did not control for. Coupon use and support contacts, each in one sentence.
5. How we checked. The 21 tests, and what passing them means and doesn't mean.
6. What could still be wrong. At least two specific threats (for example: an undocumented input to the algorithm; spillovers between household members; a 30-day window that may miss pulled-forward purchases), with what would detect each.

20.7 Answer key, final commit and Module 2 retrospective (20 min)

Once your memo is written, compare your DAG with the process that generated the data. Create day20/answer_key.py:

# day20/answer_key.py   (python -m day20.answer_key)
"""Instructor check: compare a student's DAG and adjustment set with the true data-generating process."""
import json
from pathlib import Path

from causalkit.dags import is_valid_adjustment, make_dag, minimal_adjustment_sets
from day20.my_dag import dag as student_dag

truth = json.loads(Path("data/truth/pricing_truth.json").read_text())
true_dag = make_dag([tuple(e) for e in truth["edges"]], latent=set(truth["latent"]))
T, Y = "discount", "spend_30d"

missing = set(true_dag.edges) - set(student_dag.edges)
extra = set(student_dag.edges) - set(true_dag.edges)
print(f"True effect: +${truth['true_effect_of_discount_on_spend']:.0f} per customer over 30 days")
print("Arrows missing from the student DAG:", sorted(missing) or "none")
print("Arrows the true process does not have:", sorted(extra) or "none")
for z in minimal_adjustment_sets(student_dag, T, Y):
    print(f"Student's set {sorted(z)} valid in the TRUE DAG? {is_valid_adjustment(true_dag, T, Y, z)}")

python -m day20.answer_key

True effect: +$12 per customer over 30 days
Arrows missing from the student DAG: none
Arrows the true process does not have: none
Student's set ['cart_abandon_rate', 'high_traffic_region', 'loyalty'] valid in the TRUE DAG? True

If your DAG differs, the last line is what matters most: an imperfect DAG can still give a valid adjustment set. Extra arrows that don't create new backdoor paths are harmless; a missing arrow into the treatment is not.

git add .
git commit -m "Day 20: Defending the DAG capstone"
git push

What you learned in Module 2.

Coming next: Module 3, Experimentation & A/B Testing. Day 13 showed what randomisation buys. Module 3 shows how to run experiments well: power analysis, stratification, CUPED variance reduction (the pre_orders trick from 13.4, done properly), peeking, and interference.

Day 20 checklist

• I translated a business narrative into a DAG with a justification for every arrow
• I derived and defended a minimal adjustment set, including what I left out and why
• I tested every implication of my DAG and can explain what "not rejected" means
• I estimated the effect and showed how four tempting alternatives mislead
• I wrote a memo a non-technical decision-maker can act on