The Backdoor Criterion and Adjustment Sets

15.1 Backdoor paths: the routes bias travels (45 min)

Concept. Association between treatment T and outcome Y flows along every open path between them. Paths that start with an arrow out of T (T → … → Y) carry the causal effect. Paths that start with an arrow into T (T ← … Y) are backdoor paths: they carry association that isn't causation, which is exactly the selection bias of Day 11.

The backdoor criterion (Pearl). A set of variables Z is a valid adjustment set for the effect of T on Y if:

1. no variable in Z is a descendant of T, and
2. Z blocks every backdoor path from T to Y.

If such a Z exists and is observed, comparing treated and untreated units with the same Z (and averaging over Z) identifies the causal effect. The criterion turns "what should I control for?" from a matter of taste into a question with an answer, given the DAG.

Add the shared NorthMart DAG to causalkit/northmart_dag.py, so every later script uses the same one:

"""The NorthMart targeting DAG from Day 14, shared by later days."""
from causalkit.dags import make_dag

EDGES = [
    ("loyalty", "intensity"), ("loyalty", "email"), ("intensity", "email"),
    ("intensity", "pre_orders"), ("intensity", "post_orders"), ("email", "post_orders"),
]


def northmart_dag():
    return make_dag(EDGES, latent={"intensity"})

15.2 Which sets block NorthMart's backdoors? (45 min)

Create day15/backdoor_paths.py:

# day15/backdoor_paths.py   (python -m day15.backdoor_paths)
"""List the backdoor paths from email to post_orders and check which sets block them."""
from causalkit.dags import backdoor_paths, describe_path, is_valid_adjustment, minimal_adjustment_sets, path_is_blocked
from causalkit.northmart_dag import northmart_dag

g = northmart_dag()
paths = backdoor_paths(g, "email", "post_orders")
print("Backdoor paths:")
for p in paths:
    print("  ", describe_path(g, p))

print("\nWhich adjustment sets block them?")
for z in [set(), {"loyalty"}, {"pre_orders"}, {"loyalty", "pre_orders"}, {"intensity"}]:
    status = ["blocked" if path_is_blocked(g, p, z) else "OPEN" for p in paths]
    label = "{" + ", ".join(sorted(z)) + "}"
    print(f"  {label:<26} {status}  valid: {is_valid_adjustment(g, 'email', 'post_orders', z)}")

print("\nMinimal valid sets using OBSERVED variables only:", minimal_adjustment_sets(g, "email", "post_orders"))

python -m day15.backdoor_paths

Backdoor paths:
   email <- loyalty -> intensity -> post_orders
   email <- intensity -> post_orders

Which adjustment sets block them?
  {}                         ['OPEN', 'OPEN']  valid: False
  {loyalty}                  ['blocked', 'OPEN']  valid: False
  {pre_orders}               ['OPEN', 'OPEN']  valid: False
  {loyalty, pre_orders}      ['blocked', 'OPEN']  valid: False
  {intensity}                ['blocked', 'blocked']  valid: True

Minimal valid sets using OBSERVED variables only: []

This is the central result of the module for NorthMart:

• email ← intensity → post_orders can only be blocked by controlling for intensity, which nobody observes.
• pre_orders is not on that path: it hangs off intensity as a child. Controlling for it doesn't block the path in the DAG.
• No valid adjustment set exists among the observed variables. Under our DAG, no regression on NorthMart's columns, however many you add, identifies the email's effect.

That's not a failure of the DAG; it's what the DAG is for. It tells us to stop hoping that "more controls" will fix the problem, and to look for a different design (Module 5's difference-in-differences uses pre-period orders very differently: as a baseline for each customer's change).

15.3 What adjustment does to the estimate (60 min)

Concept. In practice, analysts adjust for proxies of the latent confounder anyway. What happens? The truth file contains each customer's expected orders without the email (mu0): that is their intensity. So we can compare imperfect adjustment with the impossible ideal.

Create day15/adjustment_estimates.py:

# day15/adjustment_estimates.py   (python -m day15.adjustment_estimates)
"""What adjustment does to the estimate: each set from the DAG, against the truth."""
import polars as pl
import statsmodels.formula.api as smf

from causalkit.data import load_customers

truth = pl.read_parquet("data/truth/potential_outcomes.parquet").select("customer_id", "mu0", "tau")
df = (load_customers().join(truth, on="customer_id")
      .with_columns(pl.col("email").cast(pl.Int8), loyalty=pl.col("is_loyalty").cast(pl.Int8))
      .to_pandas())

att = df.loc[df.email == 1, "tau"].mean()
ate = df["tau"].mean()

specs = {
    "nothing (naive)": "post_orders ~ email",
    "loyalty": "post_orders ~ email + loyalty",
    "loyalty + pre_orders": "post_orders ~ email + loyalty + pre_orders",
    "loyalty + pre_orders + pre_revenue + pre_active_weeks":
        "post_orders ~ email + loyalty + pre_orders + pre_revenue + pre_active_weeks",
    "intensity itself (impossible in real life)": "post_orders ~ email + mu0",
}
rows = []
for name, f in specs.items():
    fit = smf.ols(f, data=df).fit(cov_type="HC2")
    lo, hi = fit.conf_int().loc["email"]
    rows.append({"adjusting for": name, "estimate": round(fit.params["email"], 3),
                 "ci_low": round(lo, 3), "ci_high": round(hi, 3)})

with pl.Config(fmt_str_lengths=60, tbl_width_chars=120):
    print(pl.DataFrame(rows))
print(f"Truth: ATE {ate:.3f}, ATT {att:.3f} extra orders per customer (weeks 9-16)")

python -m day15.adjustment_estimates

│ adjusting for                                         ┆ estimate ┆ ci_low ┆ ci_high │
│ nothing (naive)                                       ┆ 1.711    ┆ 1.555  ┆ 1.866   │
│ loyalty                                               ┆ 1.523    ┆ 1.362  ┆ 1.684   │
│ loyalty + pre_orders                                  ┆ 1.062    ┆ 0.916  ┆ 1.209   │
│ loyalty + pre_orders + pre_revenue + pre_active_weeks ┆ 1.061    ┆ 0.915  ┆ 1.208   │
│ intensity itself (impossible in real life)            ┆ 0.35     ┆ 0.229  ┆ 0.472   │
Truth: ATE 0.319, ATT 0.443 extra orders per customer (weeks 9-16)

Read it down the column:

• Proxies help: from 1.71 to 1.06. But the estimate is still 2.4 times the ATT, with a confidence interval nowhere near the truth.
• Adding two more pre-period variables changes nothing: they measure the same thing (8 noisy weeks of the same intensity) no better than pre_orders does.
• Controlling for the confounder itself gets close to the truth (0.35, with an interval containing both ATE and ATT). The DAG was right about what's needed; the data just don't contain it.

The confidence interval doesn't warn you. [0.92, 1.21] looks precise and excludes the truth entirely. Standard errors measure noise, not bias. The DAG is how you learn about bias.

15.4 Record how treatment was assigned (35 min)

Concept. The cheapest fix for confounding happens before the campaign: record the information that drove the decision. If marketing had stored the targeting score it used, the DAG would change.

Try it. Write day15/targeting_score.py: a DAG where marketing computes a score from loyalty and intensity, and emails customers above a cut-off (so score is the email's only parent). Find the minimal adjustment sets.

# day15/targeting_score.py   (python -m day15.targeting_score)
from causalkit.dags import make_dag, minimal_adjustment_sets

g = make_dag([
    ("loyalty", "intensity"), ("loyalty", "score"), ("intensity", "score"),
    ("score", "email"),                       # marketing emailed customers ABOVE a score cut-off
    ("intensity", "pre_orders"), ("intensity", "post_orders"), ("email", "post_orders"),
], latent={"intensity"})
print(minimal_adjustment_sets(g, "email", "post_orders"))

[{'score'}]

15.5 Day 15 mini-project: four classic DAGs (70 min)

Goal. Build intuition on four small DAGs that appear again and again in business data. For each, predict the minimal adjustment set on paper first, then check with code.

Create day15/four_dags.py:

# day15/four_dags.py   (python -m day15.four_dags)
"""Mini-project: find the adjustment sets for four classic DAGs, then draw them."""
from causalkit.dags import draw_dag, make_dag, minimal_adjustment_sets

DAGS = {
    "1_confounder": make_dag([("season", "price_cut"), ("season", "sales"), ("price_cut", "sales")]),
    "2_m_bias": make_dag([
        ("ad_budget", "price_cut"), ("ad_budget", "store_traffic"),
        ("competitor", "store_traffic"), ("competitor", "sales"), ("price_cut", "sales")],
        latent={"ad_budget", "competitor"}),
    "3_mediator": make_dag([
        ("loyalty", "price_cut"), ("loyalty", "sales"),
        ("price_cut", "basket_size"), ("basket_size", "sales"), ("price_cut", "sales")]),
    "4_proxy_chain": make_dag([
        ("demand", "forecast"), ("forecast", "price_cut"), ("demand", "sales"), ("price_cut", "sales")],
        latent={"demand"}),
}

for name, g in DAGS.items():
    sets = minimal_adjustment_sets(g, "price_cut", "sales")
    pretty = [("{" + ", ".join(sorted(s)) + "}") for s in sets] or ["none exists"]
    print(f"{name:<14} minimal adjustment sets: {', '.join(pretty)}")
    draw_dag(g, f"figures/day15_{name}.png", name.replace("_", " "), treatment="price_cut", outcome="sales")

# The M-bias trap: store_traffic happens BEFORE the price cut, yet adjusting for it opens a path
from causalkit.dags import is_valid_adjustment  # noqa: E402

print("\nM-bias: is {store_traffic} a valid adjustment set?",
      is_valid_adjustment(DAGS["2_m_bias"], "price_cut", "sales", {"store_traffic"}))

python -m day15.four_dags

1_confounder   minimal adjustment sets: {season}
2_m_bias       minimal adjustment sets: {}
3_mediator     minimal adjustment sets: {loyalty}
4_proxy_chain  minimal adjustment sets: {forecast}

M-bias: is {store_traffic} a valid adjustment set? False

• M-bias is the counterexample to "control for everything measured before treatment". store_traffic is a collider of two latent causes; controlling for it opens price_cut ← ad_budget → store_traffic ← competitor → sales.
• DAG 4 is the NorthMart lesson in reverse: here the latent confounder affects treatment only through an observed variable, so the effect is identifiable. Compare with NorthMart, where intensity affects the email directly.

Open the four figures in figures/ and add one sentence under each in docs/ESTIMANDS.md explaining its adjustment set in plain words.

15.6 Save your work with Git (15 min)

git add .
git commit -m "Day 15: backdoor criterion, adjustment sets, M-bias"
git push

Day 15 checklist

• I can find every backdoor path from a treatment to an outcome
• I can check the backdoor criterion for a candidate adjustment set
• I can explain why NorthMart has no valid adjustment set among its observed variables
• I can explain why a tight confidence interval says nothing about bias
• I can recognise M-bias and explain why "control for all pre-treatment variables" can fail