Encoding Assumptions with DoWhy

18.1 DoWhy's four steps (40 min)

Concept. DoWhy (originally from Microsoft Research, now part of the PyWhy project) organises every causal analysis into four explicit steps:

Its value is the separation: the assumptions (step 1) and the statistics (step 3) are written in different places, so a reviewer can disagree with one without re-reading the other. It also refuses to estimate when step 2 fails, unlike a regression, which always returns a number.

DoWhy works with pandas DataFrames, so we convert with .to_pandas(). It prints many library warnings; the scripts hide them so you can read the results.

18.2 Model and identify: when DoWhy says "no" (50 min)

Create day18/dowhy_workflow.py. Read it top to bottom; we'll discuss each step's output in turn.

# day18/dowhy_workflow.py   (python -m day18.dowhy_workflow)
"""DoWhy's four steps: model, identify, estimate, refute."""
import warnings

import networkx as nx
import polars as pl
from dowhy import CausalModel

from causalkit.data import load_customers

warnings.filterwarnings("ignore")      # DoWhy prints many library warnings; we read the results instead

df = (load_customers()
      .with_columns(pl.col("email").cast(pl.Int8), loyalty=pl.col("is_loyalty").cast(pl.Int8))
      .select("email", "post_orders", "loyalty", "pre_orders")
      .to_pandas())

# STEP 1 - MODEL. Honest DAG: intensity is unobserved (it is not a column in df).
honest = nx.DiGraph([("loyalty", "intensity"), ("loyalty", "email"), ("intensity", "email"),
                     ("intensity", "pre_orders"), ("intensity", "post_orders"), ("email", "post_orders")])
m_honest = CausalModel(data=df, treatment="email", outcome="post_orders", graph=honest)

# STEP 2 - IDENTIFY.
print("HONEST DAG - backdoor variables:", m_honest.identify_effect().get_backdoor_variables() or "none: not identified")

# A stronger, explicit assumption: pre_orders captures everything about intensity that drove targeting
proxy = nx.DiGraph([("loyalty", "email"), ("pre_orders", "email"),
                    ("loyalty", "post_orders"), ("pre_orders", "post_orders"), ("email", "post_orders")])
m = CausalModel(data=df, treatment="email", outcome="post_orders", graph=proxy)
estimand = m.identify_effect()
print("PROXY DAG  - backdoor variables:", estimand.get_backdoor_variables())

# STEP 3 - ESTIMATE, two ways, same estimand
lin = m.estimate_effect(estimand, method_name="backdoor.linear_regression")
ipw = m.estimate_effect(estimand, method_name="backdoor.propensity_score_weighting",
                        target_units="att")
print(f"\nLinear regression (ATE)       {lin.value:+.3f}")
print(f"Propensity weighting (ATT)    {ipw.value:+.3f}")
print("Truth: ATE +0.319, ATT +0.443")

# STEP 4 - REFUTE
for name, kwargs in [("placebo_treatment_refuter", {"placebo_type": "permute"}),
                     ("random_common_cause", {}),
                     ("data_subset_refuter", {"subset_fraction": 0.8})]:
    r = m.refute_estimate(estimand, lin, method_name=name, random_seed=18, **kwargs)
    print(f"{name:<28} new effect {r.new_effect:+.3f}")

python -m day18.dowhy_workflow

The first line of output:

HONEST DAG - backdoor variables: none: not identified

We gave DoWhy the Day 14 DAG, including intensity. Because intensity is not a column of the data, DoWhy treats it as unobserved and concludes, as causalkit did on Day 15, that no backdoor adjustment set exists. A regression would have given an estimate anyway; DoWhy won't.

18.3 Estimate under an explicit, stronger assumption (45 min)

To proceed, the script writes down a different, stronger DAG: "everything about intensity that drove targeting is captured by pre_orders and loyalty". In that DAG, pre_orders is a confounder itself, and DoWhy identifies the effect:

PROXY DAG  - backdoor variables: ['pre_orders', 'loyalty']

Linear regression (ATE)       +1.062
Propensity weighting (ATT)    +0.928
Truth: ATE +0.319, ATT +0.443

The point isn't the number (still 2–3 times too large: the stronger assumption is false). The point is that the assumption is now written down, in code, where a reviewer can read it and say "I don't believe pre-period orders capture intensity". Compare with a typical analysis, where the same assumption is hidden in a list of regression controls.

Two estimators of the same identified estimand give different numbers (1.06 vs 0.93) partly because they target different populations (ATE vs ATT) and partly because they weigh customers differently. Module 4 covers both.

18.4 Refute: what the checks can and can't see (45 min)

The last three lines:

placebo_treatment_refuter    new effect -0.000
random_common_cause          new effect +1.062
data_subset_refuter          new effect +1.062

All three pass. The estimate is still 2.4 times the truth.

These refuters detect bugs (a leaky pipeline, an estimate driven by a few customers, code that finds effects everywhere). None of them can detect an unobserved confounder: by definition, it's not in the data they test. Passing refuters is necessary, not sufficient. For hidden confounding, you need sensitivity analysis: asking how strong a hidden confounder would have to be to change the conclusion.

18.5 Day 18 mini-project: omitted-variable bias, exactly (75 min)

Concept. Economists know the omitted-variable bias (OVB) formula from introductory econometrics. If the true model includes U but you leave it out:

bias = δ × γ
  δ = how much more U the treated have, after controlling for the included variables
  γ = the effect of U on the outcome

It's also the simplest sensitivity analysis: you don't know δ or γ, but you can ask how large they would need to be to explain your estimate away. Usually that's an argument. In NorthMart, the truth file contains U (intensity, mu0), so we can compute it exactly.

Create day18/omitted_variable_bias.py:

# day18/omitted_variable_bias.py   (python -m day18.omitted_variable_bias)
"""
Mini-project, part 1: explain the gap between the estimate and the truth with the
omitted-variable-bias formula:   bias = delta x gamma
    delta = how much more intensity the emailed have, AFTER controlling for loyalty and pre_orders
    gamma = how many extra orders one unit of intensity causes
Only possible because the truth file reveals intensity (mu0). In real data, you'd vary delta and gamma.
"""
import polars as pl
import statsmodels.formula.api as smf

from causalkit.data import load_customers

truth = pl.read_parquet("data/truth/potential_outcomes.parquet").select("customer_id", "mu0", "tau")
df = (load_customers().join(truth, on="customer_id")
      .with_columns(pl.col("email").cast(pl.Int8), loyalty=pl.col("is_loyalty").cast(pl.Int8))
      .to_pandas())

short = smf.ols("post_orders ~ email + loyalty + pre_orders", df).fit()
long = smf.ols("post_orders ~ email + loyalty + pre_orders + mu0", df).fit()
delta = smf.ols("mu0 ~ email + loyalty + pre_orders", df).fit().params["email"]
gamma = long.params["mu0"]

print(f"Estimate without intensity          {short.params['email']:+.3f}")
print(f"Estimate with intensity             {long.params['email']:+.3f}")
print(f"delta (intensity gap, adjusted)     {delta:+.3f}")
print(f"gamma (orders per unit intensity)   {gamma:+.3f}")
print(f"bias = delta x gamma                {delta * gamma:+.3f}")
print(f"short - long                        {short.params['email'] - long.params['email']:+.3f}   (identical: OVB is an identity)")

print("\nSensitivity grid: how much bias would different hidden confounders create?")
grid = pl.DataFrame({"delta": [0.1, 0.25, 0.5, 0.75]}).with_columns(
    **{f"gamma={g}": (pl.col("delta") * g).round(3) for g in [0.5, 1.0, 1.5]})
print(grid)

python -m day18.omitted_variable_bias

Estimate without intensity          +1.062
Estimate with intensity             +0.353
delta (intensity gap, adjusted)     +0.651
gamma (orders per unit intensity)   +1.090
bias = delta x gamma                +0.709
short - long                        +0.709   (identical: OVB is an identity)

Sensitivity grid: how much bias would different hidden confounders create?
│ delta ┆ gamma=0.5 ┆ gamma=1.0 ┆ gamma=1.5 │
│ 0.1   ┆ 0.05      ┆ 0.1       ┆ 0.15      │
│ 0.25  ┆ 0.125     ┆ 0.25      ┆ 0.375     │
│ 0.5   ┆ 0.25      ┆ 0.5       ┆ 0.75      │
│ 0.75  ┆ 0.375     ┆ 0.75      ┆ 1.125     │

• Even after controlling for loyalty and pre-period orders, emailed customers have 0.65 more expected orders of underlying intensity (δ). Each unit of intensity adds about 1.09 orders (γ). Together: 0.71 orders of bias, exactly the gap between the two estimates.
• The grid is how you'd argue without a truth file: "for our estimate of 1.06 to fall below the break-even effect of 0.3, a hidden confounder would need δ × γ ≥ 0.76, for example an intensity gap of 0.75 with γ around 1. Given how marketing targeted, that's plausible." That sentence, not the refuters, is what a reviewer needs.

Module 7 develops formal sensitivity tools (E-values, robustness values). Today's habit: every observational estimate comes with a sentence about what hidden confounding would overturn it.

18.6 Save your work with Git (15 min)

git add .
git commit -m "Day 18: DoWhy workflow, refuters, omitted-variable bias"
git push

Day 18 checklist

• I can run DoWhy's model, identify, estimate and refute steps
• I can explain why DoWhy refuses to estimate with NorthMart's honest DAG
• I can explain what the standard refuters detect, and why they can't detect hidden confounding
• I can decompose a bias with the OVB formula and write a sensitivity sentence for an estimate