From a Causal Question to a Dataset

1.1 Codespaces and your course repository (40 min)

Concept. A causal estimate is only credible if someone else can rerun it and get the same number. That starts on Day 1: one repository, one environment, every step in a script.

Step 1 — Create a repository. On GitHub, click New repository, name it causal-decision-lab, tick Add a README, and click Create repository.

Step 2 — Launch a Codespace. On the repository page click Code → Codespaces → Create codespace on main. After a minute you'll see VS Code in your browser.

Step 3 — Find the terminal. At the bottom of the screen there is a Terminal panel (if not, press Ctrl+`). Check Python:

python --version

Any Python from 3.11 upward works. Now create the folders we will fill over three days:

mkdir -p data docs sql day01 day02 day03
ls

Step 4 — Your first script. In the Explorer (left panel), create day01/hello_causal.py:

# day01/hello_causal.py
question = "Did the spring email make customers buy more?"
treatment = "received the SPRING26 email"
outcome = "number of orders in the 8 weeks after 2 March 2026"

print("Causal question:", question)
print("Treatment:", treatment)
print("Outcome:", outcome)

Run it from the repository root:

python day01/hello_causal.py

Causal question: Did the spring email make customers buy more?
Treatment: received the SPRING26 email
Outcome: number of orders in the 8 weeks after 2 March 2026

Always run scripts from the repository root (causal-decision-lab/). All file paths in this course, like data/raw/orders.csv, are relative to it.

Using Colab instead? Start each notebook with !git clone https://github.com/<you>/causal-decision-lab.git and %cd causal-decision-lab, then run scripts with !python day01/hello_causal.py.

Try it. Edit the script so it also prints a variable unit = "customer" and a variable period = "weekly".

unit = "customer"
period = "weekly"
print("Unit of analysis:", unit)
print("Time period:", period)

1.2 Virtual environments and pinned packages (40 min)

Concept. Polars 1 and Polars 2 give different answers to some calls. If two analysts use different versions, they can get different numbers from the same code. A virtual environment is a private set of packages for this project, and requirements.txt pins the exact versions.

Example 1 — Create and activate the environment.

python -m venv .venv
source .venv/bin/activate

Your prompt now starts with (.venv). On Windows (outside Codespaces) the second line is .venv\Scripts\activate.

Example 2 — Pin the packages. Create requirements.txt in the repository root:

# Pinned so every student gets identical results (tested 31 Oct 2026, Python 3.13)
# Days 1-3: data wrangling
polars==2.1.0
duckdb==1.5.6          # not 2.x yet: dlt (Day 5) supports DuckDB 1.x only
numpy==2.4.6           # not 2.5 yet: PyMC (Day 47) needs numpy < 2.5
pyarrow==26.0.0
# Days 4-5: APIs and ingestion
requests==2.34.2
fastapi==0.150.0
uvicorn==0.54.0
dlt[duckdb]==1.32.0
# Days 6-7: transformation
dbt-core==1.12.6
dbt-duckdb==1.11.1
# Day 9: orchestration
prefect==3.9.1

Install them:

pip install -r requirements.txt
python -c "import polars, duckdb; print(polars.__version__, duckdb.__version__)"

2.1.0 1.5.6

The file already lists every package Module 1 needs (dlt, dbt, Prefect…), so you install once today and never wonder later whether your environment matches your classmates'.

Example 3 — Tell Git what not to save. Data files are rebuilt from scripts, and secrets must never be shared, so neither belongs in Git. Create .gitignore (some lines only matter from Day 4 on):

# Environments and caches
.venv/
__pycache__/

# Data is rebuilt from code, never committed
data/raw/
data/clean/
data/marts/
data/api/
data/published/
data/*.duckdb
data/*.duckdb.wal

# Secrets and local state
.dlt/secrets.toml
.dlt_pipelines/
api/clock.txt

# dbt build output
transform/target/
transform/logs/
transform/dbt_packages/
transform/.user.yml
logs/

Rule of thumb: commit code and documentation; regenerate data. That way the repository is the recipe.

Try it. Deactivate the environment (deactivate), try python -c "import polars", and note what happens. Then reactivate it.

1.3 Generating the NorthMart data (25 min)

Concept. We use simulated data so that we know the true answer. Students do not need to understand every line of the generator yet. Read the docstring at the top; by Module 4 you will be able to write such simulations yourself.

Create data/make_northmart.py:

"""
Generate the NorthMart course dataset (simulated online retailer).

Run once from the repository root:
    python data/make_northmart.py

It writes four CSV files to data/raw/:
    customers.csv       one row per customer
    products.csv        one row per product
    orders.csv          one row per order line (deliberately a little messy)
    email_campaign.csv  one row per customer who received the spring email

Everything is simulated, so we KNOW the true causal effect of the email:
treated customers place about 15% more orders in the 8 weeks after it is sent.
Marketing did not send it at random: heavy shoppers were more likely to get it.
That targeting is what makes naive comparisons misleading later in the course.
"""

from datetime import date, timedelta
from pathlib import Path

import numpy as np
import polars as pl

SEED = 2026
N_CUSTOMERS = 4_000
FIRST_MONDAY = date(2026, 1, 5)     # week 1 starts here
N_WEEKS = 16                        # weeks 1-16, through Sunday 2026-04-26
CAMPAIGN_WEEK = 9                   # email sent Monday 2026-03-02
TRUE_EFFECT = 0.15                  # +15% order rate after the email

OUT = Path(__file__).parent / "raw"


def make_customers(rng: np.random.Generator) -> pl.DataFrame:
    n = N_CUSTOMERS
    regions = ["ON", "QC", "BC", "AB", "ATL"]
    region = rng.choice(regions, size=n, p=[0.38, 0.23, 0.15, 0.12, 0.12])
    is_loyalty = rng.random(n) < 0.30
    age = rng.integers(18, 76, size=n)
    signup_offset = rng.integers(0, 3 * 365, size=n)
    signup_date = [date(2023, 1, 1) + timedelta(days=int(d)) for d in signup_offset]

    # Latent shopping intensity (orders per week). Never shown to students directly.
    base_rate = rng.lognormal(mean=-1.6, sigma=0.6, size=n) * np.where(is_loyalty, 1.6, 1.0)

    return pl.DataFrame({
        "customer_id": [f"C{i:05d}" for i in range(1, n + 1)],
        "signup_date": signup_date,
        "region": region,
        "age": age,
        "is_loyalty": is_loyalty,
        "_base_rate": base_rate,
    })


def make_products(rng: np.random.Generator) -> pl.DataFrame:
    categories = {
        "Grocery": (4, 25), "Home": (15, 120), "Electronics": (40, 400),
        "Apparel": (20, 90), "Beauty": (8, 60),
    }
    rows = []
    pid = 1
    for cat, (lo, hi) in categories.items():
        for _ in range(8):
            rows.append({"product_id": f"P{pid:03d}", "category": cat,
                         "base_price": round(float(rng.uniform(lo, hi)), 2)})
            pid += 1
    return pl.DataFrame(rows)


def assign_campaign(rng: np.random.Generator, customers: pl.DataFrame) -> np.ndarray:
    """Targeted, not random: heavier shoppers and loyalty members are more likely to get the email."""
    z = np.log(customers["_base_rate"].to_numpy())
    z = (z - z.mean()) / z.std()
    logit = -1.2 + 1.1 * z + 0.6 * customers["is_loyalty"].to_numpy()
    p = 1 / (1 + np.exp(-logit))
    return rng.random(len(p)) < p


def make_orders(rng, customers, products, treated) -> pl.DataFrame:
    base = customers["_base_rate"].to_numpy()
    cust_ids = customers["customer_id"].to_numpy()
    prod_ids = products["product_id"].to_numpy()
    prices = products["base_price"].to_numpy()

    rows = []
    order_no = 1
    for week in range(1, N_WEEKS + 1):
        season = 1.0 + 0.10 * np.sin(2 * np.pi * week / 16)        # mild seasonality
        lift = np.where(treated & (week >= CAMPAIGN_WEEK), 1 + TRUE_EFFECT, 1.0)
        n_orders = rng.poisson(base * season * lift)
        week_start = FIRST_MONDAY + timedelta(weeks=week - 1)
        for i in np.nonzero(n_orders)[0]:
            for _ in range(n_orders[i]):
                k = rng.integers(len(prod_ids))
                discount = rng.choice([0, 0, 0, 10, 20])
                rows.append({
                    "order_id": f"O{order_no:06d}",
                    "customer_id": cust_ids[i],
                    "order_date": (week_start + timedelta(days=int(rng.integers(7)))).isoformat(),
                    "product_id": prod_ids[k],
                    "quantity": int(rng.integers(1, 4)),
                    "unit_price": round(prices[k] * (1 - discount / 100), 2),
                    "discount_pct": int(discount),
                    "status": "cancelled" if rng.random() < 0.03 else "completed",
                })
                order_no += 1
    orders = pl.DataFrame(rows)

    # Real data is messy: add ~1% exact duplicate rows (a double-loaded batch).
    dupes = orders.sample(fraction=0.01, seed=SEED)
    return pl.concat([orders, dupes]).sort("order_id")


def messy_regions(rng, customers: pl.DataFrame) -> pl.DataFrame:
    """Lower-case some codes, pad some with spaces, blank out ~2%."""
    region = customers["region"].to_list()
    for i in range(len(region)):
        u = rng.random()
        if u < 0.02:
            region[i] = None
        elif u < 0.06:
            region[i] = region[i].lower()
        elif u < 0.09:
            region[i] = f" {region[i]} "
    return customers.with_columns(pl.Series("region", region))


def main() -> None:
    rng = np.random.default_rng(SEED)
    OUT.mkdir(parents=True, exist_ok=True)

    customers = make_customers(rng)
    products = make_products(rng)
    treated = assign_campaign(rng, customers)
    orders = make_orders(rng, customers, products, treated)

    campaign = (customers.filter(pl.Series(treated))
                .select("customer_id")
                .with_columns(campaign_id=pl.lit("SPRING26"),
                              sent_date=pl.lit(FIRST_MONDAY + timedelta(weeks=CAMPAIGN_WEEK - 1))))

    messy_regions(rng, customers).drop("_base_rate").write_csv(OUT / "customers.csv")
    products.write_csv(OUT / "products.csv")
    orders.write_csv(OUT / "orders.csv")
    campaign.write_csv(OUT / "email_campaign.csv")

    print(f"customers:      {customers.height:>7,}")
    print(f"products:       {products.height:>7,}")
    print(f"order lines:    {orders.height:>7,}")
    print(f"emails sent:    {campaign.height:>7,}")
    print(f"Files written to {OUT}")


if __name__ == "__main__":
    main()

Run it:

python data/make_northmart.py

customers:        4,000
products:            40
order lines:     18,542
emails sent:      1,279
Files written to /workspaces/causal-decision-lab/data/raw

Because the generator uses a fixed seed (SEED = 2026), everyone gets the same 18,542 rows. Peek at a file:

head -3 data/raw/orders.csv

order_id,customer_id,order_date,product_id,quantity,unit_price,discount_pct,status
O000001,C00003,2026-01-08,P011,1,61.62,20,completed
O000002,C00006,2026-01-10,P010,1,75.36,10,completed

Try it. In the generator, find the three lines that make the email non-random. Which customer characteristics raise the chance of receiving it?

logit = -1.2 + 1.1 * z + 0.6 * customers["is_loyalty"].to_numpy()
p = 1 / (1 + np.exp(-logit))
return rng.random(len(p)) < p

1.4 The causal vocabulary in plain Python (50 min)

Concept. Before any data work, every column needs a causal role:

The last row of covariates hides the most common mistake in applied work: using a variable measured after the treatment as a control. We call these post-treatment variables, and Module 2 explains why they bias estimates.

Example 1 — One customer as a dictionary. Create day01/roles.py:

# day01/roles.py
# One real NorthMart customer, written by hand as a Python dictionary
customer = {
    "customer_id": "C00004",
    "region": "ON",
    "age": 22,
    "is_loyalty": True,
    "orders_before": 7,     # weeks 1-8
    "orders_after": 2,      # weeks 9-16
    "got_email": True,
}

# What role does each column play in a causal question?
roles = {
    "customer_id": "unit",
    "region": "covariate",
    "age": "covariate",
    "is_loyalty": "covariate",
    "orders_before": "pre-treatment outcome",
    "orders_after": "outcome",
    "got_email": "treatment",
}

for column, role in roles.items():
    print(f"{column:<15} {role:<22} value = {customer[column]}")

change = customer["orders_after"] - customer["orders_before"]
print(f"\nChange in orders: {change:+d}")

python day01/roles.py

customer_id     unit                   value = C00004
region          covariate              value = ON
age             covariate              value = 22
is_loyalty      covariate              value = True
orders_before   pre-treatment outcome  value = 7
orders_after    outcome                value = 2
got_email       treatment              value = True

Change in orders: -5

Three Python ideas are doing the work: a dictionary ({key: value}), a for loop over .items(), and an f-string (f"{column:<15}" pads the text to 15 characters so the columns line up).

Example 2 — The fundamental problem, in one line. Customer C00004 got the email and then ordered less (7 orders before, 2 after). Did the email hurt? We can't tell: what they would have ordered without it does not exist in any dataset. Maybe they would have ordered zero. Every method in this course is a way of estimating that missing number from other customers or other periods.

orders_after_with_email = 2
orders_after_without_email = None   # never observed: the counterfactual

Try it.

1. Write a function role_of(column) that returns the role from a dictionary, or "unknown" if the column isn't listed.
2. Write pct_change(before, after) returning the percentage change, or None when before is 0.
3. (Thinking) A colleague proposes controlling for "number of discount codes used in March". Is that a covariate? Why or why not?

ROLES = {"customer_id": "unit", "got_email": "treatment", "orders_after": "outcome"}

def role_of(column):
    return ROLES.get(column, "unknown")

def pct_change(before, after):
    if before == 0:
        return None
    return (after - before) / before * 100

print(role_of("got_email"))
print(role_of("favourite_colour"))
print(pct_change(4, 5))
print(pct_change(0, 3))

treatment
unknown
25.0
None

1.5 A first look with Polars (55 min)

Concept. Polars is a fast DataFrame library: a table with named, typed columns. It is the modern replacement for pandas in this course because it is quick on large files and makes types explicit.

Example 1 — Read, count, look. Create day01/first_look.py:

# day01/first_look.py
import polars as pl

customers = pl.read_csv("data/raw/customers.csv", try_parse_dates=True)
orders = pl.read_csv("data/raw/orders.csv", try_parse_dates=True)
campaign = pl.read_csv("data/raw/email_campaign.csv", try_parse_dates=True)

print("customers:", customers.shape)
print("orders:   ", orders.shape)
print("campaign: ", campaign.shape)

print(customers.head())
print(orders.schema)
print(orders.select("quantity", "unit_price", "discount_pct").describe())

share = campaign.height / customers.height
print(f"\nShare of customers who got the email: {share:.1%}")
print("First order date:", orders["order_date"].min())
print("Last order date: ", orders["order_date"].max())

python day01/first_look.py

Key parts of the output:

customers: (4000, 5)
orders:    (18542, 8)
campaign:  (1279, 3)
...
Schema({'order_id': String, 'customer_id': String, 'order_date': Date, 'product_id': String,
        'quantity': Int64, 'unit_price': Float64, 'discount_pct': Int64, 'status': String})
...
Share of customers who got the email: 32.0%
First order date: 2026-01-05
Last order date:  2026-04-26

Notes:

• .shape is (rows, columns).
• try_parse_dates=True turns "2026-01-08" into a real Date, so we can compare and sort dates.
• .describe() gives count, nulls, mean, quartiles: the same summary you'd report in Table 1 of a paper.
• The data covers 16 weeks, 8 before the email and 8 after. That pre/post split matters all course.

Example 2 — Single columns. A column is a Series; it has its own methods.

orders["customer_id"].n_unique()     # distinct customers who ordered
orders["status"].value_counts()      # completed vs cancelled
customers["age"].mean()

Try it. Using data/raw/orders.csv:

1. How many distinct customers placed at least one order? Compare with the 4,000 customers.
2. What share of order lines are cancelled?
3. Are there duplicate rows? (Hint: orders.is_duplicated().)

import polars as pl
orders = pl.read_csv("data/raw/orders.csv", try_parse_dates=True)
print("Customers who ordered:", orders["customer_id"].n_unique())
cancelled_share = (orders["status"] == "cancelled").mean()
print(f"Cancelled share: {cancelled_share:.1%}")
print("Duplicate rows:", orders.is_duplicated().sum() // 2)

Customers who ordered: 3738
Cancelled share: 2.8%
Duplicate rows: 183

1.6 Day 1 mini-project: a data dictionary with causal roles (45 min)

Goal. Write day01/data_dictionary.py that profiles every raw file and records, for each column, its type, nulls, distinct values, an example and its causal role. It also saves docs/data_dictionary.md so your team can read it on GitHub.

# day01/data_dictionary.py
"""Profile every raw file and record the causal role of each column."""
from pathlib import Path

import polars as pl

RAW = Path("data/raw")

ROLES = {
    "customer_id": "unit",
    "signup_date": "covariate (pre-treatment)",
    "region": "covariate (pre-treatment)",
    "age": "covariate (pre-treatment)",
    "is_loyalty": "covariate (pre-treatment)",
    "order_id": "identifier",
    "order_date": "time",
    "product_id": "identifier",
    "quantity": "outcome ingredient",
    "unit_price": "outcome ingredient",
    "discount_pct": "possible mediator / other treatment",
    "status": "filter",
    "category": "covariate",
    "base_price": "covariate",
    "campaign_id": "treatment label",
    "sent_date": "treatment timing",
}


def profile(name: str, df: pl.DataFrame) -> list[dict]:
    rows = []
    for col in df.columns:
        rows.append({
            "table": name,
            "column": col,
            "dtype": str(df[col].dtype),
            "nulls": df[col].null_count(),
            "unique": df[col].n_unique(),
            "example": str(df[col].drop_nulls()[0]),
            "role": ROLES.get(col, "?"),
        })
    return rows


def main() -> None:
    all_rows = []
    for path in sorted(RAW.glob("*.csv")):
        df = pl.read_csv(path, try_parse_dates=True)
        all_rows += profile(path.stem, df)

    dictionary = pl.DataFrame(all_rows)
    with pl.Config(tbl_rows=-1, tbl_width_chars=140):
        print(dictionary)

    # Save a Markdown version for the docs/ folder
    lines = ["| table | column | dtype | nulls | unique | example | role |",
             "|---|---|---|---|---|---|---|"]
    for r in dictionary.iter_rows(named=True):
        lines.append(f"| {r['table']} | {r['column']} | {r['dtype']} | {r['nulls']} | "
                     f"{r['unique']} | {r['example']} | {r['role']} |")
    Path("docs").mkdir(exist_ok=True)
    Path("docs/data_dictionary.md").write_text("# NorthMart data dictionary\n\n" + "\n".join(lines) + "\n")
    print("\nSaved docs/data_dictionary.md")


if __name__ == "__main__":
    main()

python day01/data_dictionary.py

│ customers      ┆ region       ┆ String  ┆ 74    ┆ 16     ┆ ON         ┆ covariate (pre-treatment)       │
...
│ orders         ┆ order_id     ┆ String  ┆ 0     ┆ 18359  ┆ O000001    ┆ identifier                      │
...
Saved docs/data_dictionary.md

Read the profile like a detective:

• region has 16 distinct values, but Canada has 5 regions in our data. Something is messy.
• region has 74 nulls.
• order_id has 18,359 distinct values in 18,542 rows. IDs should be unique, so there are duplicates.

These three findings become Day 2's cleaning tasks.

Extension. Add a pct_null column (nulls as a share of rows) and flag any column above 1%.

"pct_null": round(df[col].null_count() / df.height * 100, 2),

print(dictionary.filter(pl.col("pct_null") > 1).select("table", "column", "pct_null"))

1.7 Save your work with Git (15 min)

git add .
git status          # data/raw should NOT appear: .gitignore works
git commit -m "Day 1: environment, NorthMart data generator, data dictionary"
git push

Open your repository on GitHub and click docs/data_dictionary.md: the table renders as a web page.

Day 1 checklist

• I can open a Codespace and run a script from the repository root
• I can create and activate a virtual environment and install pinned packages
• I can explain unit, treatment, outcome, time, covariate and post-treatment variable
• I can read a CSV with Polars and inspect its shape, schema and summary
• I have a data dictionary that records the causal role of every column