Causal-Ready Data Engineering
Turning messy transactions into tested panels with treatment, timing, outcomes and pre-period covariates.
What you'll learn
- 1Days 1–3Causal vocabulary, Polars cleaning, DuckDB and the customer × week panel.
By the end: Students turn four messy raw files from a simulated online retailer into a tested customer × week panel with treatment, timing, outcome and pre-period covariate columns, ready for causal analysis.
- 2Days 4–7Ingesting API data with dlt and modelling the panel in dbt.
By the end: Students pull NorthMart data from a live REST API, load it incrementally into DuckDB with dlt, and rebuild the customer × week panel as a tested dbt project that reproduces the Day 3 panel exactly.
- 3Days 8–10Data contracts for leakage and treatment timing, orchestration with Prefect.
By the end: Students run a scheduled, self-checking pipeline that publishes a documented Causal Data Mart, refuses to publish when the data would mislead a causal analysis, and has been replayed week by week through the whole study period.
Tools
Mini-capstone
Causal Data Mart: a dbt-modelled customer × week panel with contracts that catch post-treatment leakage.
The ten days
Before you start: module setup and project structure
Course: Applied Causal Inference & Decision Science Audience: Economists and analysts with basic Python and statistics Environment: GitHub Codespaces (VS Code in the browser) or Google Colab Time per day: ~5 hours (including breaks and exercises) Goal by end of Day 3: Students turn four messy raw files from a simulated online retailer into a tested customer × week panel with treatment, timing, outcome and pre-period covariate columns, ready for causal analysis.
How to use this guide
Each day is split into timed blocks. Every block follows the same rhythm:
- Concept: a short explanation, always tied to a causal question.
- Examples: small, progressive code. Type it yourself; don't copy-paste.
- Try it: a quick exercise. Solutions are hidden in collapsible blocks.
The running case: NorthMart
NorthMart is a simulated Canadian online retailer. On Monday 2 March 2026 marketing sent the SPRING26 email to about a third of customers. Management asks one question:
Did the spring email make customers buy more?
We will answer that question properly in Modules 2–7. Module 1 builds the data that makes an honest answer possible.
Because the data is simulated, we know the truth: the email raised order frequency by about 15%. We will use that truth to show how far naive analysis can be from it. One more detail matters: marketing did not pick recipients at random. It favoured heavy shoppers and loyalty members.
Tested versions
Every script in these notes was run on 31 October 2026 with Python 3.13, Polars 2.1.0 and DuckDB 1.5.6 (full list in requirements.txt). DuckDB 2.0 exists, but the ingestion tool we use from Day 5 (dlt) does not support it yet, so the whole course stays on 1.5.6. Answers you find online may target other versions; trust the pinned ones.
Target folder structure (built up over three days)
causal-decision-lab/
├── .gitignore
├── requirements.txt
├── data/
│ ├── make_northmart.py # provided: generates the raw data
│ ├── raw/ # 4 CSV files (not committed)
│ ├── clean/ # 4 Parquet files (not committed)
│ ├── marts/ # analysis-ready tables (not committed)
│ └── northmart.duckdb # local warehouse (not committed)
├── docs/
│ └── data_dictionary.md
├── sql/
│ └── panel_customer_week.sql
├── day01/
│ ├── hello_causal.py
│ ├── roles.py
│ ├── first_look.py
│ └── data_dictionary.py
├── day02/
│ ├── expressions.py
│ ├── clean.py
│ ├── summaries.py
│ ├── lazy_demo.py
│ └── naive_comparison.py
└── day03/
├── duck_intro.py
├── build_warehouse.py
├── calendar_spine.py
├── windows.py
└── build_panel.py