Data Contracts: Leakage and Treatment Timing

8.1 From tests to contracts (40 min)

Concept. A data contract is a promise about data that is written down, owned by someone, and enforced automatically. Tests are how we enforce them. The difference is intent: a contract states why the rule matters and what happens when it breaks.

For causal work, the most important contracts are not about tidiness. They protect the assumptions every estimate relies on:

Each contract has a severity:

• error: stop. Nothing downstream is rebuilt or published.
• warn: report, keep going. A human decides.

Overlap problems are warnings because the right response (pool regions, drop a group, change the estimand) is an analytical choice, not a data bug.

8.2 Schema contracts: names and types that cannot drift (50 min)

Concept. Estimation code in Modules 2–8 reads panel_customer_week by column name and type. If someone renames n_orders, or a type changes from integer to text, every later analysis breaks or, worse, silently casts. A dbt model contract fixes the columns' names and types: the build fails if the SQL produces anything else.

A surprise worth seeing first. Ask DuckDB what type pre_orders has today:

cd transform
dbt show --profiles-dir . --inline "select typeof(pre_orders) as t from {{ ref('panel_customer_week') }} limit 1"

| t       |
| ------- |
| HUGEINT |

SUM over integers returns a 128-bit integer in DuckDB. Harmless in SQL, but Polars and pandas handle it differently, and in Module 6 machine-learning libraries may refuse it. A contract makes us choose types deliberately.

Step 1 — make the type explicit. In models/marts/panel_customer_week.sql, wrap pre_orders in a cast:

    cast(sum(f.n_orders) filter (where f.week_no < {{ var("campaign_week") }})
        over (partition by f.customer_id) as bigint)               as pre_orders,

Step 2 — add a variable for the email date in dbt_project.yml (under vars:), used by today's contracts:

  campaign_date: "2026-03-02"   # features may only use data from BEFORE this date

Step 3 — enforce the contract. Replace models/marts/_marts.yml with this version. Every column now has a data_type, contract: enforced: true switches checking on, and constraints are written into the table definition itself. (The second model, customer_features, is created in 8.3.)

version: 2

models:
  - name: panel_customer_week
    description: "One row per customer per week. Outcomes are 0 (not NULL) in weeks without orders."
    config:
      contract:
        enforced: true          # the build fails if a column's name or type changes
    data_tests:
      - unique_combination:
          arguments:
            columns: ["customer_id", "week_start"]
    columns:
      - name: customer_id
        data_type: varchar
        constraints: [{type: not_null}]
      - name: week_start
        data_type: date
        constraints: [{type: not_null}]
      - name: week_no
        data_type: bigint
      - name: event_week
        data_type: bigint
        description: "Weeks relative to the email: 0 = the email week, negative = before."
      - name: treated
        data_type: integer
        description: "1 if the customer ever received the SPRING26 email."
        data_tests:
          - accepted_values:
              arguments:
                values: [0, 1]
                quote: false
      - name: post
        data_type: integer
        data_tests:
          - accepted_values:
              arguments:
                values: [0, 1]
                quote: false
      - name: treated_post
        data_type: integer
      - name: n_orders
        data_type: bigint
        constraints: [{type: not_null}]
      - name: revenue
        data_type: double
        constraints: [{type: not_null}]
      - name: region
        data_type: varchar
      - name: age
        data_type: integer
      - name: is_loyalty
        data_type: integer
      - name: pre_orders
        data_type: bigint
        description: "Orders in weeks 1-8. Fixed per customer: a pre-treatment covariate."
        constraints: [{type: not_null}]
      - name: pre_revenue
        data_type: double

  - name: customer_features
    description: "One row per customer: treatment plus covariates computed ONLY from data before the email."
    config:
      contract:
        enforced: true
    columns:
      - name: customer_id
        data_type: varchar
        constraints: [{type: not_null}, {type: unique}]
      - name: treated
        data_type: integer
      - name: region
        data_type: varchar
      - name: age
        data_type: integer
      - name: is_loyalty
        data_type: integer
      - name: tenure_days
        data_type: bigint
        description: "Days from signup to the email date."
      - name: pre_orders
        data_type: bigint
      - name: pre_revenue
        data_type: double
      - name: pre_active_weeks
        data_type: bigint
      - name: pre_discount_share
        data_type: double
        description: "Share of pre-period orders that carried a discount (NULL if no orders)."
      - name: days_since_last_order
        data_type: bigint
        description: "Days from the last pre-period order to the email date (NULL if no orders)."

See it fail. Temporarily remove the cast(... as bigint) you just added and run:

dbt run --profiles-dir . --select panel_customer_week

ERROR creating sql table model marts.panel_customer_week
  This model has an enforced contract that failed.
  | column_name | definition_type | contract_type | mismatch_reason    |
  | ----------- | --------------- | ------------- | ------------------ |
  | pre_orders  | HUGEINT         | BIGINT        | data type mismatch |

The model is not built, so the previous version of the table stays in place. Put the cast back.

8.3 The leakage contract: a time-travel test (65 min)

Concept. Leakage means a variable that is supposed to describe units before treatment secretly contains information from after it. It's the data version of a bad control (Day 1, 1.4). Leakage is hard to spot by reading SQL because the mistake is usually one missing WHERE.

The time-travel test catches it mechanically:

1. Build the features from the full data, as the pipeline normally does.
2. Build them again from orders that physically end before the email date.
3. Compare. If any value differs, some feature looked into the future.

To run the same SQL twice on different inputs, we write it once as a macro with the orders table as an argument.

Step 1 — the macro. Create transform/macros/customer_features.sql:

{#
  Customer features "as of" the email date.
  The orders relation is an argument so a test can call this macro on
  orders that physically stop before the email date, and compare.
#}
{% macro customer_features_from(orders_relation) %}
with pre_orders as (
    select *
    from {{ orders_relation }}
    where order_date < date '{{ var("campaign_date") }}'
),
per_customer as (
    select
        customer_id,
        count(distinct order_id)                                   as pre_orders,
        sum(revenue)                                               as pre_revenue,
        count(distinct date_trunc('week', order_date))             as pre_active_weeks,
        avg(case when discount_pct > 0 then 1.0 else 0.0 end)      as pre_discount_share,
        max(order_date)                                            as last_order
    from pre_orders
    group by customer_id
)
select
    c.customer_id,
    cast(r.customer_id is not null as integer)                                     as treated,
    c.region,
    c.age,
    cast(c.is_loyalty as integer)                                                  as is_loyalty,
    cast(date_diff('day', c.signup_date, date '{{ var("campaign_date") }}') as bigint) as tenure_days,
    cast(coalesce(p.pre_orders, 0) as bigint)                                      as pre_orders,
    round(coalesce(p.pre_revenue, 0), 2)                                           as pre_revenue,
    cast(coalesce(p.pre_active_weeks, 0) as bigint)                                as pre_active_weeks,
    round(p.pre_discount_share, 4)                                                 as pre_discount_share,
    cast(date_diff('day', p.last_order, date '{{ var("campaign_date") }}') as bigint) as days_since_last_order
from {{ ref('stg_customers') }} as c
left join per_customer as p using (customer_id)
left join {{ ref('stg_campaign_recipients') }} as r
    on r.customer_id = c.customer_id and r.campaign_id = '{{ var("campaign_id") }}'
{% endmacro %}

Step 2 — the model. transform/models/marts/customer_features.sql is one line:

-- One row per customer, for the estimation modules (matching, weighting, causal ML).
{{ customer_features_from(ref('stg_orders')) }}

Step 3 — the contract. Create transform/tests/assert_features_do_not_leak.sql:

-- LEAKAGE CONTRACT ("time travel" test).
-- Rebuild the features from orders that physically end before the email date.
-- If any feature changes, it was using post-treatment information.
{% set truncated_orders %}
    (select * from {{ ref('stg_orders') }} where order_date < date '{{ var("campaign_date") }}')
{% endset %}

with as_built as (
    select * from {{ ref('customer_features') }}
),
time_travel as (
    {{ customer_features_from(truncated_orders) }}
),
differences as (
    (select 'only in as_built' as side, * from as_built
     except select 'only in as_built', * from time_travel)
    union all
    (select 'only in time_travel' as side, * from time_travel
     except select 'only in time_travel', * from as_built)
)
select * from differences

EXCEPT returns rows in the first set that are missing from the second. Doing it both ways and stacking the results lists every difference.

dbt build --profiles-dir . --select customer_features+

The test passes: every feature uses only pre_orders.

See it catch a leak. In the macro, change from pre_orders (inside per_customer) to from {{ orders_relation }}, as if a colleague forgot the date filter. Then:

dbt build --profiles-dir . --select customer_features+

FAIL 6246 assert_features_do_not_leak ................................... [FAIL 6246 in 0.14s]

Look at the model with dbt show --select customer_features: nothing looks wrong. Every number is plausible. Only the time-travel test reveals that pre_orders, pre_revenue and the rest now count post-email orders, which would make every estimate in Module 4 absorb part of the email's effect. Undo the change and rebuild.

8.4 Treatment, completeness and overlap contracts (40 min)

Create four more files in transform/tests/. Each header comment states the contract and its reason.

Timing: all recipients were emailed on the official date.

-- TREATMENT-TIMING CONTRACT: every recipient of the campaign got it on the official date.
-- A backdated or late send would silently mix treatment timing.
select customer_id, sent_date
from {{ ref('stg_campaign_recipients') }}
where campaign_id = '{{ var("campaign_id") }}'
  and sent_date != date '{{ var("campaign_date") }}'

Fixed treatment: a customer is treated in every week or in none.

-- TREATMENT CONTRACT: the treatment group is a property of the customer, not of the week.
select customer_id
from {{ ref('panel_customer_week') }}
group by customer_id
having count(distinct treated) > 1

Completeness: every week has orders.

-- COMPLETENESS CONTRACT: NorthMart sells every week. A week with zero orders
-- almost always means a load was missed (e.g. a broken incremental cursor).
select week_start, sum(n_orders) as orders
from {{ ref('panel_customer_week') }}
group by week_start
having sum(n_orders) = 0

Overlap (warning): every region has at least 30 emailed and 30 non-emailed customers.

-- OVERLAP (POSITIVITY) WARNING: every region needs both emailed and non-emailed customers,
-- otherwise there is nobody to compare with. A warning, not an error: the analyst decides.
{{ config(severity='warn') }}

select
    region,
    sum(treated)            as treated_customers,
    sum(1 - treated)        as control_customers
from {{ ref('customer_features') }}
group by region
having sum(treated) < 30 or sum(1 - treated) < 30

dbt build --profiles-dir .

WARN 1 warn_overlap_by_region ......................................... [WARN 1 in 0.08s]
Done. PASS=31 WARN=1 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=32

Read the warning. Which region fails? Run the test's query yourself:

dbt show --profiles-dir . --inline "select region, sum(treated) t, sum(1-treated) c from {{ ref('customer_features') }} group by 1 order by 1"

The five provinces are fine. The UNKNOWN group (74 customers with no recorded region) has only 25 emailed customers. That isn't a bug to fix: it's a fact any analysis by region must handle, and Day 10's data card records it.

Try it. Write a contract that no customer signed up after the email date (they could not have been targeted by it). Use customer_features.tenure_days.

-- tests/assert_signup_before_email.sql
-- A customer who signed up after the email could not have been targeted by it.
select customer_id, tenure_days
from {{ ref('customer_features') }}
where tenure_days < 0

Done. PASS=1 WARN=0 ERROR=0 ...

8.5 Day 8 mini-project: the fire drill (60 min)

Goal. Contracts you've never seen fail are contracts you can't trust. Damage the raw data on purpose, four different ways, and record which contract catches each fault.

Create day08/break_things.py:

# day08/break_things.py
"""
Fire drill: damage the raw data on purpose, then check which contract catches it.

    python day08/break_things.py backdate     # 50 emails recorded two weeks early
    python day08/break_things.py drop_week    # a missed load: week 5 has no orders
    python day08/break_things.py duplicate    # 100 orders loaded twice
    python day08/break_things.py orphan       # 5 orders from a customer who doesn't exist
    python day08/break_things.py restore      # reload clean data from the API

After each fault, run `dbt build --profiles-dir .` in transform/ and read the results.
The NorthMart API must be running for `restore`.
"""
import shutil
import subprocess
import sys

import duckdb

DB = "data/northmart.duckdb"

FAULTS = {
    "backdate": """
        UPDATE raw.campaign_recipients SET sent_date = '2026-02-16'
        WHERE customer_id IN (SELECT customer_id FROM raw.campaign_recipients ORDER BY customer_id LIMIT 50)
    """,
    "drop_week": """
        DELETE FROM raw.orders WHERE order_date BETWEEN '2026-02-02' AND '2026-02-08'
    """,
    "duplicate": """
        INSERT INTO raw.orders
        SELECT * REPLACE ('9999999999.0' AS _dlt_load_id)
        FROM raw.orders ORDER BY order_id LIMIT 100
    """,
    "orphan": """
        UPDATE raw.orders SET customer_id = 'C99999'
        WHERE order_id IN (SELECT order_id FROM raw.orders WHERE status = 'completed' ORDER BY order_id LIMIT 5)
    """,
}


def restore() -> None:
    """Throw away the raw data and the pipeline's memory, then reload everything."""
    con = duckdb.connect(DB)
    con.sql("DROP SCHEMA IF EXISTS raw CASCADE")
    con.close()
    shutil.rmtree(".dlt_pipelines/northmart", ignore_errors=True)
    subprocess.run([sys.executable, "api/clock.py", "2026-04-26"], check=True)
    subprocess.run([sys.executable, "day05/northmart_pipeline.py"], check=True)


def main() -> None:
    if len(sys.argv) != 2 or sys.argv[1] not in [*FAULTS, "restore"]:
        print(__doc__)
        sys.exit(1)
    name = sys.argv[1]
    if name == "restore":
        restore()
        return
    con = duckdb.connect(DB)
    con.sql(FAULTS[name])
    con.close()
    print(f"Fault '{name}' injected into the raw schema. Now run dbt build.")


if __name__ == "__main__":
    main()

With the NorthMart API running, run each fault, rebuild, read the result, and restore before the next one:

python day08/break_things.py backdate
cd transform && dbt build --profiles-dir . ; cd ..
python day08/break_things.py restore

Repeat with drop_week, duplicate and orphan. The results:

backdate:   FAIL 50 assert_single_send_date        ... Done. PASS=19 WARN=0 ERROR=1 SKIP=12
drop_week:  FAIL 1  assert_no_empty_weeks          ... Done. PASS=30 WARN=1 ERROR=1 SKIP=0
duplicate:  (all contracts pass)                   ... Done. PASS=31 WARN=1 ERROR=0 SKIP=0
orphan:     FAIL 5  relationships_stg_orders_customer_id... Done. PASS=19 WARN=0 ERROR=1 SKIP=12

What to notice:

• backdate and orphan fail in staging, so dbt skips the 12 downstream nodes: the panel is not rebuilt on bad data.
• drop_week passes every staging test. The missing orders leave no orphan, no duplicate, no wrong type; revenue still reconciles (both sides lost the same week). Only the completeness contract notices. Internal consistency can't detect missing data; you need a rule about what should exist.
• duplicate causes no damage. Day 6's row_number() safety net in stg_orders absorbs it. A passing build is the correct result here.

dbt builds before it tests. In the drop_week case the panel model itself ran, then its test failed. The table in DuckDB now holds the damaged version. That is why Day 9 adds a separate publish step that runs only after every contract passes.

8.6 The contract register and Git (15 min)

Contracts belong in a document people can read without opening SQL. Create docs/CONTRACTS.md:

# NorthMart data contracts

Each contract is a promise the data must keep before anyone estimates a causal effect from it.
"Error" stops the build and nothing downstream is rebuilt; "warn" reports and lets the analyst decide.

| ID | Contract | Where | Severity | Causal reason |
|---|---|---|---|---|
| C1 | Panel columns keep their names and types | `_marts.yml` (`contract: enforced`) | error | Estimation code downstream breaks or silently mis-casts |
| C2 | One row per customer per week | `unique_combination`, `assert_panel_is_complete` | error | Missing or doubled unit-weeks bias averages |
| C3 | Outcomes are 0, never NULL | `not_null` constraints | error | NULL outcomes silently drop units |
| C4 | Panel revenue equals order revenue | `assert_revenue_reconciles` | error | Outcomes lost between layers |
| C5 | Every order belongs to a known customer | `relationships` test on `stg_orders` | error | Orders can't be assigned to treatment or control |
| C6 | All emails sent on the official date | `assert_single_send_date` | error | Mixed timing breaks the pre/post split |
| C7 | No treatment before the email | `assert_no_treatment_before_email` | error | "Effects" before treatment are impossible |
| C8 | Treatment group fixed per customer | `assert_treatment_is_fixed` | error | Units switching groups break the comparison |
| C9 | Features use only pre-email data | `assert_features_do_not_leak` (time travel) | error | Post-treatment controls absorb the effect |
| C10 | Every week has orders | `assert_no_empty_weeks` | error | A missed load looks like a collapse in demand |
| C11 | Each region has 30+ treated and 30+ control customers | `warn_overlap_by_region` | warn | No overlap means no valid comparison in that group |

## Known warnings

- **C11, region UNKNOWN:** only 25 emailed customers have no recorded region. Analyses by region should
  pool or drop this group and say so.

## Fire-drill results (day08/break_things.py)

| Fault injected | Caught by | Outcome |
|---|---|---|
| 50 emails backdated two weeks | C6 | Build stopped; panel not rebuilt |
| Week 5 orders missing | C10 | Build stopped at the panel test |
| 100 orders loaded twice | (staging deduplication) | No damage: all contracts pass |
| 5 orders from an unknown customer | C5 | Build stopped; panel not rebuilt |

git add .
git commit -m "Day 8: causal data contracts, time-travel leakage test, fire drill"
git push

Day 8 checklist

• I can explain the difference between a test and a contract, and between error and warn
• I can enforce a dbt model contract and read a type-mismatch failure
• I can detect feature leakage with a time-travel test built on a macro
• I can write timing, fixed-treatment, completeness and overlap contracts
• I have run the fire drill and know which contract catches which fault