Commit 1818503
Restructure canonical validation: replace one-shot script with regression tests
Per review: the validation script + findings doc were one-shot
artifacts that would age poorly. Replace them with
``tests/test_br_dr_canonical_datasets.py`` — pytest regression
guards that assert canonical properties (direction, PT verdict tier,
HonestDiD breakdown_M tier, cross-estimator consistency) on each
canonical fit.
Uses the ``_construct_*`` fallback data from ``diff_diff.datasets``
so tests have no network dependency (same pattern
``test_datasets.py`` already uses).
Tests cover:
- Card-Krueger (1994): positive sign, CI includes zero, "consistent
with no effect" prose.
- mpdta (CS 2021 benchmark): negative ATT, breakdown_M > 1.0,
no_detected_violation pre-trends.
- Castle Doctrine (Cheng-Hoekstra 2013) under CS: positive ATT,
clear_violation pre-trends, fragile sensitivity (breakdown_M < 0.5).
- Castle Doctrine cross-estimator consistency: SA agrees with CS on
direction and PT verdict bin.
- Treatment-label capitalization bugs: ``NJ`` abbreviation and
``Castle Doctrine`` proper noun preserved through BR's sentence
capitalization.
- ``breakdown_M == 0`` edge case: BR summary uses smallest-grid-point
wording, not the degenerate ``0x`` multiplier.
Drops:
- ``docs/validation/validate_br_dr_canonical.py`` — one-shot script,
replaced by the regression tests.
- ``docs/validation/br_dr_canonical_validation.md`` — raw dump,
regenerable on demand if needed but not checked in.
- ``docs/validation/br_dr_canonical_findings.md`` — summary now
lives in the regression-test docstrings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 73ef44c commit 1818503
4 files changed
Lines changed: 356 additions & 1206 deletions
File tree
- docs/validation
- tests
This file was deleted.
0 commit comments