Skip to content

Commit 1818503

Browse files
igerberclaude
andcommitted
Restructure canonical validation: replace one-shot script with regression tests
Per review: the validation script + findings doc were one-shot artifacts that would age poorly. Replace them with ``tests/test_br_dr_canonical_datasets.py`` — pytest regression guards that assert canonical properties (direction, PT verdict tier, HonestDiD breakdown_M tier, cross-estimator consistency) on each canonical fit. Uses the ``_construct_*`` fallback data from ``diff_diff.datasets`` so tests have no network dependency (same pattern ``test_datasets.py`` already uses). Tests cover: - Card-Krueger (1994): positive sign, CI includes zero, "consistent with no effect" prose. - mpdta (CS 2021 benchmark): negative ATT, breakdown_M > 1.0, no_detected_violation pre-trends. - Castle Doctrine (Cheng-Hoekstra 2013) under CS: positive ATT, clear_violation pre-trends, fragile sensitivity (breakdown_M < 0.5). - Castle Doctrine cross-estimator consistency: SA agrees with CS on direction and PT verdict bin. - Treatment-label capitalization bugs: ``NJ`` abbreviation and ``Castle Doctrine`` proper noun preserved through BR's sentence capitalization. - ``breakdown_M == 0`` edge case: BR summary uses smallest-grid-point wording, not the degenerate ``0x`` multiplier. Drops: - ``docs/validation/validate_br_dr_canonical.py`` — one-shot script, replaced by the regression tests. - ``docs/validation/br_dr_canonical_validation.md`` — raw dump, regenerable on demand if needed but not checked in. - ``docs/validation/br_dr_canonical_findings.md`` — summary now lives in the regression-test docstrings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 73ef44c commit 1818503

4 files changed

Lines changed: 356 additions & 1206 deletions

File tree

docs/validation/br_dr_canonical_findings.md

Lines changed: 0 additions & 175 deletions
This file was deleted.

0 commit comments

Comments
 (0)