Merge pull request #267 from igerber/survey-real-data-validation

Add survey real-data validation against R using federal survey datasets

Author: igerber

.github/workflows/ai_pr_review.yml

@@ -147,20 +147,13 @@ jobs:
             echo "Changed files:"
             git --no-pager diff --name-status "$BASE_SHA" "$HEAD_SHA"
             echo ""
-            # Identify large data files excluded from the unified diff
-            EXCLUDED_GLOBS="benchmarks/data/real/*.json benchmarks/data/real/*.csv"
-            excluded_files=$(git --no-pager diff --name-only "$BASE_SHA" "$HEAD_SHA" -- $EXCLUDED_GLOBS)
-            if [ -n "$excluded_files" ]; then
-              echo "NOTE: The following files are excluded from the unified diff below"
-              echo "due to size (they are generated data/golden-value files). Their"
-              echo "filenames appear in the 'Changed files' list above, but their"
-              echo "content is NOT shown. Review coverage for these files is metadata-only."
-              echo "$excluded_files" | sed 's/^/  - /'
-              echo ""
-            fi
             echo "Unified diff (context=5):"
+            # Exclude large generated/data files from the full diff to stay
+            # within the model's input limit. The --name-status above still
+            # lists them. Narrowed to real-data assets and notebook outputs.
             git --no-pager diff --unified=5 "$BASE_SHA" "$HEAD_SHA" \
-              -- . ':!benchmarks/data/real/*.json' ':!benchmarks/data/real/*.csv'
+              -- . ':!benchmarks/data/real/*.json' ':!benchmarks/data/real/*.csv' \
+                   ':!docs/tutorials/*.ipynb'
           } >> "$PROMPT"
 
       - name: Run Codex

.gitignore

@@ -55,6 +55,7 @@ Thumbs.db
 
 # Benchmarks - generated data and results (can be regenerated)
 benchmarks/data/synthetic/*.csv
+benchmarks/data/real/raw/
 benchmarks/results/
 
 # Rust build artifacts

CHANGELOG.md

@@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- **Survey real-data validation** (Phase 9) — 15 cross-validation tests against R's `survey` package using three real federal survey datasets:
+  - **API** (R `survey` package): TSL variance with strata, FPC, subpopulations, covariates, and Fay's BRR replicates
+  - **NHANES** (CDC/NCHS): TSL variance with strata + PSU + nest=TRUE, validating the ACA young adult coverage provision DiD
+  - **RECS 2020** (U.S. EIA): JK1 replicate weight variance with 60 pre-computed replicate columns
+  - ATT, SE, df, and CI match R to machine precision (< 1e-10) where directly comparable; known deviations documented in REGISTRY.md (TWFE SE differs due to unit FE absorption; subpopulation df differs due to strata preservation)
+
 ## [2.8.4] - 2026-04-04
 
 ### Added