Address CI review; add memory tracking and BRFSS allocator attribution

Addresses the four CI review findings:

- BRR -> JK1 rename. generate_survey_did_data(include_replicate_weights=
  True) emits JK1 delete-one-PSU weights per prep.py:1248; Scenario 2 was
  labeling them as BRR, which uses a different variance formula. Fixed
  script, phase label, scenario doc data-shape text, and example code
  snippet.
- Exit-code propagation. run_scenario now records a module-level
  failure flag; an atexit handler os._exit(1)s if any phase recorded
  ok=False. run_all.py's subprocess return-code check now reliably
  surfaces phase failures. Verified with a forced-failure harness test.
- Path references. bench_shared.py and run_all.py docstrings plus
  performance-plan.md prose normalized to
  benchmarks/speed_review/baselines/.
- Contributor README. "Commit HTMLs" instruction removed; flame HTMLs
  are gitignored and regenerated per run.

Adds memory measurement:

- psutil background RSS sampler (10ms) in run_scenario writes a memory
  field to every scenario JSON: start, peak, growth-during-run. Zero
  timing impact (background thread, single-syscall samples).
- mem_profile_brfss.py - standalone tracemalloc allocator attribution
  for the BRFSS-1M scenario. Separate from the timing harness so its
  2-5x overhead does not contaminate wall-clock baselines.

Memory findings extend the optimization priority list without changing
the #1 recommendation. Headline insight: BRFSS aggregate_survey at 1M
rows grows only 23 MB of working memory (vs 46 MB input), and
tracemalloc's net-retained allocation is 0.6 MB. The 24-second cost is
pure CPU - confirms the precompute-scaffolding fix is low-risk and fits
in any deployment target including 512 MB Lambda.

Secondary finding: staggered CS chain allocates 252-322 MB at 1,500
units (peak RSS 486-589 MB). Fine for workstations, tight for Lambda-
tier deployments. Flagged as a lower-priority follow-up.

Still measurement only. No changes under diff_diff/ or rust/.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: igerber

benchmarks/speed_review/README.md

@@ -19,22 +19,31 @@ at data shapes anchored to applied-econ conventions.
 ```
 benchmarks/speed_review/
 ├── README.md                           # this file
-├── bench_shared.py                     # timing + pyinstrument harness
+├── bench_shared.py                     # timing + pyinstrument + RSS harness
 ├── run_all.py                          # orchestrator (both backends)
 ├── bench_campaign_staggered.py         # Scenario 1: CS + 8-step chain
 ├── bench_brand_awareness_survey.py     # Scenario 2: DiD + SurveyDesign
 ├── bench_brfss_panel.py                # Scenario 3: aggregate_survey -> CS
 ├── bench_geo_few_markets.py            # Scenario 4: SDiD + jackknife
 ├── bench_reversible_dcdh.py            # Scenario 5: dCDH L_max + TSL
 ├── bench_dose_response.py              # Scenario 6: ContinuousDiD splines
+├── mem_profile_brfss.py                # tracemalloc allocator attribution
+│                                       #   for BRFSS-1M (standalone)
 ├── bench_callaway.py                   # pre-existing CS scaling sweep
 ├── baseline_results.json               # pre-existing CS baseline
 └── baselines/                          # this effort's output
-    ├── <scenario>_<backend>.json       # phase-level wall-clock (committed)
+    ├── <scenario>_<backend>.json       # phase-level wall-clock + peak RSS
+    ├── mem_profile_brfss_large_<backend>.txt   # tracemalloc top-N sites
     └── profiles/                       # flame HTMLs (gitignored)
         └── <scenario>_<backend>.html   # pyinstrument flame output
 ```
 
+Each JSON baseline records both timing (per-phase wall-clock) and memory
+(start/peak/growth from a psutil background sampler at 10 ms). The
+`mem_profile_brfss.py` script does a separate tracemalloc pass on the
+BRFSS-1M scenario - this is kept out of the main timing harness because
+tracemalloc has 2-5x overhead and would contaminate wall-clock baselines.
+
 **Note on profile HTMLs.** pyinstrument flames are ~500KB-1.2MB each and are
 regenerated on every run; they live under `baselines/profiles/` which is
 gitignored. The key hotspots identified from them are already captured in
@@ -77,6 +86,7 @@ the findings doc is the decision output.
 2. Add `bench_<name>.py` following the existing scripts: build data, define
    `phases` as a list of `(label, callable)` tuples, call `run_scenario`.
 3. Register it in `run_all.py`'s `SCRIPTS` dict.
-4. Run under both backends, commit the refreshed `baselines/*.json` and the
-   corresponding `baselines/profiles/*.html`.
+4. Run under both backends and commit the refreshed `baselines/*.json`.
+   The `baselines/profiles/*.html` flame HTMLs are gitignored and
+   regenerated per run - do not commit them.
 5. Add a per-scenario finding paragraph to `docs/performance-plan.md`.

benchmarks/speed_review/baselines/brand_awareness_survey_large_python.json

@@ -2,40 +2,47 @@
   "scenario": "brand_awareness_survey_large",
   "backend": "python",
   "has_rust_backend": false,
-  "total_seconds": 0.7940070000000001,
+  "total_seconds": 0.9061127080000002,
+  "memory": {
+    "available": true,
+    "start_mb": 189.33,
+    "peak_mb": 335.59,
+    "growth_mb": 146.27,
+    "sampler_interval_s": 0.01
+  },
   "phases": {
     "1_naive_fit_no_survey_design": {
-      "seconds": 0.013499665999999966,
+      "seconds": 0.013970540999999947,
       "ok": true,
       "error": null
     },
     "2_tsl_strata_psu_fpc": {
-      "seconds": 0.03187458300000001,
+      "seconds": 0.03233287500000004,
       "ok": true,
       "error": null
     },
-    "3_replicate_weights_brr": {
-      "seconds": 0.3442796670000001,
+    "3_replicate_weights_jk1": {
+      "seconds": 0.44611704099999994,
       "ok": true,
       "error": null
     },
     "4_multi_outcome_loop_3_metrics": {
-      "seconds": 0.19682533299999982,
+      "seconds": 0.21938504199999986,
       "ok": true,
       "error": null
     },
     "5_check_parallel_trends": {
-      "seconds": 0.030179500000000026,
+      "seconds": 0.04051395800000002,
       "ok": true,
       "error": null
     },
     "6_placebo_refit_pre_period": {
-      "seconds": 0.043751333999999975,
+      "seconds": 0.016386375000000175,
       "ok": true,
       "error": null
     },
     "7_event_study_plus_honest_did": {
-      "seconds": 0.13358487500000016,
+      "seconds": 0.13737600000000016,
       "ok": true,
       "error": null
     }

benchmarks/speed_review/baselines/brand_awareness_survey_large_rust.json

@@ -2,40 +2,47 @@
   "scenario": "brand_awareness_survey_large",
   "backend": "rust",
   "has_rust_backend": true,
-  "total_seconds": 0.828119375,
+  "total_seconds": 0.8791467499999999,
+  "memory": {
+    "available": true,
+    "start_mb": 187.97,
+    "peak_mb": 315.2,
+    "growth_mb": 127.23,
+    "sampler_interval_s": 0.01
+  },
   "phases": {
     "1_naive_fit_no_survey_design": {
-      "seconds": 0.014049749999999861,
+      "seconds": 0.013859375000000007,
       "ok": true,
       "error": null
     },
     "2_tsl_strata_psu_fpc": {
-      "seconds": 0.029422499999999907,
+      "seconds": 0.02124400000000004,
       "ok": true,
       "error": null
     },
-    "3_replicate_weights_brr": {
-      "seconds": 0.36754912500000003,
+    "3_replicate_weights_jk1": {
+      "seconds": 0.42970375000000005,
       "ok": true,
       "error": null
     },
     "4_multi_outcome_loop_3_metrics": {
-      "seconds": 0.16490987499999998,
+      "seconds": 0.2112943330000001,
       "ok": true,
       "error": null
     },
     "5_check_parallel_trends": {
-      "seconds": 0.03375229199999996,
+      "seconds": 0.038379208000000276,
       "ok": true,
       "error": null
     },
     "6_placebo_refit_pre_period": {
-      "seconds": 0.06475750000000025,
+      "seconds": 0.025571082999999994,
       "ok": true,
       "error": null
     },
     "7_event_study_plus_honest_did": {
-      "seconds": 0.15367104200000004,
+      "seconds": 0.1390828329999998,
       "ok": true,
       "error": null
     }

benchmarks/speed_review/baselines/brand_awareness_survey_medium_python.json

@@ -2,40 +2,47 @@
   "scenario": "brand_awareness_survey_medium",
   "backend": "python",
   "has_rust_backend": false,
-  "total_seconds": 0.48956791599999994,
+  "total_seconds": 0.560188,
+  "memory": {
+    "available": true,
+    "start_mb": 131.69,
+    "peak_mb": 187.8,
+    "growth_mb": 56.11,
+    "sampler_interval_s": 0.01
+  },
   "phases": {
     "1_naive_fit_no_survey_design": {
-      "seconds": 0.01289191699999992,
+      "seconds": 0.011834042000000045,
       "ok": true,
       "error": null
     },
     "2_tsl_strata_psu_fpc": {
-      "seconds": 0.035409875000000035,
+      "seconds": 0.03354012500000003,
       "ok": true,
       "error": null
     },
-    "3_replicate_weights_brr": {
-      "seconds": 0.12633833299999997,
+    "3_replicate_weights_jk1": {
+      "seconds": 0.21381758399999995,
       "ok": true,
       "error": null
     },
     "4_multi_outcome_loop_3_metrics": {
-      "seconds": 0.17774295900000003,
+      "seconds": 0.13717983300000003,
       "ok": true,
       "error": null
     },
     "5_check_parallel_trends": {
-      "seconds": 0.018629792000000034,
+      "seconds": 0.018324165999999975,
       "ok": true,
       "error": null
     },
     "6_placebo_refit_pre_period": {
-      "seconds": 0.0519646250000001,
+      "seconds": 0.058137000000000105,
       "ok": true,
       "error": null
     },
     "7_event_study_plus_honest_did": {
-      "seconds": 0.06657341699999986,
+      "seconds": 0.08734375000000005,
       "ok": true,
       "error": null
     }

benchmarks/speed_review/baselines/brand_awareness_survey_medium_rust.json

@@ -2,40 +2,47 @@
   "scenario": "brand_awareness_survey_medium",
   "backend": "rust",
   "has_rust_backend": true,
-  "total_seconds": 0.535454792,
+  "total_seconds": 0.5398647089999999,
+  "memory": {
+    "available": true,
+    "start_mb": 133.16,
+    "peak_mb": 185.38,
+    "growth_mb": 52.22,
+    "sampler_interval_s": 0.01
+  },
   "phases": {
     "1_naive_fit_no_survey_design": {
-      "seconds": 0.011897708999999979,
+      "seconds": 0.011500667000000075,
       "ok": true,
       "error": null
     },
     "2_tsl_strata_psu_fpc": {
-      "seconds": 0.03526237499999996,
+      "seconds": 0.03384820799999999,
       "ok": true,
       "error": null
     },
-    "3_replicate_weights_brr": {
-      "seconds": 0.185435083,
+    "3_replicate_weights_jk1": {
+      "seconds": 0.191542875,
       "ok": true,
       "error": null
     },
     "4_multi_outcome_loop_3_metrics": {
-      "seconds": 0.14044966699999994,
+      "seconds": 0.105974083,
       "ok": true,
       "error": null
     },
     "5_check_parallel_trends": {
-      "seconds": 0.019051875000000162,
+      "seconds": 0.02876208299999994,
       "ok": true,
       "error": null
     },
     "6_placebo_refit_pre_period": {
-      "seconds": 0.05337804200000007,
+      "seconds": 0.06280441700000017,
       "ok": true,
       "error": null
     },
     "7_event_study_plus_honest_did": {
-      "seconds": 0.08997387500000009,
+      "seconds": 0.10540583399999992,
       "ok": true,
       "error": null
     }

benchmarks/speed_review/baselines/brand_awareness_survey_small_python.json

@@ -2,40 +2,47 @@
   "scenario": "brand_awareness_survey_small",
   "backend": "python",
   "has_rust_backend": false,
-  "total_seconds": 0.15087129199999993,
+  "total_seconds": 0.15974079200000002,
+  "memory": {
+    "available": true,
+    "start_mb": 115.44,
+    "peak_mb": 125.66,
+    "growth_mb": 10.22,
+    "sampler_interval_s": 0.01
+  },
   "phases": {
     "1_naive_fit_no_survey_design": {
-      "seconds": 0.0017902499999999932,
+      "seconds": 0.0016714159999999811,
       "ok": true,
       "error": null
     },
     "2_tsl_strata_psu_fpc": {
-      "seconds": 0.00610949999999999,
+      "seconds": 0.0061952499999999855,
       "ok": true,
       "error": null
     },
-    "3_replicate_weights_brr": {
-      "seconds": 0.02120725000000001,
+    "3_replicate_weights_jk1": {
+      "seconds": 0.018200666000000032,
       "ok": true,
       "error": null
     },
     "4_multi_outcome_loop_3_metrics": {
-      "seconds": 0.011621500000000062,
+      "seconds": 0.02470079199999997,
       "ok": true,
       "error": null
     },
     "5_check_parallel_trends": {
-      "seconds": 0.001833375000000026,
+      "seconds": 0.008862999999999954,
       "ok": true,
       "error": null
     },
     "6_placebo_refit_pre_period": {
-      "seconds": 0.027076792000000016,
+      "seconds": 0.024017708000000026,
       "ok": true,
       "error": null
     },
     "7_event_study_plus_honest_did": {
-      "seconds": 0.081212583,
+      "seconds": 0.07607645800000007,
       "ok": true,
       "error": null
     }

benchmarks/speed_review/baselines/brand_awareness_survey_small_rust.json

@@ -2,40 +2,47 @@
   "scenario": "brand_awareness_survey_small",
   "backend": "rust",
   "has_rust_backend": true,
-  "total_seconds": 0.200881125,
+  "total_seconds": 0.19896133300000007,
+  "memory": {
+    "available": true,
+    "start_mb": 116.34,
+    "peak_mb": 129.73,
+    "growth_mb": 13.39,
+    "sampler_interval_s": 0.01
+  },
   "phases": {
     "1_naive_fit_no_survey_design": {
-      "seconds": 0.0018462080000000158,
+      "seconds": 0.0019397500000000178,
       "ok": true,
       "error": null
     },
     "2_tsl_strata_psu_fpc": {
-      "seconds": 0.005704333000000061,
+      "seconds": 0.005711999999999939,
       "ok": true,
       "error": null
     },
-    "3_replicate_weights_brr": {
-      "seconds": 0.015561500000000006,
+    "3_replicate_weights_jk1": {
+      "seconds": 0.011531958999999925,
       "ok": true,
       "error": null
     },
     "4_multi_outcome_loop_3_metrics": {
-      "seconds": 0.05937758399999993,
+      "seconds": 0.06204845800000003,
       "ok": true,
       "error": null
     },
     "5_check_parallel_trends": {
-      "seconds": 0.00939004099999996,
+      "seconds": 0.00982324999999995,
       "ok": true,
       "error": null
     },
     "6_placebo_refit_pre_period": {
-      "seconds": 0.025794415999999987,
+      "seconds": 0.024675957999999998,
       "ok": true,
       "error": null
     },
     "7_event_study_plus_honest_did": {
-      "seconds": 0.08319054199999998,
+      "seconds": 0.08321629100000005,
       "ok": true,
       "error": null
     }