Skip to content

Commit 079fef0

Browse files
igerberclaude
andcommitted
Extend dCDH heterogeneity SE to cell-period allocator
Lifts the NotImplementedError gate for heterogeneity= + within-group- varying PSU/strata under survey designs. The heterogeneity WLS coefficient IF psi_g is now attributed in full to the (g, out_idx) post-period cell and expanded to observation level as psi_i = psi_g * (w_i / W_{g, out_idx}) — the DID_l single-cell convention shipped in PR #323. Under PSU=group the per-obs distribution differs from the legacy psi_i = psi_g * (w_i / W_g) expansion, but the PSU-level aggregate telescopes to psi_g in both paths, so Binder TSL variance and Rao-Wu replicate variance are byte-identical under PSU=group; under within-group-varying PSU, mass lands in the post-period PSU of the transition. Tests: flipped the gating test to assert all five inference fields finite; added PSU-level byte-identity unit test constructing both psi_obs arrays and asserting compute_survey_if_variance agreement within ULP; added nest=True + varying-strata + heterogeneity smoke test (newly-unblocked regime); added multi-horizon smoke test; added slow-tier MC null-coverage test (500 reps, within-group- varying PSU, empirical 95% coverage inside [0.925, 0.975]). n_bootstrap > 0 + within-group-varying PSU remains gated (follow-up PR). Updated REGISTRY.md heterogeneity Note + Survey IF expansion Note scope-limitations paragraph; updated _compute_heterogeneity_test docstring + the stale legacy-allocator comment in _survey_se_from_group_if; added CHANGELOG Changed entry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 61aea00 commit 079fef0

5 files changed

Lines changed: 404 additions & 69 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
99

1010
### Changed
1111
- Add Zenodo DOI badge to README; upgrade the BibTeX citation block with the concept DOI (`10.5281/zenodo.19646175`) and list author as Isaac Gerber (matching `CITATION.cff`). Add `doi:` and `identifiers:` entries (concept + versioned) to `CITATION.cff`. DOI was minted by Zenodo when v3.1.3 was released.
12+
- **`ChaisemartinDHaultfoeuille` heterogeneity + within-group-varying PSU/strata now supported** - `fit(heterogeneity=..., survey_design=...)` no longer raises `NotImplementedError` when the resolved design's PSU or strata vary across the cells of a group. The heterogeneity WLS coefficient IF is now expanded to observation level via the cell-period allocator (`ψ_i = ψ_g * (w_i / W_{g, out_idx})` on the post-period cell), consistent with the DID_l post-period single-cell convention shipped in v3.1.x. Under PSU=group the PSU-level Binder TSL variance and Rao-Wu replicate variance are byte-identical to the previous release (the PSU-level aggregate telescopes to `ψ_g` in both expansions); under within-group-varying PSU, mass lands in the post-period PSU of the transition. `n_bootstrap > 0` combined with within-group-varying PSU remains gated with `NotImplementedError` — the PSU-level Hall-Mammen wild bootstrap still uses the legacy group-level PSU map and will be extended in a follow-up PR.
1213

1314
## [3.1.3] - 2026-04-18
1415

diff_diff/chaisemartin_dhaultfoeuille.py

Lines changed: 72 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -665,10 +665,7 @@ def fit(
665665
clustered bootstrap. **Out-of-scope combinations raise
666666
``NotImplementedError``**: (a) replicate weights with
667667
``n_bootstrap > 0`` (replicate variance is closed-form;
668-
bootstrap would double-count variance); (b)
669-
``heterogeneity=`` with PSU/strata that vary within group
670-
(heterogeneity WLS still uses the legacy group-level IF
671-
expansion; follow-up PR extends it); (c) ``n_bootstrap >
668+
bootstrap would double-count variance); (b) ``n_bootstrap >
672669
0`` with PSU that varies within group (PSU-level bootstrap
673670
still uses the legacy group-level PSU map; follow-up PR
674671
extends it). See REGISTRY.md
@@ -828,39 +825,25 @@ def fit(
828825

829826
# Cell-period IF allocator contract: strata and PSU must be
830827
# constant within each (g, t) cell, a strict relaxation of
831-
# the previous within-group constancy rule. Two out-of-scope
832-
# combinations are gated with NotImplementedError until the
833-
# corresponding follow-up PRs extend them:
834-
# - heterogeneity= + within-group-varying PSU/strata
835-
# (PR 3: cell-period allocator for the WLS psi_obs)
828+
# the previous within-group constancy rule. One out-of-scope
829+
# combination remains gated with NotImplementedError until
830+
# the corresponding follow-up PR extends it:
836831
# - n_bootstrap > 0 + within-group-varying PSU
837832
# (PR 4: cell-level Hall-Mammen wild bootstrap)
838-
strata_varies, psu_varies = _strata_psu_vary_within_group(
833+
_, psu_varies = _strata_psu_vary_within_group(
839834
resolved_survey, data, group, survey_weights,
840835
)
841-
if strata_varies or psu_varies:
842-
if heterogeneity is not None:
843-
raise NotImplementedError(
844-
"heterogeneity= is not supported under a survey "
845-
"design whose PSU or strata vary within group. "
846-
"The heterogeneity WLS path uses the legacy "
847-
"group-level IF expansion and will be extended "
848-
"to the cell-period allocator in a follow-up "
849-
"PR. For now, either (a) set heterogeneity=None, "
850-
"or (b) collapse PSU/strata to be constant "
851-
"within each group."
852-
)
853-
if self.n_bootstrap > 0:
854-
raise NotImplementedError(
855-
"n_bootstrap > 0 is not supported under a "
856-
"survey design whose PSU varies within group. "
857-
"The PSU-level Hall-Mammen wild bootstrap uses "
858-
"the legacy group-level PSU map and will be "
859-
"extended to cell-level PSU in a follow-up PR. "
860-
"For now, use n_bootstrap=0 (analytical TSL "
861-
"variance, which fully supports within-group-"
862-
"varying PSU via the cell-period allocator)."
863-
)
836+
if psu_varies and self.n_bootstrap > 0:
837+
raise NotImplementedError(
838+
"n_bootstrap > 0 is not supported under a "
839+
"survey design whose PSU varies within group. "
840+
"The PSU-level Hall-Mammen wild bootstrap uses "
841+
"the legacy group-level PSU map and will be "
842+
"extended to cell-level PSU in a follow-up PR. "
843+
"For now, use n_bootstrap=0 (analytical TSL "
844+
"variance, which fully supports within-group-"
845+
"varying PSU via the cell-period allocator)."
846+
)
864847
_validate_cell_constant_strata_psu(
865848
resolved_survey, data, group, time, survey_weights,
866849
)
@@ -3734,15 +3717,29 @@ def _compute_heterogeneity_test(
37343717
Required when ``obs_survey_info`` is supplied.
37353718
obs_survey_info : dict, optional
37363719
Observation-level survey info with keys ``group_ids`` (raw per-row
3737-
group labels), ``weights`` (per-row survey weights), and ``resolved``
3738-
(ResolvedSurveyDesign). When provided, the regression uses WLS with
3739-
per-group weights W_g = sum of obs survey weights. SE is computed
3740-
via Binder TSL IF expansion through ``compute_survey_if_variance``
3741-
by default; under a replicate-weight design (BRR/Fay/JK1/JKn/SDR),
3742-
dispatches to ``compute_replicate_if_variance`` for Rao-Wu-style
3743-
variance. The effective df for t-critical values follows the
3744-
site-level ``min(df_s, n_valid_het - 1)`` rule and the helper
3745-
mutates ``replicate_n_valid_list`` so the final
3720+
group labels), ``time_ids`` (raw per-row period labels),
3721+
``weights`` (per-row survey weights), ``resolved``
3722+
(ResolvedSurveyDesign), and ``periods`` (sorted canonical period
3723+
array matching ``Y_mat``'s column order). When provided, the
3724+
regression uses WLS with per-group weights
3725+
``W_g = sum of obs survey weights in group g``. The group-level
3726+
WLS coefficient IF is
3727+
``ψ_g = inv(X'WX)[1,:] @ x_g * W_g * r_g``. Under the cell-period
3728+
allocator, ``ψ_g`` is attributed in full to the ``(g, out_idx)``
3729+
post-period cell and expanded to observation level as
3730+
``ψ_i = ψ_g * (w_i / W_{g, out_idx})`` for obs in that cell,
3731+
zero elsewhere. Under PSU=group this produces the same PSU-level
3732+
aggregate as the legacy ``ψ_i = ψ_g * (w_i / W_g)`` expansion,
3733+
so Binder TSL variance (and Rao-Wu replicate variance) are
3734+
byte-identical to the pre-cell-period release; under
3735+
within-group-varying PSU, mass lands in the post-period PSU of
3736+
the transition. SE is computed through
3737+
``compute_survey_if_variance`` by default; under a
3738+
replicate-weight design (BRR/Fay/JK1/JKn/SDR), dispatches to
3739+
``compute_replicate_if_variance`` for Rao-Wu-style variance. The
3740+
effective df for t-critical values follows the site-level
3741+
``min(df_s, n_valid_het - 1)`` rule and the helper mutates
3742+
``replicate_n_valid_list`` so the final
37463743
``_effective_df_survey(...)`` sees this site's n_valid.
37473744
replicate_n_valid_list : list[int], optional
37483745
Shared accumulator for replicate-weight ``n_valid`` counts across
@@ -3931,15 +3928,35 @@ def _compute_heterogeneity_test(
39313928
XtWX_inv = np.linalg.pinv(XtWX)
39323929
psi_g = (XtWX_inv[1, :] @ design.T) * W_elig * r_g # (n_eligible,)
39333930

3934-
# Expand to obs level: ψ_i = ψ_g * (w_i / W_g) for i in group g.
3935-
psi_obs = np.zeros(len(obs_w_raw))
3931+
# Cell-period allocator: attribute ψ_g in full to the
3932+
# (g, out_idx) post-period cell (DID_l single-cell convention,
3933+
# see REGISTRY.md ChaisemartinDHaultfoeuille survey IF
3934+
# expansion Note). Observation-level expansion:
3935+
# ψ_i = ψ_g * (w_i / W_{g, out_idx})
3936+
# for obs in (g, out_idx), zero elsewhere. Under PSU=group the
3937+
# per-observation distribution differs from the legacy
3938+
# ψ_i = ψ_g * (w_i / W_g) path, but the PSU-level aggregate
3939+
# telescopes to the same ψ_g — so compute_survey_if_variance
3940+
# and compute_replicate_if_variance (both aggregate to PSU
3941+
# first) produce byte-identical variance. Under within-group-
3942+
# varying PSU, mass lands in the post-period PSU of the
3943+
# transition, which is what Binder TSL needs.
3944+
obs_tids = np.asarray(obs_survey_info["time_ids"])
3945+
periods_arr = np.asarray(obs_survey_info["periods"])
3946+
psi_obs = np.zeros(len(obs_w_raw), dtype=np.float64)
39363947
for e_idx, g_idx in enumerate(eligible):
39373948
gid = gid_list[g_idx]
3938-
mask_g = (obs_gids_raw == gid) & valid
3939-
w_sum_g = obs_w_raw[mask_g].sum()
3940-
if w_sum_g > 0:
3941-
psi_obs[mask_g] = psi_g[e_idx] * (
3942-
obs_w_raw[mask_g] / w_sum_g
3949+
out_idx = first_switch_idx[g_idx] - 1 + l_h
3950+
t_val_out = periods_arr[out_idx]
3951+
mask_cell = (
3952+
(obs_gids_raw == gid)
3953+
& (obs_tids == t_val_out)
3954+
& valid
3955+
)
3956+
w_cell = obs_w_raw[mask_cell].sum()
3957+
if w_cell > 0:
3958+
psi_obs[mask_cell] = psi_g[e_idx] * (
3959+
obs_w_raw[mask_cell] / w_cell
39433960
)
39443961

39453962
# Dispatch: replicate-weight variance (BRR/Fay/JK1/JKn/SDR)
@@ -5766,10 +5783,12 @@ def _survey_se_from_group_if(
57665783
else:
57675784
# Legacy group-level allocator (no per-period attribution
57685785
# provided, or time/period info unavailable). Preserved for
5769-
# paths that haven't threaded per-period attribution through
5770-
# yet (e.g., the heterogeneity psi_obs construction in
5771-
# _compute_heterogeneity_test — gated to within-group-constant
5772-
# PSU in Stage 2 per PR 2 scope).
5786+
# defensive fallback and for unit tests that exercise the
5787+
# legacy allocator. No current caller in fit() uses this
5788+
# branch — ATT / joiners / leavers / placebos all thread
5789+
# U_centered_per_period, and heterogeneity (as of PR 3)
5790+
# constructs its own cell-period psi_obs and calls
5791+
# compute_survey_if_variance directly.
57735792
group_to_u = {gid: U_centered[idx] for idx, gid in enumerate(eligible_groups)}
57745793
u_obs_eff = np.array([group_to_u.get(gid, 0.0) for gid in gids_eff])
57755794
unique_gids, inverse = np.unique(gids_eff, return_inverse=True)

0 commit comments

Comments
 (0)