Address PR #110 feedback round 10: absorbing-state gap detection and n_post_periods fix

P1: Fix absorbing-state validation to catch 1→0 violations across missing period gaps.
    The old vectorized check only looked at adjacent periods, missing violations like
    D[2]=1, missing [3,4], D[5]=0. Now checks each unit's observed D sequence.

P3: Fix n_post_periods to count periods with actual D=1 observations, matching the
    docstring claim, rather than calendar periods from first treatment.

Also updates methodology registry documentation for both changes.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Author: igerber

diff_diff/trop.py

@@ -912,23 +912,23 @@ def fit(
 
         # Validate D is monotonic non-decreasing per unit (absorbing state)
         # D[t, i] must satisfy: once D=1, it must stay 1 for all subsequent periods
-        # Vectorized check: diff(D, axis=0) should never be negative
-        # Issue 3 fix: Only check transitions where BOTH periods are observed
-        d_diff = np.diff(D, axis=0)
-
-        # Valid transition mask: neither the current nor next period is missing
-        # missing_mask[:-1] = source period missing, missing_mask[1:] = target period missing
-        valid_transition = ~(missing_mask[:-1] | missing_mask[1:])
-
-        # Only flag violations where both periods are observed
-        violations = (d_diff < 0) & valid_transition
-
-        if np.any(violations):
-            # Find which units violate the absorbing state constraint
-            violating_units_mask = np.any(violations, axis=0)
-            violating_unit_ids = [all_units[i] for i in np.where(violating_units_mask)[0]]
+        # Issue 3 fix (round 10): Check each unit's OBSERVED D sequence for monotonicity
+        # This catches 1→0 violations that span missing period gaps
+        # Example: D[2]=1, missing [3,4], D[5]=0 is a real violation even though
+        # adjacent period transitions don't show it (the gap hides the transition)
+        violating_units = []
+        for unit_idx in range(n_units):
+            # Get observed D values for this unit (where not missing)
+            observed_mask = ~missing_mask[:, unit_idx]
+            observed_d = D[observed_mask, unit_idx]
+
+            # Check if observed sequence is monotonically non-decreasing
+            if len(observed_d) > 1 and np.any(np.diff(observed_d) < 0):
+                violating_units.append(all_units[unit_idx])
+
+        if violating_units:
             raise ValueError(
-                f"Treatment indicator is not an absorbing state for units: {violating_unit_ids}. "
+                f"Treatment indicator is not an absorbing state for units: {violating_units}. "
                 f"D[t, unit] must be monotonic non-decreasing (once treated, always treated). "
                 f"If this is event-study style data, convert to absorbing state: "
                 f"D[t, i] = 1 for all t >= first treatment period."
@@ -960,7 +960,9 @@ def fit(
             raise ValueError("Could not infer post-treatment periods from D matrix")
 
         n_pre_periods = first_treat_period
-        n_post_periods = n_periods - first_treat_period
+        # Count periods where D=1 is actually observed (matches docstring)
+        # Per docstring: "Number of post-treatment periods (periods with D=1 observations)"
+        n_post_periods = int(np.sum(np.any(D[first_treat_period:, :] == 1, axis=1)))
 
         if n_pre_periods < 2:
             raise ValueError("Need at least 2 pre-treatment periods")

docs/methodology/REGISTRY.md

@@ -550,7 +550,8 @@ Q(λ) = Σ_{j,s: D_js=0} [τ̂_js^loocv(λ)]²
   - Handling: Raises `ValueError` with list of violating unit IDs and remediation guidance
   - Error message includes: "convert to absorbing state: D[t, i] = 1 for all t >= first treatment period"
   - **Rationale**: Event-style D (0→1→0) silently biases ATT; runtime validation prevents misuse
-  - **Unbalanced panels**: Missing unit-period observations are allowed. Monotonicity validation only checks transitions between observed periods. A unit with D=1 at t=3 and missing data at t=5 is NOT flagged as a violation (the apparent 1→0 transition is due to missing data, not a real violation).
+  - **Unbalanced panels**: Missing unit-period observations are allowed. Monotonicity validation checks each unit's *observed* D sequence for monotonicity, which correctly catches 1→0 violations that span missing period gaps (e.g., D[2]=1, missing [3,4], D[5]=0 is detected as a violation even though the gap hides the transition in adjacent-period checks).
+  - **n_post_periods metadata**: Counts periods where D=1 is actually observed (at least one unit has D=1), not calendar periods from first treatment. In unbalanced panels where treated units are missing in some post-treatment periods, only periods with observed D=1 values are counted.
 - Wrong D specification: if user provides event-style D (only first treatment period),
   the absorbing-state validation will raise ValueError with helpful guidance
 - **LOOCV failure metadata**: When LOOCV fits fail in the Rust backend, the first failed observation coordinates (t, i) are returned to Python for informative warning messages

tests/test_trop.py

@@ -2565,3 +2565,93 @@ def test_infinity_grid_values_with_final_score_computation(self, simple_panel_da
             assert np.isfinite(results.loocv_score), (
                 "LOOCV score should be finite when computed with converted inf values"
             )
+
+    def test_violation_across_missing_gap_caught(self):
+        """Test that 1→0 violations spanning missing periods are caught.
+
+        Issue: If periods [3, 4] are missing and D[2]=1, D[5]=0, this is a
+        real violation that must be detected even though the adjacent
+        period transitions don't show it (the gap hides the transition).
+
+        PR #110 round 10 fix: Check each unit's observed D sequence for
+        monotonicity, not just adjacent periods in the full time grid.
+        """
+        data = []
+
+        # Unit 0: control, complete
+        for t in range(6):
+            data.append({"unit": 0, "period": t, "outcome": 10.0 + t, "treated": 0})
+
+        # Unit 1: VIOLATION across gap
+        # Observed at [0, 1, 2, 5], missing [3, 4]
+        # D[2]=1, D[5]=0 is a real violation spanning the gap
+        for t in [0, 1, 2, 5]:
+            treated = 1 if t == 2 else 0  # Only treated at period 2
+            data.append({"unit": 1, "period": t, "outcome": 10.0 + t, "treated": treated})
+
+        # Unit 2: control, complete
+        for t in range(6):
+            data.append({"unit": 2, "period": t, "outcome": 10.0 + t, "treated": 0})
+
+        df = pd.DataFrame(data)
+        trop_est = TROP(
+            lambda_time_grid=[0.0],
+            lambda_unit_grid=[0.0],
+            lambda_nn_grid=[0.0],
+            n_bootstrap=5,
+        )
+
+        with pytest.raises(ValueError, match="absorbing state"):
+            trop_est.fit(
+                df,
+                outcome="outcome",
+                treatment="treated",
+                unit="unit",
+                time="period",
+            )
+
+    def test_n_post_periods_counts_observed_treatment(self):
+        """Test n_post_periods counts periods with actual D=1 observations.
+
+        Per docstring: "Number of post-treatment periods (periods with D=1 observations)"
+
+        This tests that n_post_periods reflects periods where treatment is
+        actually observed, not just calendar periods from first treatment.
+        """
+        data = []
+
+        # Create panel where period 5 exists but has no D=1 observations
+        # (all treated units are missing at period 5)
+        for unit in range(3):
+            for period in range(6):
+                # Units 1, 2 are treated from period 3, but missing at period 5
+                if unit in [1, 2] and period == 5:
+                    continue  # Skip - creates unbalanced panel
+                treated = 1 if (unit in [1, 2] and period >= 3) else 0
+                data.append({
+                    "unit": unit,
+                    "period": period,
+                    "outcome": 10.0 + period,
+                    "treated": treated,
+                })
+
+        df = pd.DataFrame(data)
+        trop_est = TROP(
+            lambda_time_grid=[0.0],
+            lambda_unit_grid=[0.0],
+            lambda_nn_grid=[0.0],
+            n_bootstrap=5,
+            seed=42,
+        )
+        results = trop_est.fit(
+            df,
+            outcome="outcome",
+            treatment="treated",
+            unit="unit",
+            time="period",
+        )
+
+        # Periods with D=1 observations: 3, 4 (not 5 - missing for treated units)
+        assert results.n_post_periods == 2, (
+            f"Expected 2 post-periods with D=1, got {results.n_post_periods}"
+        )