Merge pull request #372 from igerber/docs-refresh

Docs refresh: trim README to a 188-line landing page; redirect contributor conventions

Author: igerber

.claude/commands/dev-checklists.md

@@ -152,8 +152,11 @@ Final checklist before approving a PR:
 
 3. **Documentation Sync**:
    - [ ] Docstrings updated for all changed signatures
-   - [ ] README updated if user-facing behavior changes
-   - [ ] REGISTRY.md updated if methodology edge cases change
+   - [ ] `diff_diff/guides/llms.txt` updated if a new estimator/feature appears in the public API (this is the AI-agent contract; it cascades to RTD)
+   - [ ] `docs/api/*.rst` updated for new modules / signatures
+   - [ ] `docs/references.rst` updated if a new scholarly source is cited
+   - [ ] `README.md` updated ONLY if (a) new estimator catalog one-liner, (b) hero/badges/tagline change, or (c) top-level capability paragraph (Diagnostics & Sensitivity, Survey Support). Do NOT add usage examples, parameter tables, or per-estimator sections.
+   - [ ] `REGISTRY.md` updated if methodology edge cases change
 
 ## Quick Reference: Common Patterns to Check
 

.claude/commands/docs-check.md

@@ -12,35 +12,54 @@ Verify that documentation is complete and includes appropriate scholarly referen
 The user may provide an optional argument: `$ARGUMENTS`
 
 - If empty or "all": Run all checks (including map validation)
-- If "readme": Check README.md sections only
-- If "refs" or "references": Check scholarly references only
+- If "readme": Check the README catalog one-liner only
+- If "refs" or "references": Check scholarly references in `docs/references.rst` only
 - If "api": Check API documentation (RST files) only
 - If "tutorials": Check tutorial coverage only
-- If "map": Validate docs/doc-deps.yaml integrity only
+- If "map": Validate `docs/doc-deps.yaml` integrity only
+
+## Documentation surface map (post 2026-04 docs refresh)
+
+The README is a **landing page**, not the documentation. Each estimator/feature has documentation across multiple authoritative surfaces:
+
+- **`diff_diff/guides/llms.txt`** - AI-agent contract (one-line catalog entry per estimator with paper citation + RTD link). Source of truth that mirrors into RTD via `html_extra_path` and into the wheel via `get_llm_guide()`.
+- **`docs/api/*.rst`** - Sphinx API reference (autoclass).
+- **`docs/references.rst`** - Bibliography (one entry per scholarly source, organized by sub-section).
+- **`docs/tutorials/*.ipynb`** - Hands-on examples.
+- **`README.md`** - **One-line catalog entry only** under `## Estimators` (or `## Diagnostics & Sensitivity` for diagnostic-class features). No usage examples, no parameter tables, no per-estimator section.
 
 ## Estimators and Required Documentation
 
 The following estimators/features MUST have documentation:
 
-### Core Estimators (require README section + API docs + references)
-
-| Estimator | README Section | API RST | Reference Category |
-|-----------|----------------|---------|-------------------|
-| DifferenceInDifferences | "Basic Difference-in-Differences" | estimators.rst | "Difference-in-Differences" |
-| TwoWayFixedEffects | "Two-Way Fixed Effects" | estimators.rst | "Two-Way Fixed Effects" |
-| MultiPeriodDiD | "Multi-Period" | estimators.rst | "Multi-Period and Staggered" |
-| SyntheticDiD | "Synthetic DiD" or "Synthetic Difference" | estimators.rst | "Synthetic Difference-in-Differences" |
-| CallawaySantAnna | "Callaway" or "Staggered" | staggered.rst | "Multi-Period and Staggered" |
-| SunAbraham | "Sun" and "Abraham" | staggered.rst | "Multi-Period and Staggered" |
-| TripleDifference | "Triple Diff" or "DDD" | triple_diff.rst | "Triple Difference" |
-| TROP | "TROP" or "Triply Robust" | trop.rst | "Triply Robust Panel" |
-| HonestDiD | "Honest DiD" or "sensitivity" | honest_did.rst | "Honest DiD" |
-| BaconDecomposition | "Bacon" or "decomposition" | estimators.rst | "Multi-Period and Staggered" |
-
-### Supporting Features (require README mention + API docs)
-
-| Feature | README Mention | API RST |
-|---------|----------------|---------|
+### Core Estimators (require llms.txt entry + README catalog line + API docs + references)
+
+| Estimator | llms.txt entry | README catalog | API RST | Reference Category |
+|-----------|---------------|----------------|---------|-------------------|
+| DifferenceInDifferences | "DifferenceInDifferences" | "DifferenceInDifferences" | estimators.rst | "Difference-in-Differences" |
+| TwoWayFixedEffects | "TwoWayFixedEffects" | "TwoWayFixedEffects" | estimators.rst | "Two-Way Fixed Effects" |
+| MultiPeriodDiD | "MultiPeriodDiD" | "MultiPeriodDiD" | estimators.rst | "Multi-Period and Staggered" |
+| SyntheticDiD | "SyntheticDiD" | "SyntheticDiD" | estimators.rst | "Synthetic Difference-in-Differences" |
+| CallawaySantAnna | "CallawaySantAnna" | "CallawaySantAnna" | staggered.rst | "Multi-Period and Staggered" |
+| SunAbraham | "SunAbraham" | "SunAbraham" | staggered.rst | "Multi-Period and Staggered" |
+| ImputationDiD | "ImputationDiD" | "ImputationDiD" | imputation.rst | "Multi-Period and Staggered" |
+| TwoStageDiD | "TwoStageDiD" | "TwoStageDiD" | two_stage.rst | "Multi-Period and Staggered" |
+| ChaisemartinDHaultfoeuille | "ChaisemartinDHaultfoeuille" | "ChaisemartinDHaultfoeuille" | chaisemartin_dhaultfoeuille.rst | "Multi-Period and Staggered" |
+| EfficientDiD | "EfficientDiD" | "EfficientDiD" | efficient_did.rst | "Multi-Period and Staggered" |
+| StackedDiD | "StackedDiD" | "StackedDiD" | stacked_did.rst | "Multi-Period and Staggered" |
+| ContinuousDiD | "ContinuousDiD" | "ContinuousDiD" | continuous_did.rst | "Multi-Period and Staggered" |
+| HeterogeneousAdoptionDiD | "HeterogeneousAdoptionDiD" | "HeterogeneousAdoptionDiD" | had.rst | "Heterogeneous Adoption (No-Untreated Designs)" |
+| TripleDifference | "TripleDifference" | "TripleDifference" | triple_diff.rst | "Triple Difference" |
+| StaggeredTripleDifference | "StaggeredTripleDifference" | "StaggeredTripleDifference" | staggered.rst | "Triple Difference" |
+| WooldridgeDiD | "WooldridgeDiD" | "WooldridgeDiD" | wooldridge_etwfe.rst | "Multi-Period and Staggered" |
+| TROP | "TROP" | "TROP" | trop.rst | "Triply Robust Panel" |
+| HonestDiD | n/a (in `## Diagnostics`) | n/a (in `## Diagnostics`) | honest_did.rst | "Honest DiD" |
+| BaconDecomposition | "BaconDecomposition" | "BaconDecomposition" | bacon.rst | "Multi-Period and Staggered" |
+
+### Supporting Features (require llms.txt mention + API docs; README mention only if landing-page-relevant)
+
+| Feature | llms.txt Mention | API RST |
+|---------|------------------|---------|
 | Wild bootstrap | "wild" and "bootstrap" | utils.rst |
 | Cluster-robust SE | "cluster" | utils.rst |
 | Parallel trends | "parallel trends" | utils.rst |
@@ -50,7 +69,7 @@ The following estimators/features MUST have documentation:
 
 ## Required Scholarly References
 
-Each estimator category MUST have at least one scholarly reference in README.md:
+Each estimator category MUST have at least one scholarly reference in `docs/references.rst`:
 
 ### Reference Requirements
 
@@ -95,33 +114,61 @@ Goodman-Bacon Decomposition:
 
 Determine which checks to run based on `$ARGUMENTS`.
 
-### 2. README Section Check
+### 2. llms.txt + README Catalog Check
+
+For each estimator/diagnostic in the table above:
 
-For each estimator in the table above:
-1. Read README.md
-2. Search for the required section/mention (case-insensitive)
-3. Report missing sections
+1. Read `diff_diff/guides/llms.txt` and verify the name appears under the right section:
+   - **Estimators** (e.g. CallawaySantAnna, SunAbraham, TROP, BaconDecomposition): under `## Estimators`
+   - **Diagnostics-class** (HonestDiD, and any future diagnostic-only entries): under `## Diagnostics and Sensitivity Analysis`
+2. Read `README.md` and verify the name appears in the matching flat catalog:
+   - **Estimators**: in the `## Estimators` section
+   - **Diagnostics-class** (HonestDiD): in the `## Diagnostics & Sensitivity` section
+3. Report missing entries
 
 ```bash
-# Example: Check if "Callaway" appears in README
-grep -i "callaway" README.md
+# Extract the README ## Estimators section. Use a flag-based awk because the
+# range form `awk '/^## Estimators/,/^## /'` self-terminates on the opening H2.
+extract_section() {
+  awk -v target="$1" '
+    $0 == "## " target { flag=1; next }
+    flag && /^## / { flag=0 }
+    flag { print }
+  ' README.md
+}
+
+# Example: an estimator (lives in ## Estimators)
+extract_section "Estimators" | grep -c 'CallawaySantAnna'
+
+# Example: a diagnostic (lives in ## Diagnostics & Sensitivity)
+extract_section "Diagnostics & Sensitivity" | grep -c 'Honest DiD'
+
+# Always verify both surfaces
+grep -c 'CallawaySantAnna' diff_diff/guides/llms.txt
 ```
 
+Do NOT search for per-estimator README sections - they were intentionally removed in the 2026-04 docs refresh. The README's `## Estimators` and `## Diagnostics & Sensitivity` headings are the only valid catalog surfaces.
+
 ### 3. Scholarly References Check
 
 For each reference category:
-1. Search README.md References section for required citations
+1. Search `docs/references.rst` for required citations (NOT README.md - the bibliography moved out of README in the 2026-04 docs refresh)
 2. Verify author names and year appear together
 3. Report missing references
 
-Check patterns (case-insensitive):
+Check patterns (case-insensitive, run against `docs/references.rst`):
 - "Arkhangelsky.*2021" for Synthetic DiD
 - "Callaway.*Sant.Anna.*2021" for staggered
 - "Rambachan.*Roth.*2023" for Honest DiD
 - "Athey.*Imbens.*Qu.*Viviano.*2025" for TROP
 - "Goodman.Bacon.*2021" for Bacon decomposition
 - etc.
 
+```bash
+# Example
+grep -i 'Arkhangelsky.*2021' docs/references.rst
+```
+
 ### 4. API Documentation Check
 
 For each RST file in `docs/api/`:
@@ -176,12 +223,12 @@ Generate a summary report:
 ```
 === Documentation Completeness Check ===
 
-README Sections:
-  [PASS] DifferenceInDifferences - Found in "Basic Difference-in-Differences"
-  [PASS] CallawaySantAnna - Found in "Staggered Adoption"
-  [FAIL] NewEstimator - NOT FOUND
+llms.txt + README Catalog:
+  [PASS] DifferenceInDifferences - Found in llms.txt and README Estimators catalog
+  [PASS] CallawaySantAnna - Found in both surfaces
+  [FAIL] NewEstimator - missing from llms.txt and README catalog
 
-Scholarly References:
+Scholarly References (docs/references.rst):
   [PASS] Synthetic DiD - Arkhangelsky et al. (2021)
   [PASS] Honest DiD - Rambachan & Roth (2023)
   [FAIL] Bacon Decomposition - Missing Goodman-Bacon (2021)

.claude/commands/pre-merge-check.md

@@ -187,7 +187,9 @@ Based on your changes to: <list of changed files>
 ```
 ### Documentation Sync
 - [ ] Docstrings updated for changed function signatures
-- [ ] README updated if user-facing behavior changes
+- [ ] `diff_diff/guides/llms.txt` updated if the public API surface changed (AI-agent contract)
+- [ ] `docs/api/*.rst` and `docs/references.rst` updated as appropriate
+- [ ] `README.md` updated ONLY for landing-page-relevant changes (catalog one-liner, hero/badges/tagline, top-level capability paragraph). Per CONTRIBUTING.md, README is not the place for usage examples or per-estimator sections.
 ```
 
 #### If This Appears to Be a Bug Fix

.claude/commands/review-plan.md

@@ -225,7 +225,7 @@ Check for **missing related changes**:
 - Tests for new/changed functionality
 - `__init__.py` export updates
 - `get_params()` / `set_params()` updates for new parameters
-- Documentation updates (README, RST, tutorials, CONTRIBUTING.md, CLAUDE.md if design patterns change)
+- Documentation updates (`diff_diff/guides/llms.txt` for new public-API surfaces, `docs/api/*.rst`, `docs/references.rst` for new citations, tutorials, CONTRIBUTING.md, CLAUDE.md if design patterns change). README updates only if the change affects the landing page (new estimator catalog one-liner, hero/badges/tagline, top-level capability paragraph) - per CONTRIBUTING.md, README is not the place for usage examples or per-estimator sections.
 - For bug fixes: did the plan grep for ALL occurrences of the pattern, or just the one reported?
 
 Check for **unnecessary additions**:

.claude/commands/review-pr.md

@@ -45,8 +45,10 @@ Analyze PRs across 6 dimensions:
 
 ### 4. Documentation Review
 - Docstrings for new/modified functions
-- README updates if needed
-- API documentation (RST files)
+- `diff_diff/guides/llms.txt` updated if a new public-API surface landed (AI-agent contract)
+- API documentation (RST files in `docs/api/`)
+- `docs/references.rst` updated for new scholarly citations
+- README updated ONLY for landing-page-relevant changes (catalog one-liner, hero/badges/tagline, top-level capability paragraph). Per CONTRIBUTING.md, README is not the place for usage examples or per-estimator sections.
 - Inline comments for complex logic
 
 ### 5. Performance
@@ -133,7 +135,7 @@ If no PR number is provided, use AskUserQuestion to request it.
 
 ## Part 4: Documentation Assessment
 
-[Check for docstrings, README updates, API docs as needed]
+[Check docstrings, llms.txt for new public-API surfaces, API RST docs, references.rst for new citations, README only for landing-page-relevant changes]
 
 ---
 

BRIEFING.md

@@ -1,59 +1,80 @@
-# dcdh-by-path — Briefing
-
-## The ask
-
-Clément de Chaisemartin (dCDH author) suggested implementing the `by_path`
-option from R's `did_multiplegt_dyn`. It disaggregates the dynamic event-study
-by observed treatment trajectory so practitioners can compare paths like:
-
-- `(0,1,0,0)` — one pulse
-- `(0,1,1,0)` — two periods on, then off
-- `(0,1,1,1)` — three periods on, then off
-- `(0,1,0,1)` vs `(0,1,1,0)` — sequencing
-
-Use case: "is a single pulse enough, or do you need sustained exposure?"
-
-## Where we stand today
-
-`diff_diff/chaisemartin_dhaultfoeuille.py` implements `ChaisemartinDHaultfoeuille`.
-
-- Supports reversible on/off treatments (the only estimator in the library
-  that does)
-- **Currently drops multi-switch groups by default** (`drop_larger_lower=True`) —
-  exactly the groups `by_path` wants to keep and compare
-- Stratifies by direction cohort (`DID_+`, `DID_-`, `S_g = sign(Δ)`) but not
-  by trajectory
-- No `by_path`, `treatment_path`, or path-enumeration code exists anywhere
-- Not on ROADMAP.md; not in TODO.md
-
-## Shape of the work
-
-1. Parameter: likely `by_path: bool = False` (implies `drop_larger_lower=False`)
-2. Enumerate unique treatment histories `(D_{g,1}, …, D_{g,T})` per group;
-   optionally accept a user-specified subset of paths of interest
-3. Per-path `DID_{g,l}` aggregation with influence-function SEs per path
-4. Result container extension: `path_effects` dict keyed by trajectory tuple,
-   each holding ATT + SE + CI vectors
-5. Decide interaction with `drop_larger_lower`: probably forbid both being
-   non-default simultaneously, or have `by_path` override
-6. REGISTRY.md section on path-heterogeneity methodology + deviation notes
-7. Methodology reference: `did_multiplegt_dyn` manual §on `by_path`; dCDH
-   dynamic paper for the `DID_{g,l}` building block (already cited in REGISTRY)
-
-## Open methodology questions (for plan mode)
-
-- Which paths are enumerable? All observed, or user-specified subset only?
-  R's default behavior on cardinality control is worth checking.
-- How does path stratification interact with the current cohort pooling
-  `(D_{g,1}, F_g, S_g)` used for variance recentering — does it still apply
-  per path?
-- Placebo and TWFE diagnostics: compute per-path or overall only?
-- Bootstrap interaction: per-path bootstrap blocks vs single bootstrap with
-  per-path aggregation
-
-## Before starting
-
-- Pull the R manual section on `by_path` for `did_multiplegt_dyn` — the option
-  spec there is load-bearing; don't infer from usage examples alone
-- Methodology changes: consult `docs/methodology/REGISTRY.md` first
-- New estimator surface → budget ~12-20 CI review rounds
+# docs-refresh — Briefing
+
+## The goal
+
+Two-part documentation sweep, sequenced as one initiative across multiple PRs:
+
+1. **README.md aggressive trim**
+2. **RTD staleness audit + targeted fixes**
+
+Tutorial work is OUT OF SCOPE — that's a separate worktree (`dcdh-tutorial`).
+
+## Why now
+
+Recent releases (3.0.x → 3.3.0) shipped a lot of new surface area without
+proportional README/RTD updates:
+
+- HeterogeneousAdoptionDiD (entirely new estimator, multi-phase)
+- profile_panel() + llms-autonomous.txt
+- dCDH by_path + R parity
+- SDiD survey support across all three variance methods
+- BR/DR target_parameter (schema 2.0)
+- TROP backend parity
+
+README is too long for skim consumption (SEO + first-impression problem).
+RTD likely has stale pages, missing API references, and outdated examples.
+
+## Sequencing
+
+### PR 1 — README aggressive trim
+Target a tight shape:
+- One-line value prop
+- Install (`pip install diff-diff`)
+- Minimal working example (5-10 lines, one estimator)
+- Estimator-list one-liner with link to RTD for full reference
+- Citation + license
+
+Aggressive cuts. Anything that belongs on RTD goes to RTD (or stays there if
+already there). Don't try to be the docs.
+
+Out of scope: rewriting RTD content that the README links to.
+
+### PR 2+ — RTD staleness audit + fixes
+
+Audit step (read-only):
+- Walk `docs/` and identify pages missing post-3.0.x estimators / surfaces
+- Cross-reference `docs/doc-deps.yaml` to surface known dependency drift
+- Categorize: missing API page, stale example, broken link, outdated narrative
+
+Then fix in scoped PRs (one PR per coherent batch — e.g., "Add HAD API reference
++ choosing-estimator entry", "Refresh practitioner decision tree for 3.3.0").
+
+## What to read first
+
+- `README.md` (current state, length)
+- `docs/index.rst` (RTD entry point)
+- `docs/doc-deps.yaml` (source-to-doc dependency map)
+- `docs/api/` (API reference pages — what's missing)
+- `docs/methodology/REGISTRY.md` (don't reformat; just cross-check it's
+  referenced from RTD where appropriate)
+- `CLAUDE.md` "Documenting Deviations" section (label patterns, don't violate)
+
+## Memory rules to honor
+
+- Hyphens, not em dashes (writing style)
+- No competitor mentions in formal docs (ROADMAP / user-facing)
+- No version numbers as RTD section headings
+- diff-diff perspective (not neutral comparisons)
+- Tutorial-scope discipline does NOT apply here — this is reference docs
+
+## Out of scope
+
+- New tutorials (separate `dcdh-tutorial` worktree owns DCDH; HAD tutorial queued after)
+- ROADMAP.md restructuring (separate concern)
+- BR/DR positioning beyond "experimental preview" framing (per memory)
+
+## Cleanup note
+
+This BRIEFING.md was accidentally committed to main from a prior worktree
+session. Long-term, drop it from main and add to .gitignore so worktree
+briefings stay local.

CLAUDE.md

@@ -114,6 +114,21 @@ category (`Methodology/Correctness`, `Performance`, or `Testing/Docs`):
 |-------|----------|----|----------|
 | Description of deferred item | `file.py` | #NNN | Medium/Low |
 
+## README discipline
+
+`README.md` is a **landing page**, not the documentation. Target ~190 lines. The 3,119-line README that existed before the 2026-04 docs refresh grew because workflow conventions told contributors to add to README on every change.
+
+When adding new functionality, the source of truth is:
+
+- **`diff_diff/guides/llms.txt`** for the AI-agent contract (one-line catalog entry per estimator with paper citation + RTD link). This file is bundled in the wheel and published on RTD via `docs/conf.py` `html_extra_path`.
+- **`docs/api/*.rst`** for full API reference.
+- **`docs/references.rst`** for scholarly citations.
+- **`docs/tutorials/*.ipynb`** for hands-on examples.
+- **`CHANGELOG.md`** for release notes.
+- **`README.md`** for ONE LINE in the `## Estimators` flat catalog (or `## Diagnostics & Sensitivity` for diagnostic-class features). Do NOT add usage examples, parameter tables, per-estimator sections, or full bibliographies.
+
+`/docs-impact` and `/docs-check` enforce these surfaces. See `CONTRIBUTING.md` "README is a landing page, not the docs" for the full convention.
+
 ## Testing Conventions
 
 - **`ci_params` fixture** (session-scoped in `conftest.py`): Use `ci_params.bootstrap(n)` and