Skip to content

Commit a01600d

Browse files
committed
CI R1: add missing-notebook skip guard + tighten generic anchors
CI AI review on #439 R1: P1: tests/_tutorial_drift.py read docs/tutorials/*.ipynb unconditionally, but the Rust-test CI matrix copies only tests/ to /tmp/tests and runs pytest there (no docs/), so the new tests would FileNotFoundError instead of skipping cleanly. Added pytest.skip() guard in _read_notebook() matching the repo convention used in test_notebook_md_extract.py and test_nprobust_port.py. P3: tightened two generic anchors that could false-pass: - T21 'WAS' (matches many incidental occurrences) -> 'target = `WAS`' (the exact paper-step-1 phrasing, single occurrence). - T20 'median' + '$25K' (split, can both pass independently even if the sentence drifts) -> 'median ~$25K' (the exact tilde-prefixed phrase from the design-fit narrative).
1 parent 9945d30 commit a01600d

3 files changed

Lines changed: 25 additions & 5 deletions

File tree

tests/_tutorial_drift.py

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,24 @@
2020

2121

2222
def _read_notebook(nb_relpath: str) -> dict:
23-
"""Load a notebook by repo-relative path (e.g. ``docs/tutorials/X.ipynb``)."""
23+
"""Load a notebook by repo-relative path (e.g. ``docs/tutorials/X.ipynb``).
24+
25+
Skips the calling test via ``pytest.skip(...)`` when the notebook file
26+
is not present. The Rust-test CI job (and the isolated-install job)
27+
copies only ``tests/`` to ``/tmp/tests`` and runs from there, without
28+
``docs/`` available. The repo convention is to skip cleanly when
29+
artifacts are absent rather than fail (see e.g.
30+
``tests/test_notebook_md_extract.py`` and ``tests/test_nprobust_port.py``).
31+
"""
32+
import pytest
33+
2434
nb_path = Path(__file__).resolve().parents[1] / nb_relpath
35+
if not nb_path.exists():
36+
pytest.skip(
37+
f"Notebook {nb_relpath!r} not available in this CI environment "
38+
"(isolated-install job copies only tests/, not docs/); "
39+
"rendered-surface cross-check requires a full repo checkout."
40+
)
2541
return json.loads(nb_path.read_text())
2642

2743

tests/test_t20_had_brand_campaign_drift.py

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -314,8 +314,9 @@ def test_notebook_quotes_match_pinned_constants():
314314
# Placebo-magnitude prose claim (locked analytically above by
315315
# test_event_study_pre_atts_near_zero with the ±0.1 envelope).
316316
"±0.06",
317-
# Sample summary in the design-fit narrative.
318-
"median",
319-
"$25K",
317+
# Sample-summary phrase in the design-fit narrative. Use the
318+
# exact tilde-prefixed form so a future drift in the sentence
319+
# (e.g. "median around $25K") would surface here.
320+
"median ~$25K",
320321
]
321322
assert_quotes_in_rendered(T20_NOTEBOOK, expected_quotes, surface="rendered")

tests/test_t21_had_pretest_workflow_drift.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -434,7 +434,10 @@ def test_notebook_quotes_match_pinned_constants():
434434
"0.2899", # Yatchew side-panel null=mean_independence p-value
435435
# Design auto-detect outcome (also pinned by overall-path tests).
436436
"continuous_at_zero",
437-
"WAS",
437+
# Use the exact paper-step-1 phrasing with target=`WAS` so we
438+
# don't false-pass on the many incidental occurrences of "WAS"
439+
# elsewhere in the prose.
440+
"target = `WAS`",
438441
# Overall Yatchew p-value (analytical short-circuit on this DGP).
439442
"1.0000",
440443
# Overall Yatchew sigma2_lin in the rendered output.

0 commit comments

Comments
 (0)