Skip to content

Commit 79088f0

Browse files
fix(governance): make validate-hypatia-baseline actually consult .hypatia-baseline.json (#455)
## Summary The estate-wide `validate-hypatia-baseline` job in `governance-reusable.yml` **never read `.hypatia-baseline.json`** — it ran `hypatia-cli.sh scan .`, counted raw findings with `jq length`, and failed if >0. Its error message ("findings not in baseline") was misleading; the baseline had **zero** functional effect beyond flipping the `has_baseline` trigger. This is the estate-wide fix that **standards#449 explicitly deferred** ("does not touch `governance-reusable.yml` itself … has the same bug for every repo that carries a real baseline"). This wires the **already-present but unused** `scripts/apply-baseline.sh` into the job. ## What changed (one job, `validate-hypatia-baseline`) 1. **Sparse-checkout `standards/scripts`** to get `apply-baseline.sh` (mirrors the existing `language-policy` job's `.standards-checkout` pattern, `ref: main`). 2. **Relativize finding paths** before matching. hypatia's `honest_completion`/`code_safety` modules emit **absolute** `.file` paths (`Path.expand(repo)`), while baselines use repo-relative paths (per the schema + `docs/HYPATIA-BASELINE-FORMAT.adoc`) — so a naive wire-in would never match (the hypatia#566 gap). Repo root maps to `"."` so root-scoped findings (`no_tests`) match. 3. **Run `apply-baseline.sh … blocking`** with `BLOCKING_THRESHOLD: info`, preserving the historical strictness: any finding **not** acknowledged by an active (non-expired) baseline entry still fails the gate — **no silent-green** — and only exact matches are suppressed. ## Why it's safe to land - **Opt-in per repo.** Caller repos SHA-pin the reusable, so **nothing changes for any repo** (incl. neurophone) until it bumps its `governance.yml` pin. - **No silent-green by construction.** `apply-baseline.sh` keeps every non-baselined finding for the gate; new findings still fail. - **Touches no licence/SPDX logic.** ## Validation (done locally — standards' own CI can't exercise this) standards carries **no** `.hypatia-baseline.json`, so `has_baseline=false` and this job **skips** on standards' own CI. I therefore ground-truthed the wiring against synthetic fixtures shaped like real hypatia output (absolute paths) + a neurophone-shaped baseline: ``` === relativized paths === crates/lsm/benches/bench.rs . android/app/src/main/java/ai/neurophone/NativeLib.java TEST 1: all findings baselined -> exit 0 (pass; all suppressed) ✓ TEST 2: add one NEW non-baselined finding -> exit 1 (fail; no silent-green) ✓ TEST 3: kept=1 (crates/esn/src/lib.rs), suppressed=3 ✓ ``` Relativization handles the two hypatia path shapes (`<abs>/crates/...` and the bare repo root → `"."`); `apply-baseline.sh`'s `file`/`file_pattern` glob matching and `expires_at` expiry do the rest. ## Decisions for you 1. **Strictness.** I chose `BLOCKING_THRESHOLD: info` to preserve today's "fail on any finding" behaviour. If you'd prefer the gate only block on `>= high` (letting new low/medium through), flip that env value. 2. **Uppercase-`type` residual (pre-existing, not fixed here).** hypatia emits `SD022`/`GS007` for `structural_drift`/`git_state`, which violate the baseline **schema**'s lowercase `type` pattern. The filter matches them fine (it doesn't schema-validate), but such baseline entries are schema-invalid. Recommend a separate follow-up (relax the schema pattern, or lowercase hypatia's output) — I can file it. 3. **Rollout.** Recommend: merge here, then repin **one** caller (e.g. neurophone) and confirm its `validate-hypatia-baseline` goes green-on-baseline / red-on-new-finding before wider repins. Left as a **draft** for your review since it edits a security-scan gate shared estate-wide. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0172RBMz3qYjb1ttzD2i7RNh --- _Generated by [Claude Code](https://claude.ai/code/session_0172RBMz3qYjb1ttzD2i7RNh)_ Co-authored-by: Claude <noreply@anthropic.com>
1 parent 2bad986 commit 79088f0

1 file changed

Lines changed: 43 additions & 9 deletions

File tree

.github/workflows/governance-reusable.yml

Lines changed: 43 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -106,18 +106,52 @@ jobs:
106106
mix deps.get && mix escript.build
107107
fi
108108
109+
# A reusable workflow only auto-checks-out its own YAML, not sibling
110+
# scripts. Sparse-check-out standards' scripts/ to get apply-baseline.sh,
111+
# mirroring the language-policy job below. Pinned to main because
112+
# github.workflow_sha resolves to the *caller* repo's SHA (which would 404).
113+
- name: Check out standards for the baseline filter
114+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
115+
with:
116+
repository: hyperpolymath/standards
117+
ref: main
118+
path: .standards-checkout
119+
sparse-checkout: |
120+
scripts
121+
sparse-checkout-cone-mode: false
122+
109123
- name: Run Hypatia scan (Baseline validation)
124+
env:
125+
# Preserve the historical strictness: fail on ANY finding not
126+
# acknowledged by .hypatia-baseline.json (not just >= high). Set to
127+
# 'high' to only block on high/critical.
128+
BLOCKING_THRESHOLD: info
110129
run: |
111130
echo "Scanning repository: ${{ github.repository }} (checking baseline)"
112-
HYPATIA_FORMAT=json "$HOME/hypatia/hypatia-cli.sh" scan . > hypatia-findings.json
113-
114-
FINDING_COUNT=$(jq '. | length' hypatia-findings.json 2>/dev/null || echo 0)
115-
116-
if [ "$FINDING_COUNT" -gt 0 ]; then
117-
echo "::error::Baseline validation failed. Found $FINDING_COUNT findings not in baseline."
118-
exit 1
119-
fi
120-
echo "Baseline validation successful. No new findings."
131+
HYPATIA_FORMAT=json "$HOME/hypatia/hypatia-cli.sh" scan . > hypatia-findings.raw.json
132+
133+
# Relativize finding paths before matching. Hypatia's honest_completion
134+
# and code_safety modules emit ABSOLUTE host paths in `.file`
135+
# (Path.expand(repo)), while .hypatia-baseline.json uses repo-relative
136+
# paths per the schema + docs/HYPATIA-BASELINE-FORMAT.adoc. Strip the
137+
# workspace prefix so baseline file/file_pattern entries can match, and
138+
# map the repo root itself to "." so root-scoped findings (no_tests)
139+
# match. Upstream fix tracked in hypatia#566; this is the stopgap.
140+
jq --arg root "$PWD" '
141+
map(if (.file? // "") == $root then .file = "."
142+
elif (.file? // "") | startswith($root + "/")
143+
then .file = (.file | ltrimstr($root + "/"))
144+
else . end)
145+
' hypatia-findings.raw.json > hypatia-findings.json
146+
147+
# Filter against the per-repo baseline. apply-baseline.sh KEEPS every
148+
# finding not matched by an active (non-expired) baseline entry — so a
149+
# genuinely new finding still fails the gate (no silent-green) — and
150+
# suppresses only exact matches. Exits non-zero iff a kept finding is at
151+
# or above BLOCKING_THRESHOLD. (Validated against synthetic fixtures;
152+
# see the PR description.)
153+
bash .standards-checkout/scripts/apply-baseline.sh \
154+
hypatia-findings.json .hypatia-baseline.json blocking
121155
language-policy:
122156
name: Language / package anti-pattern policy
123157
runs-on: ${{ inputs.runs-on }}

0 commit comments

Comments
 (0)