Context
mix test is red on main: 73 failures / 1398 tests. That number has been treated as
one undifferentiated problem. It is not — it splits cleanly into two classes with
completely different fixes, and this issue covers the smaller, tractable one.
Measured by running the suspect files in isolation (--seed 0) and then together:
| run |
failures |
workflow_audit_test.exs alone |
2 |
rules/cicd_rules_vlang_test.exs alone |
2 |
root_hygiene_test.exs alone |
1 |
dashboard_test.exs alone |
1 |
rules/workflow_audit_chapel_test.exs alone |
1 |
| sum |
7 |
| all five together |
7 |
Identical. These fail alone exactly as they fail in-suite, so they are real deterministic
failures, not test pollution. That distinguishes them from the other ~66 failures, which
concentrate in contract_test / blackboard_test / reflexive_test / kin_test and are
order-dependent — tracked separately in #643.
Splitting these out means 7 bounded, independently-fixable defects stop being hidden behind
a 73-failure wall.
The seven
test/rules/cicd_rules_vlang_test.exs:45 — vlang_detected does not exempt Coq
proof scripts (.v is also Coq's extension).
test/rules/cicd_rules_vlang_test.exs:64 — vlang_detected does not exempt Verilog
and linguist samples. left: %{...}, right: nil.
test/root_hygiene_test.exs:101 — scan_required_missing/1 no longer flags a missing
SECURITY.md; assert security != nil gets nil.
test/rules/workflow_audit_chapel_test.exs:126 — chapel-find-comm-gasnet not fired on
the brittle printchplenv glob. right: [] — no recipes at all.
test/dashboard_test.exs:39 — GET / does not reference every
Hypatia.Telemetry.all_events() name in the JS handler list.
test/workflow_audit_test.exs:8 — assert length(findings) == 17, actual 6.
test/workflow_audit_test.exs:233 — assert report.missing_count == 15, actual 4.
Items 1 and 2 are scanner false positives, not merely failing tests. A vlang_detected
rule that fires on Coq and Verilog files emits wrong findings into the estate's own scan
results — the exact failure mode CLAUDE.md § Scanner Hygiene exists to prevent.
Hypotheses — explicitly NOT yet verified
Stated as leads, not conclusions; each needs checking against the rule source before acting.
- 6 and 7 look like drift between rule and test: a hardcoded required-workflow list
shrank (17 → 6, 15 → 4) and the assertions were not updated. If so the fix is to derive
the expected count from the rule's own list rather than hardcode it — otherwise this
recurs on the next change.
- 1, 2 and 4 look like genuine rule-logic gaps (missing extension exemptions; a recipe
predicate that no longer matches).
- 3 and 5 are unclassified.
Do not assume the counts in 6/7 are "just stale" — if the required-workflow list shrank
unintentionally, the rule is the bug and the test is correct.
Acceptance criteria
Test plan
mix test test/workflow_audit_test.exs test/rules/cicd_rules_vlang_test.exs \
test/root_hygiene_test.exs test/dashboard_test.exs \
test/rules/workflow_audit_chapel_test.exs --seed 0
Expect 88 tests, 0 failures (currently 88 tests, 7 failures).
Then confirm the total drops: full mix test should fall from 73 failures to ~66, with the
remainder being the #643 cluster.
Model tier
Sonnet — five independent, well-localised test/rule pairs in one repo, each with a
concrete assertion and no new design required. Items 1 and 2 warrant a little more care as
they change scanner output that feeds the estate.
Related
Context
mix testis red onmain: 73 failures / 1398 tests. That number has been treated asone undifferentiated problem. It is not — it splits cleanly into two classes with
completely different fixes, and this issue covers the smaller, tractable one.
Measured by running the suspect files in isolation (
--seed 0) and then together:workflow_audit_test.exsalonerules/cicd_rules_vlang_test.exsaloneroot_hygiene_test.exsalonedashboard_test.exsalonerules/workflow_audit_chapel_test.exsaloneIdentical. These fail alone exactly as they fail in-suite, so they are real deterministic
failures, not test pollution. That distinguishes them from the other ~66 failures, which
concentrate in
contract_test/blackboard_test/reflexive_test/kin_testand areorder-dependent — tracked separately in #643.
Splitting these out means 7 bounded, independently-fixable defects stop being hidden behind
a 73-failure wall.
The seven
test/rules/cicd_rules_vlang_test.exs:45—vlang_detecteddoes not exempt Coqproof scripts (
.vis also Coq's extension).test/rules/cicd_rules_vlang_test.exs:64—vlang_detecteddoes not exempt Verilogand linguist samples.
left: %{...},right: nil.test/root_hygiene_test.exs:101—scan_required_missing/1no longer flags a missingSECURITY.md;assert security != nilgetsnil.test/rules/workflow_audit_chapel_test.exs:126—chapel-find-comm-gasnetnot fired onthe brittle
printchplenvglob.right: []— no recipes at all.test/dashboard_test.exs:39—GET /does not reference everyHypatia.Telemetry.all_events()name in the JS handler list.test/workflow_audit_test.exs:8—assert length(findings) == 17, actual 6.test/workflow_audit_test.exs:233—assert report.missing_count == 15, actual 4.Items 1 and 2 are scanner false positives, not merely failing tests. A
vlang_detectedrule that fires on Coq and Verilog files emits wrong findings into the estate's own scan
results — the exact failure mode
CLAUDE.md§ Scanner Hygiene exists to prevent.Hypotheses — explicitly NOT yet verified
Stated as leads, not conclusions; each needs checking against the rule source before acting.
shrank (17 → 6, 15 → 4) and the assertions were not updated. If so the fix is to derive
the expected count from the rule's own list rather than hardcode it — otherwise this
recurs on the next change.
predicate that no longer matches).
Do not assume the counts in 6/7 are "just stale" — if the required-workflow list shrank
unintentionally, the rule is the bug and the test is correct.
Acceptance criteria
assertion was wrong.
mix test <the five files> --seed 0reports 0 failures.false positive on a
.vCoq file and a Verilog file.Test plan
mix test test/workflow_audit_test.exs test/rules/cicd_rules_vlang_test.exs \ test/root_hygiene_test.exs test/dashboard_test.exs \ test/rules/workflow_audit_chapel_test.exs --seed 0Expect
88 tests, 0 failures(currently88 tests, 7 failures).Then confirm the total drops: full
mix testshould fall from 73 failures to ~66, with theremainder being the #643 cluster.
Model tier
Sonnet — five independent, well-localised test/rule pairs in one repo, each with a
concrete assertion and no new design required. Items 1 and 2 warrant a little more care as
they change scanner output that feeds the estate.
Related
cx/L3)