Skip to content

test: 7 DETERMINISTIC Elixir failures (fail in isolation) — separate from the #643 order-dependent cluster #645

Description

@hyperpolymath

Context

mix test is red on main: 73 failures / 1398 tests. That number has been treated as
one undifferentiated problem. It is not — it splits cleanly into two classes with
completely different fixes, and this issue covers the smaller, tractable one.

Measured by running the suspect files in isolation (--seed 0) and then together:

run failures
workflow_audit_test.exs alone 2
rules/cicd_rules_vlang_test.exs alone 2
root_hygiene_test.exs alone 1
dashboard_test.exs alone 1
rules/workflow_audit_chapel_test.exs alone 1
sum 7
all five together 7

Identical. These fail alone exactly as they fail in-suite, so they are real deterministic
failures, not test pollution
. That distinguishes them from the other ~66 failures, which
concentrate in contract_test / blackboard_test / reflexive_test / kin_test and are
order-dependent — tracked separately in #643.

Splitting these out means 7 bounded, independently-fixable defects stop being hidden behind
a 73-failure wall.

The seven

  1. test/rules/cicd_rules_vlang_test.exs:45vlang_detected does not exempt Coq
    proof scripts (.v is also Coq's extension).
  2. test/rules/cicd_rules_vlang_test.exs:64vlang_detected does not exempt Verilog
    and linguist samples. left: %{...}, right: nil.
  3. test/root_hygiene_test.exs:101scan_required_missing/1 no longer flags a missing
    SECURITY.md; assert security != nil gets nil.
  4. test/rules/workflow_audit_chapel_test.exs:126chapel-find-comm-gasnet not fired on
    the brittle printchplenv glob. right: [] — no recipes at all.
  5. test/dashboard_test.exs:39GET / does not reference every
    Hypatia.Telemetry.all_events() name in the JS handler list.
  6. test/workflow_audit_test.exs:8assert length(findings) == 17, actual 6.
  7. test/workflow_audit_test.exs:233assert report.missing_count == 15, actual 4.

Items 1 and 2 are scanner false positives, not merely failing tests. A vlang_detected
rule that fires on Coq and Verilog files emits wrong findings into the estate's own scan
results — the exact failure mode CLAUDE.md § Scanner Hygiene exists to prevent.

Hypotheses — explicitly NOT yet verified

Stated as leads, not conclusions; each needs checking against the rule source before acting.

  • 6 and 7 look like drift between rule and test: a hardcoded required-workflow list
    shrank (17 → 6, 15 → 4) and the assertions were not updated. If so the fix is to derive
    the expected count from the rule's own list rather than hardcode it — otherwise this
    recurs on the next change.
  • 1, 2 and 4 look like genuine rule-logic gaps (missing extension exemptions; a recipe
    predicate that no longer matches).
  • 3 and 5 are unclassified.

Do not assume the counts in 6/7 are "just stale" — if the required-workflow list shrank
unintentionally, the rule is the bug and the test is correct.

Acceptance criteria

  • Each of the seven either passes, or is closed with a recorded reason why the
    assertion was wrong.
  • mix test <the five files> --seed 0 reports 0 failures.
  • Items 1 and 2 are confirmed fixed by a case that would previously have produced a
    false positive on a .v Coq file and a Verilog file.
  • No assertion in 6/7 hardcodes a count that the rule's own data already determines.

Test plan

mix test test/workflow_audit_test.exs test/rules/cicd_rules_vlang_test.exs \
         test/root_hygiene_test.exs test/dashboard_test.exs \
         test/rules/workflow_audit_chapel_test.exs --seed 0

Expect 88 tests, 0 failures (currently 88 tests, 7 failures).

Then confirm the total drops: full mix test should fall from 73 failures to ~66, with the
remainder being the #643 cluster.

Model tier

Sonnet — five independent, well-localised test/rule pairs in one repo, each with a
concrete assertion and no new design required. Items 1 and 2 warrant a little more care as
they change scanner output that feeds the estate.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    cx/L2Scoped — one repo, needs code reading + tests, no new designrdy/readyUnblocked — a bot can start nowvfy/testCompletion provable by a test

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions