Skip to content

[quality] Recorded CI exemptions cite substitute gates that nothing verifies still exist #1665

Closed
#1666
Closed
[quality] Recorded CI exemptions cite substitute gates that nothing verifies still exist#1665
#1666
Labels
agent/qualityApproved by a Hive merger/owner for auto-merge on green CIstage/040-covered-by-prAn open PR covers this issuetestingApproved by a Hive merger/owner for auto-merge on green CI

Description

@hivecommons-hive
hivecommons-hive
bot
opened on Oct 19, 2026

Finding

tests/workflow-scripts.test.mjs keeps two exemption maps that let a gate stay out of CI as long as the omission is a decision on the record:

  • GATES_NOT_RUN_BY_CI — why a validate:*/check:* script is run by no workflow.
  • REFRESH_SCRIPTS_NOT_RUN_BY_CI — why a collect:*/fetch:* script is refreshed by no workflow.

Every reason argues its case by naming another file. The validate:end-user-talks entry is the load-bearing example:

Exists for the Netlify publish path … the PR lane already enforces the same link allow-list through tests/end-user-talks-data.test.mjs under npm run test:unit, so a separate ci.yml step would only run the contract a second time.

Four guards already cover these maps — the key is a defined script, the reason is non-empty, no workflow contradicts the entry — but none of them reads the reason for what it points at.

So: rename or delete tests/end-user-talks-data.test.mjs and the pull-request lane loses the end-user-talks link allow-list outright (no ci.yml step runs validate:end-user-talks, by design, because that test exists), while every check in the repository stays green and the exemption still reads as a standing decision. The recorded decision survives the evidence it rested on. The same holds for each collector exemption that cites its scripts/ header.

Verified on a621619: moving tests/end-user-talks-data.test.mjs aside leaves the whole suite green except the rendering tests — no guard names the broken citation.

This is not a coverage gap. Unit coverage of src/ and scripts/ is 100.00% lines / 100.00% regions (npm run test:unit:coverage, node v22, commit a621619), and the e2e lane gates at --check-source 100 --require-source-files. It is a missing contract over a hand-maintained rationale, not uncovered code.

Recommendation

Add one guard to tests/workflow-scripts.test.mjs asserting that every repository file a recorded exemption cites still exists, over both maps, in the same shape the surrounding guards use:

  • extract the citations as a pure function of the map so fixtures can prove the guard reports a stale one (the evidence convention tests/data-gate-contract.test.mjs sets out);
  • pin it against vacuity — both maps must be seen to cite something;
  • scope the pattern to .mjs files under tests/ and scripts/, which is what an exemption has so far had reason to name, so prose and Markdown citations are not dragged in.

Test-only change, no production code and no workflow edit.

Priority

  • Impact: medium — it does not break anything today, but it is the one way the end-user-talks link allow-list can leave the PR lane silently.
  • Effort: low — one test file.

Filed by quality agent (hold-gated mode)

🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: a621619

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

Activity

  1. hivecommons-hive
    added
    testingApproved by a Hive merger/owner for auto-merge on green CI
    agent/qualityApproved by a Hive merger/owner for auto-merge on green CI
    stage/010-actionableTriaged; eligible for the hive to pick up
    on Oct 19, 2026
  2. hivecommons-hive
    linked a pull request that will close this issuetest: fail when a recorded CI exemption cites a file that is gone #1666on Oct 19, 2026
  3. hivecommons-hive
    added
    stage/040-covered-by-prAn open PR covers this issue
    and removed
    stage/010-actionableTriaged; eligible for the hive to pick up
    on Oct 19, 2026

  4. hivecommons-hive commented on Oct 19, 2026

    @hivecommons-hive
    hivecommons-hiveBot
    on Oct 19, 2026 – with hivecommons-hive
    ContributorAuthor

    Folded finding. This was filed as a separate issue, but 1 open agent-filed issue (plus 5 closed as completed in the last 14 days) already cite ci.yml, so it was folded into the oldest of them instead of opening another. Treat this issue as the tracker for that component: prefer one structural fix that covers every folded variant over a PR per variant.

    [quality] Stale gate rationale in two test headers predates the end-user-talks validator

    Finding

    Two test headers still describe a world two pull requests have since changed, and both are load-bearing prose — the repository uses these headers as the record of why a gate exists, so a reader reasoning from them reaches a wrong conclusion about what covers data/end-user-talks.json.

    1. tests/end-user-talks-data.test.mjs:7-10 opens with:

    data/end-user-talks.json is the only committed dataset under data/ with no gate of its own: there is no scripts/validate-*.mjs for it in the ci.yml validate job, and the two tests that read it are rendering contracts rather than data contracts.

    scripts/validate-end-user-talks.mjs has existed since #1624 and is chained into validate:all, so "no gate of its own" is false. "The only committed dataset" is also no longer accurate on its own terms — tests/data-gate-contract.test.mjs:7-9 records that data/milestones.json likewise has no validator.

    What the header loses by being stale is the thing a reader most needs: the two gates do not back each other up. No ci.yml step runs validate:end-user-talks — the omission is deliberate and recorded in GATES_NOT_RUN_BY_CI in tests/workflow-scripts.test.mjs, on the grounds that this very test enforces the same contract on the pull-request lane. So this file is the whole of that lane's cover, and the header does not say so.

    2. tests/data-gate-contract.test.mjs:13-15 cites the stale sentence as evidence:

    … and the header of tests/end-user-talks-data.test.mjs still records it as "the only committed dataset under data/ with no gate of its own".

    Correcting (1) strands this quotation, so the two have to move together. Its underlying point — that until #1383 the dataset had no reader at all — stays true and is kept.

    Verified on a621619: scripts/validate-end-user-talks.mjs exists and validate:all chains it (package.json:72); the validate job in ci.yml runs 13 validators and not this one. No test asserts on either header's wording, so correcting the prose breaks nothing.

    Recommendation

    Rewrite the first paragraph of tests/end-user-talks-data.test.mjs to state the wiring as it now stands — this file is the pull-request lane's data contract, scripts/validate-end-user-talks.mjs covers the Netlify publish path via validate:all, and the GATES_NOT_RUN_BY_CI exemption means neither backs the other up — while keeping the rest of the header, which is still accurate and still the reason the file exists. Update the quotation in tests/data-gate-contract.test.mjs in the same change so the cross-reference survives.

    Prose only: no assertion, no production code, no workflow change.

    Priority

    • Impact: low — nothing is mis-gated today; the risk is a future reader deleting or thinning this file because its own header says the dataset is covered elsewhere.
    • Effort: low — two comment blocks in two test files.

    Filed by quality agent (hold-gated mode)

    🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: a621619

    — hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

    Folded finding. This was filed as a separate issue, but 1 open agent-filed issue (plus 5 closed as completed in the last 14 days) already cite ci.yml, so it was folded into the oldest of them instead of opening another. Treat this issue as the tracker for that component: prefer one structural fix that covers every folded variant over a PR per variant.

    [quality] ArchitectureCard's assets?. short-circuit is never exercised in a browser

    Finding

    ArchitectureCard in src/components/ReferenceArchitectures/index.js resolves
    its logo through an optional chain:

    const logoAsset = architecture.assets?.find((asset) =>
  /logo|wordmark/i.test(asset),
);

    All eight entries in data/architectures/catalog.json carry an assets array,
    so in a browser the chain is always entered. Its nullish short-circuit — the
    arm a catalog regenerated without the field would take — has never executed on
    the real site.

    Unit evidence

    npm run test:unit:coverage at da7701b reports the file at 100.00% lines /
    100.00% regions    . tests/reference-architectures.test.mjs already pins the
    arm directly (a card with no assets array at all still renders initials), so
    this is not a missing unit test.

    End-to-end evidence

    Artifact e2e-coverage (id 11825781781) of CI run
    https://github.com/cncf/endusers/actions/runs/38418930699 (workflow ci.yml,
    job End-to-end coverage, head 1e37cdc, 2026-10-13T15:04:47Z) reports:

    src/components/ReferenceArchitectures/index.js | 100.00 | 95.65 |  | 13 34

    100% lines, 95.65% regions (44/46), uncovered regions at lines 13 and 34.

    Per the coverage-evidence rules this is unit-covered but not end-to-end
    covered → priority 2 (medium)    . The two coverage sources are not combinable at
    statement granularity, so they were analyzed separately.

    Recommendation

    Use the mechanism tests/tools/e2e-data-fixtures.cjs already documents for
    exactly this shape — an additive overlay, not a new fixture build:

    "setWhere": {
  "id=swisscom-cloud-native-telco": { "assets": null }
}

    swisscom-cloud-native-telco carries no logo or wordmark asset to begin with,
    so the chain already resolved to undefined for it and the rendered card is
    unchanged — only which arm produced the undefined differs. allianz and
    swisscom-kubernetes-service keep their arrays and their lack of a logo asset,
    so the entered-but-unmatched arm stays covered in the same build. No second
    Docusaurus compile is added.

    Then add a case to tests/e2e/data-fixtures.spec.js asserting the nullish card
    still renders and renders the no-logo shape, beside the unmatched and matched
    cards on the same page.

    • Overlay assets: null onto one catalog entry in tests/e2e/fixtures/data/architectures/catalog.json
    • Assert the short-circuit arm in tests/e2e/data-fixtures.spec.js

    Priority

    • Impact: medium
    • Effort: low

    Scope note: this issue is confined to ReferenceArchitectures' asset
    resolution and its fixture/spec. It does not overlap the freshness-note,
    initials(), GroupLinkStatus, FilterToolbar or MemberDirectory clusters
    held open by #1683, #1680, #1677, #1675, #1679 and #1682.

    🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: da7701b

    — hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

    Folded finding. This was filed as a separate issue, but 1 open agent-filed issue (plus 5 closed as completed in the last 14 days) already cite ci.yml, so it was folded into the oldest of them instead of opening another. Treat this issue as the tracker for that component: prefer one structural fix that covers every folded variant over a PR per variant.

    [quality] ReferenceArchitectures: the optional chain on architecture.assets never short-circuits in the e2e coverage build

    Finding

    ArchitectureCard in src/components/ReferenceArchitectures/index.js resolves
    its logo through an optional chain:

    const logoAsset = architecture.assets?.find((asset) =>
  /logo|wordmark/i.test(asset),
);

    All eight entries in data/architectures/catalog.json carry an assets array,
    so in a browser the chain is always entered. Its nullish short-circuit — the
    arm a catalog regenerated without the field would take — has never executed on
    the real site.

    Unit evidence

    npm run test:unit:coverage at da7701b reports the file at 100.00% lines /
    100.00% regions    . tests/reference-architectures.test.mjs already pins the
    arm directly (a card with no assets array at all still renders initials), so
    this is not a missing unit test.

    End-to-end evidence

    Artifact e2e-coverage (id 11825781781) of CI run
    https://github.com/cncf/endusers/actions/runs/38418930699 (workflow ci.yml,
    job End-to-end coverage, head 1e37cdc, 2026-10-13T15:04:47Z) reports:

    src/components/ReferenceArchitectures/index.js | 100.00 | 95.65 |  | 13 34

    100% lines, 95.65% regions (44/46), uncovered regions at lines 13 and 34.

    Per the coverage-evidence rules this is unit-covered but not end-to-end
    covered → priority 2 (medium)    . The two coverage sources are not combinable at
    statement granularity, so they were analyzed separately.

    Recommendation

    Use the mechanism tests/tools/e2e-data-fixtures.cjs already documents for
    exactly this shape — an additive overlay, not a new fixture build:

    "setWhere": {
  "id=swisscom-cloud-native-telco": { "assets": null }
}

    swisscom-cloud-native-telco carries no logo or wordmark asset to begin with,
    so the chain already resolved to undefined for it and the rendered card is
    unchanged — only which arm produced the undefined differs. allianz and
    swisscom-kubernetes-service keep their arrays and their lack of a logo asset,
    so the entered-but-unmatched arm stays covered in the same build. No second
    Docusaurus compile is added.

    Then add a case to tests/e2e/data-fixtures.spec.js asserting the nullish card
    still renders and renders the no-logo shape, beside the unmatched and matched
    cards on the same page.

    • Overlay assets: null onto one catalog entry in tests/e2e/fixtures/data/architectures/catalog.json
    • Assert the short-circuit arm in tests/e2e/data-fixtures.spec.js

    Priority

    • Impact: medium
    • Effort: low

    Scope note: this issue is confined to ReferenceArchitectures' asset
    resolution and its fixture/spec. It does not overlap the freshness-note,
    initials(), GroupLinkStatus, FilterToolbar or MemberDirectory clusters
    held open by #1683, #1680, #1677, #1675, #1679 and #1682.

    🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: da7701b

    — hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

  5. mrbobbytables
    closed this as completedin #1666on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/qualityApproved by a Hive merger/owner for auto-merge on green CIstage/040-covered-by-prAn open PR covers this issuetestingApproved by a Hive merger/owner for auto-merge on green CI

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions