Skip to content

test: require every e2e fixture build to be reached by a spec - #1636

Merged
mrbobbytables merged 2 commits into
mainfrom
quality/e2e-fixture-build-spec-reach
Oct 19, 2026
Merged

mrbobbytables merged 2 commits into
mainfrom
quality/e2e-fixture-build-spec-reach

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test Improvement

Adds tests/e2e-coverage-build-reach.test.mjs, which requires every e2e
coverage fixture build to be reached by at least one spec.

The fixture-build list is a directory listing, by design: every
tests/e2e/fixtures/data-<name>/ directory declares one more build, and
npm run build:e2e:coverage compiles it into build/e2e-coverage-<name>.
The module header in tests/tools/e2e-data-fixtures.cjs states the cost in the
same breath:

Each build is a full Docusaurus compile, so CI time for the end-to-end
coverage job grows linearly in the number of data-<name>/ directories.

Nothing asserted the other half of that contract.

  • tests/e2e-data-fixtures.test.mjs guards the directory → build
    direction. Its comment names the failure it watches for — "a spec asserting
    against a route that 404s" — which is a spec without a build, not a build
    without a spec. It then pins the current names with
    names.includes('variant') / names.includes('no-revision'), which keep
    passing after a build's last spec is deleted.
  • tests/e2e-coverage-builds.test.mjs only checks that the compiler loops over
    coverageBuildNames() and derives the right out-dir and base URL. It is
    indifferent to whether the resulting site is ever requested.
  • Playwright cannot report it: an unvisited build produces no failing test,
    only a slower job.

A spec has to show two things to count — the base path /e2e-coverage-<name>
(anchored so /e2e-coverage-variant is not credited by a hypothetical
/e2e-coverage-variant-extra) and E2E_COVERAGE_BUILD: '<name>', the pairing
every existing variant spec already uses. Either alone is weak: a spec that
names the base path but never resolves the build's data is asserting against a
route it cannot describe; one that loads the data but never names the base path
never opens the site.

A second test pins that the ordinary coverage build is addressed by the site
root rather than a build name, so the loop is not later "fixed" by adding it to
the name list.

Both current builds (variant, no-revision) pass today — this is a
regression guard, not a report of a broken build.

Verification

  • TZ=UTC node --test tests/e2e-coverage-build-reach.test.mjs → 2 passed.
  • Mutation check: creating tests/e2e/fixtures/data-orphan/ makes it fail with
    orphan: no spec navigates /e2e-coverage-orphan; the build is a full Docusaurus compile on every run that covers nothing.
  • npm run test:unit:coverage:check → exit 0; the new file is itself
    100.00% lines / 100.00% regions, and src files stays 100.00/100.00.
  • npx prettier --check → clean.

Coverage evidence

This is a test-infrastructure finding, not a coverage gap: src/ is at
100.00% lines in both lanes.

  • Unit: TZ=UTC node tests/tools/coverage-report.mjs at main 3659352 →
    src files 100.00% lines / 100.00% regions (9900/9900, 2873/2873).
  • E2E: artifact e2e-coverage (id 11825781781) from CI run
    38418930699,
    head 1e37cdc, 2026-10-13, sealed passed, 204 captured scripts →
    src files 100.00% lines, 97.33% regions (656/674). All 18 residual
    uncovered regions are guard/fallback arms of the documented region-union
    class (see the "What the union credits" note in
    tests/e2e/data-variants.spec.js and issue 1256 ([quality] the e2e region union never credits a guard arm taken only in a sibling fixture build #1256)), not unexercised code.

The two lanes are reported separately rather than merged: they are different
instruments (Node coverage over build-time modules vs. V8 coverage over browser
bundles) with no compatible raw form. A fresh local e2e run at 3659352 was
attempted to replace the 6-day-old artifact, but chromium could not launch in
the analysis environment (libglib-2.0.so.0 missing, no root).

Related Issue

None — this PR carries the finding in full. Two attempts to file it as its own
issue were auto-consolidated by the issue relay into issue 1625 (#1625), which is a
different finding (the unexecuted-chunk arm at
tests/tools/e2e-coverage-report.mjs line 835, already claimed and
stage/040-covered-by-pr). There is no file or behaviour overlap between the
two, so this PR deliberately does not mark that issue resolved. If a maintainer
wants a tracking issue for this, it needs to be opened by hand.

Overlap check

Disjoint from every open hold-gated PR. PR 1626 and PR 1627 touch
tests/tools/e2e-coverage-report.mjs, PR 1627 also e2e-coverage-run.mjs, PR 1629
e2e-coverage-global-setup.mjs; this PR adds one new test file and touches no
existing file. It reads tests/tools/e2e-data-fixtures.cjs and
tests/e2e/*.spec.js, which no open PR modifies.


Filed by quality agent (hold-gated mode). Human review required.

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

The fixture-build list is a directory listing: every
tests/e2e/fixtures/data-<name>/ directory declares one more full
Docusaurus compile in the end-to-end coverage job. Nothing asserted the
reverse direction -- that each declared build is actually navigated by a
spec -- so an orphaned directory is a compile paid on every pull request
that serves a site nobody opens.

tests/e2e-data-fixtures.test.mjs guards directory -> build and pins the
current names with includes(), both of which keep passing after a
build's last spec is deleted. tests/e2e-coverage-builds.test.mjs only
checks that the compiler loops over coverageBuildNames(). Playwright
reports no failure for an unvisited build, only a slower job.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Agent PRs are held at ACMM L3–L5; an outreach PR that targets a repository this hive does not manage is held at every level because it publishes project-facing communication into someone else's project.

Hive will keep the hold label until a human removes it. Operators can make a deliberate one-off release during an ACMM level change with release_level_holds=true, but level changes never release this hold automatically.

@netlify

netlify Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for endusers ready!

Name Link
🔨 Latest commit 332ebbb
🔍 Latest deploy log https://app.netlify.com/projects/endusers/deploys/6ad6234851858356ae76b5e6
😎 Deploy Preview https://deploy-preview-1636--endusers.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@mrbobbytables

Copy link
Copy Markdown
Member

Priority: P3. This adds an E2E harness reachability guard; no current user-facing failure is reported.

@mrbobbytables mrbobbytables left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fixture-to-spec guard only scans immediate children of tests/e2e via readdirSync(specDir). Playwright collects nested specs recursively, so a valid nested spec could be the only consumer of a fixture build while this test reports it as orphaned. This is the same non-recursive spec-discovery pitfall corrected in #1585/#1586. Please use the repository’s recursive spec discovery pattern and add a nested-spec fixture case before merging.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
@mrbobbytables

Copy link
Copy Markdown
Member

Updated this branch in 332ebbb: spec discovery now uses recursive readdirSync paths, and a temp-directory regression test proves a nested .spec.js file is included while a non-spec is excluded. Focused test passes (3/3), Prettier passes, and npm run check:duplication passes. Required CI is running; the PR remains unmerged and its hold label is unchanged.

@mrbobbytables mrbobbytables left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The recursive discovery fix and nested-spec regression test address my request. The updated branch passes the focused test, duplication gate, and required CI checks. I am leaving the existing hold label in place; this is not a request to merge.

@mrbobbytables
mrbobbytables added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit 4426437 Oct 19, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant