tests/scripts/test_check_site.py calls hugo with no guard for its absence, so agent-record is RED on main and on every pull request branched from it.
What happens
SiteGuardTests::test_rendered_benchmark_index_links_resolve_to_emitted_pages (added by #1714) runs:
result = subprocess.run(["hugo", "--minify", "-s", str(ROOT / "website"), ...])
subprocess.run raises FileNotFoundError when the program does not exist. It does not return a non-zero code that assertEqual(result.returncode, 0) could report — it raises before the assertion, which unittest records as an ERROR:
ERROR: test_rendered_benchmark_index_links_resolve_to_emitted_pages (__main__.SiteGuardTests.test_rendered_benchmark_index_links_resolve_to_emitted_pages)
Traceback (most recent call last):
FileNotFoundError: [Errno 2] No such file or directory: 'hugo'
FAILED (errors=1)
##[error]Process completed with exit code 1.
The agent-record job's runner is a plain ubuntu-latest and installs no Hugo, so the step exits 1 and the job goes red.
It is red on main, not only on branches
Every branch that merges current main inherits it. It is not attributable to the branch that reports it, which is the property that makes this expensive: a red that every pull request carries trains readers to skip the job that carries it.
Why it passes locally and fails in CI
The failure is environmental, not logical. On a developer box with Hugo on PATH the test runs and passes; the runner has no Hugo, so the same commit is green locally and red in CI. Running the suite locally is therefore not evidence about this job, which is the second cost.
The precedent this contradicts
#1661 / #1662 (TEST-SHELLCHECK-SKIP-GUARD) landed the rule for exactly this shape: skip, do not error, when the binary is absent. A tool-dependent test that hard-errors on a missing tool reports an infrastructure gap as a code verdict. The repair should follow that file's guard rather than invent a second spelling.
Required repair
Guard the Hugo invocation the way test_shellcheck_skip_guard guards shellcheck: resolve the binary first (shutil.which("hugo")) and self.skipTest(...) with a message naming the absent tool when it is missing. Do not delete the assertion, and do not widen it — when Hugo IS present the test must still hold the rendered benchmark index links to the emitted pages, which is the guarantee #1714 added it for.
A red-before / green-after mutation is owed on the guard itself: a skip guard that never skips, and a skip guard that always skips, must be distinguishable, or the guard becomes a mute switch on the assertion it protects.
Owner
Filed by the LTX25-DIT-ATTN-FA2-HD128 session (#1551), which hit it as an inherited red. It is a one-line gate repair and a unit of work in its own right, so it is fixed on its own branch rather than folded into that row's pull request.
tests/scripts/test_check_site.pycallshugowith no guard for its absence, soagent-recordis RED onmainand on every pull request branched from it.What happens
SiteGuardTests::test_rendered_benchmark_index_links_resolve_to_emitted_pages(added by #1714) runs:subprocess.runraisesFileNotFoundErrorwhen the program does not exist. It does not return a non-zero code thatassertEqual(result.returncode, 0)could report — it raises before the assertion, whichunittestrecords as an ERROR:The
agent-recordjob's runner is a plainubuntu-latestand installs no Hugo, so the step exits 1 and the job goes red.It is red on
main, not only on branchesmain@1db7e59cfc9d815f25f7cdf96e08740aa1c142e4, run 32586062675, job 97066276883agent-record: failure, with the text above.6b37934b8, run 32594387836, job 97082975595: failure, byte-identical text.Every branch that merges current
maininherits it. It is not attributable to the branch that reports it, which is the property that makes this expensive: a red that every pull request carries trains readers to skip the job that carries it.Why it passes locally and fails in CI
The failure is environmental, not logical. On a developer box with Hugo on
PATHthe test runs and passes; the runner has no Hugo, so the same commit is green locally and red in CI. Running the suite locally is therefore not evidence about this job, which is the second cost.The precedent this contradicts
#1661 / #1662 (
TEST-SHELLCHECK-SKIP-GUARD) landed the rule for exactly this shape: skip, do not error, when the binary is absent. A tool-dependent test that hard-errors on a missing tool reports an infrastructure gap as a code verdict. The repair should follow that file's guard rather than invent a second spelling.Required repair
Guard the Hugo invocation the way
test_shellcheck_skip_guardguardsshellcheck: resolve the binary first (shutil.which("hugo")) andself.skipTest(...)with a message naming the absent tool when it is missing. Do not delete the assertion, and do not widen it — when Hugo IS present the test must still hold the rendered benchmark index links to the emitted pages, which is the guarantee #1714 added it for.A red-before / green-after mutation is owed on the guard itself: a skip guard that never skips, and a skip guard that always skips, must be distinguishable, or the guard becomes a mute switch on the assertion it protects.
Owner
Filed by the
LTX25-DIT-ATTN-FA2-HD128session (#1551), which hit it as an inherited red. It is a one-line gate repair and a unit of work in its own right, so it is fixed on its own branch rather than folded into that row's pull request.