test: add 15 PHP security scanner fixtures #150

Workflow file for this run

.github/workflows/validate-test-cases.yml at 4f27502

	name: corpus safety rules

	# scripts/validate-test-case.sh encodes the five safety rules that keep this
	# repository "annotated, inert vulnerability patterns" rather than something
	# that actually does harm, plus a check that the answer key still describes the
	# corpus. Until now nothing ran it. Both it and the catalog regeneration were
	# unticked boxes in .github/pull_request_template.md, which is to say the rules
	# were enforced by whether a contributor remembered them.
	#
	# They did not, and the failure was quiet in the way honour-system checks
	# usually are. PR #161 arrived with a catalog that still described 60 test
	# cases and knew nothing about the three it added -- a scanner scored against
	# that answer key would have been marked on a key missing the questions -- and
	# with host literals in RFC 5737 documentation addresses where rule 2 asks
	# for a reserved name. Both are exactly what the script checks, and both
	# reached review because the only workflow here was the ThreatCrush scan, which
	# has nothing to say about either.
	#
	# This runs on push as well as on pull_request on purpose. Catalog conflicts
	# are structural here: every test-case PR rewrites the generated index whole,
	# so any two open at once conflict, and the resolution happens at merge time
	# where no PR check is watching. A merge resolved by hand in the GitHub UI can
	# land a catalog that matches neither side, and master is the one place that
	# must never be wrong.

	on:
	pull_request:
	push:
	branches: [master]

	# Read-only: this job inspects the tree and writes nothing back.
	permissions:
	contents: read

	concurrency:
	group: corpus-safety-${{ github.ref }}
	cancel-in-progress: true

	jobs:
	validate:
	name: Validate the test-case corpus
	runs-on: ubuntu-latest
	timeout-minutes: 10

	steps:
	- uses: actions/checkout@v4

	# Pinned rather than taking whatever the runner image ships. The
	# generator is stdlib-only, so the version does not matter today -- but a
	# silent bump is the kind of thing that turns "the catalog is stale" into
	# a mystery, and this job's whole value is that its failures are legible.
	- uses: actions/setup-python@v5
	with:
	python-version: "3.12"

	- name: Check the safety rules
	run: bash scripts/validate-test-case.sh

	# The script names the fix ("run: python3 scripts/generate-catalog.py")
	# but not what the fix would change, and a contributor reading a red check
	# has no way to tell a forgotten regeneration from a genuine disagreement
	# about the corpus. Regenerate here and print the diff so the failure
	# explains itself. Nothing is committed -- the token is read-only and the
	# correction belongs in the contributor's branch, not in a bot commit that
	# would quietly hide how often this is missed.
	- name: Show what a regenerated catalog would change
	if: failure()
	run: \|
	python3 scripts/generate-catalog.py \|\| exit 0
	if git diff --quiet -- docs/VULNERABILITY_CATALOG.md vulns/VULNERABILITY_CATALOG.json; then
	echo "The catalog is current; the failure above is a safety rule, not catalog drift."
	exit 0
	fi
	{
	echo "### Catalog is stale"
	echo
	echo 'Run `python3 scripts/generate-catalog.py` and commit the result.'
	echo
	echo '```'
	git diff --stat -- docs/VULNERABILITY_CATALOG.md vulns/VULNERABILITY_CATALOG.json
	echo '```'
	} >> "$GITHUB_STEP_SUMMARY"
	git --no-pager diff -- docs/VULNERABILITY_CATALOG.md vulns/VULNERABILITY_CATALOG.json

Navigation Menu

Uh oh!

test: add 15 PHP security scanner fixtures #150

Workflow file

test: add 15 PHP security scanner fixtures #150

Uh oh!

Workflow file for this run