Run the shared-core gate per PHP version, off the commit job's path

The gate compiled turbo-ext against every supported PHP version on one
runner and took 28 minutes, which the commit job waited for. Each version
now compiles on its own leg and uploads a fingerprint of the result: a hash
per function, data section and relocation table (gate.py fingerprint). A
compare job checks them against the version the cores are compiled against
(gate.py compare). Like a core, a leg reuses the fingerprint of an earlier
run from identical build inputs (turbo-origins), so a push that does not
touch the extension only runs the comparison.

The gate also covers the 8.3-8.5 thread-safe builds now, since the same
core serves them, and reports a function that exists in only one version's
build. The commit job no longer waits for it: like the other jobs that only
test the shipped binaries, a failure is reported but does not stop
publication. check-linux.sh runs both halves locally.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017h1GfB7vfJPgwcTHD9Fth6

Author: ondrejmirtes

.github/scripts/find-turbo-origins.sh

@@ -1,12 +1,13 @@
 #!/usr/bin/env bash
 # Finds, for every shared core the turbo-compile-core jobs in phar.yml
-# produce (phpstan_turbo_core-<platform>), the newest earlier run on $BRANCH
-# that compiled it from build inputs identical to HEAD's, and writes
-# {"<artifact name>": <run id>} to the `origins` step output. A core leg
-# whose artifact is listed downloads it from that run instead of building
-# (and profiling) it again (see the turbo-origins job in phar.yml). The
-# per-PHP-version extensions are not reused: they compile in a minute,
-# against whichever core this run carries.
+# produce (phpstan_turbo_core-<platform>) and every fingerprint the
+# turbo-shared-core-gate legs produce (turbo-gate-fingerprint-<version>),
+# the newest earlier run on $BRANCH that compiled it from build inputs
+# identical to HEAD's, and writes {"<artifact name>": <run id>} to the
+# `origins` step output. A leg whose artifact is listed downloads it from
+# that run instead of building (and profiling) it again (see the
+# turbo-origins job in phar.yml). The per-PHP-version extensions are not
+# reused: they compile in a minute, against whichever core this run carries.
 #
 # The search walks HEAD's first-parent history (on a pull request's merge
 # commit, the base branch) and asks the API for the runs of each commit
@@ -48,11 +49,12 @@ BUILD_INPUT_PATHS=(
 	.github/scripts/install-alpine-php.sh
 	.github/scripts/install-php86-windows.sh
 	.github/scripts/download-php-windows-devel.sh
+	turbo-ext/bin/shared-core
 )
 WORKFLOW_PATH=".github/workflows/phar.yml"
-# Workflow-level env and the three core compile jobs, as JSON: comment-only
-# edits do not change it.
-COMPILE_JOBS_QUERY='[.env, .jobs["turbo-compile-core"], .jobs["turbo-compile-core-musl-arm64"], .jobs["turbo-compile-core-windows"]]'
+# Workflow-level env, the three core compile jobs and the gate legs, as
+# JSON: comment-only edits do not change it.
+COMPILE_JOBS_QUERY='[.env, .jobs["turbo-compile-core"], .jobs["turbo-compile-core-musl-arm64"], .jobs["turbo-compile-core-windows"], .jobs["turbo-shared-core-gate"]]'
 # Commits whose runs are looked up through the API, newest first. The
 # commits git alone rules out do not count.
 COMMITS_LIMIT=50
@@ -126,7 +128,7 @@ while read -r sha; do
 		fi
 		if ! origins="$(jq -c --argjson run "$run_id" --arg artifacts "$artifacts" '
 			($artifacts | split("\n")) as $names
-			| reduce ($names[] | select(startswith("phpstan_turbo_core-"))) as $name (.;
+			| reduce ($names[] | select(startswith("phpstan_turbo_core-") or startswith("turbo-gate-fingerprint-"))) as $name (.;
 				if has($name) or ($names | any(. == "turbo-reused-" + $name)) then . else .[$name] = $run end)
 			' <<< "$origins")"; then
 			origins='{}'

.github/workflows/phar.yml

@@ -350,7 +350,8 @@ jobs:
     # (.github/scripts/find-turbo-origins.sh has the exact conditions), and
     # its turbo-compile-core leg skips the build altogether: it runs on
     # ubuntu-latest, outside any build container, and only carries that core
-    # over into this run. The per-version extensions always compile (a
+    # over into this run. The same goes for each turbo-shared-core-gate
+    # leg's fingerprint. The per-version extensions always compile (a
     # minute each) against it, then go through the version check and the
     # differential tests of the turbo-differential jobs like any build —
     # they compare the native side with the PHP side of this commit, which
@@ -1030,31 +1031,114 @@ jobs:
     # compile to identical code against each supported version's — a source
     # reading a member whose offset moved, a constant whose value changed or
     # a function whose signature did would otherwise misbehave on the other
-    # versions only, on the paths that touch it. Compiles every source
-    # against each version's (and the 8.6 thread-safe) headers in the CI
-    # build images with clang, and fails on any difference outside the
-    # version-specific sources (turbo-ext/bin/shared-core/check-linux.sh).
+    # versions only, on the paths that touch it. Each leg compiles every
+    # source against one version's headers in its CI build image with clang
+    # and uploads a fingerprint of the result (a hash per function, data
+    # section and relocation table, see turbo-ext/bin/shared-core/gate.py);
+    # turbo-shared-core-gate-compare compares them. A leg whose fingerprint
+    # turbo-origins found in an earlier run from identical build inputs only
+    # carries it over, like a reused core.
+    needs: turbo-origins
     runs-on: "ubuntu-latest"
-    timeout-minutes: 40
+    timeout-minutes: 30
+
+    permissions:
+      contents: read # actions/checkout of this repository
+      actions: read # downloads a reused fingerprint from an earlier run, see turbo-origins
+
+    env:
+      GATE_ARTIFACT: "turbo-gate-fingerprint-${{ matrix.php-version }}"
+      ORIGIN_RUN_ID: ${{ fromJSON(needs.turbo-origins.outputs.origins || '{}')[format('turbo-gate-fingerprint-{0}', matrix.php-version)] }}
+
+    strategy:
+      fail-fast: false
+      matrix:
+        # every PHP the cores run on; thread-safe builds are "<minor>-zts"
+        php-version: ["8.3", "8.4", "8.5", "8.6", "8.3-zts", "8.4-zts", "8.5-zts", "8.6-zts"]
 
     steps:
       - name: Harden the runner (Audit all outbound calls)
         uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
         with:
           egress-policy: audit
 
+      - name: "Download the fingerprint compiled from identical build inputs"
+        if: env.ORIGIN_RUN_ID != ''
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: ${{ env.GATE_ARTIFACT }}
+          path: "fingerprints"
+          run-id: ${{ env.ORIGIN_RUN_ID }}
+          github-token: ${{ github.token }}
+
+      - name: "Record the run that compiled the fingerprint"
+        if: env.ORIGIN_RUN_ID != ''
+        run: echo "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$ORIGIN_RUN_ID" | tee turbo-reused.txt
+
       - name: "Checkout"
+        if: env.ORIGIN_RUN_ID == ''
         uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0
         with:
           persist-credentials: false
 
-      - name: "Compare the shared code across the supported PHP versions"
+      - name: "Compile against the PHP ${{ matrix.php-version }} headers and fingerprint the result"
+        if: env.ORIGIN_RUN_ID == ''
         env:
           SHARED_CORE_WORK_DIR: ${{ runner.temp }}/shared-core
+          GATE_VERSIONS: ${{ matrix.php-version }}
         run: |
-          mkdir -p "$SHARED_CORE_WORK_DIR"
-          bash turbo-ext/bin/shared-core/check-linux.sh
+          mkdir -p "$SHARED_CORE_WORK_DIR" fingerprints
+          GATE_JOBS="$(nproc)" bash turbo-ext/bin/shared-core/fingerprint-linux.sh
+          cp "$SHARED_CORE_WORK_DIR/fingerprints/fingerprint-$GATE_VERSIONS.tsv" fingerprints/
+
+      - name: "Upload the fingerprint"
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: ${{ env.GATE_ARTIFACT }}
+          path: "fingerprints/fingerprint-${{ matrix.php-version }}.tsv"
+          if-no-files-found: "error"
+
+      - name: "Mark the fingerprint as reused"
+        if: env.ORIGIN_RUN_ID != ''
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: "turbo-reused-${{ env.GATE_ARTIFACT }}"
+          path: "turbo-reused.txt"
+          if-no-files-found: "error"
+
+  turbo-shared-core-gate-compare:
+    name: "Turbo Shared Core Gate (compare)"
+    # Fails on any function, data section or relocation table outside the
+    # version-specific sources (Abi.cpp, Shadow.cpp, TrustedTypes.cpp,
+    # main.cpp) whose fingerprint differs from the build against the
+    # version the cores are compiled against (turbo-compile-core). Not a
+    # gate of the commit job, like the other jobs that only test the
+    # shipped bits (see there).
+    needs: turbo-shared-core-gate
+    runs-on: "ubuntu-latest"
+    timeout-minutes: 5
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
+        with:
+          egress-policy: audit
 
+      - name: "Checkout"
+        uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0
+        with:
+          persist-credentials: false
+          sparse-checkout: turbo-ext/bin/shared-core
+
+      - name: "Download the fingerprints"
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          pattern: "turbo-gate-fingerprint-*"
+          path: "fingerprints"
+          merge-multiple: true
+
+      - name: "Compare the shared code across the supported PHP versions"
+        run: python3 -I turbo-ext/bin/shared-core/gate.py compare 8.5 fingerprints
   turbo-docker-run:
     name: "Run with Turbo Extension (Docker PHP builds)"
     needs: "turbo-compile"
@@ -2491,7 +2575,8 @@ jobs:
     # guards the version constant the staging logic compares. The jobs
     # that only test those bits are deliberately not gates: the
     # "Run with Turbo Extension" jobs (turbo-differential,
-    # turbo-differential-musl, turbo-run, turbo-docker-run) and
+    # turbo-differential-musl, turbo-run, turbo-docker-run), the shared-core
+    # gate (turbo-shared-core-gate, turbo-shared-core-gate-compare) and
     # turbo-phpize, whose build nothing ships (PIE
     # builds from the phpstan/turbo-ext subsplit). They produce nothing the
     # commit consumes, waiting for them would delay every dev phar by their
@@ -2503,7 +2588,6 @@ jobs:
     needs:
       - compiler-tests
       - turbo-version
-      - turbo-shared-core-gate
       - turbo-compile
       - turbo-compile-musl-arm64
       - turbo-compile-windows

turbo-ext/CLAUDE.md

@@ -335,8 +335,9 @@ every supported PHP (README.md, "Shared core"). In those sources:
   functions (never data), and the core never calls them.
 
 Run `turbo-ext/bin/shared-core/check-linux.sh` (needs Docker) after touching
-engine-facing code; CI's `turbo-shared-core-gate` fails on any difference it
-finds, and never dismiss one without naming its mechanism.
+engine-facing code; CI's `turbo-shared-core-gate-compare` fails on any
+difference it finds (without holding up the phar commit, so watch for it),
+and never dismiss one without naming its mechanism.
 
 ## Zend-level gotchas (each of these cost real debugging time)
 

turbo-ext/README.md

@@ -475,10 +475,18 @@ SHARED_CORE_WORK_DIR=/tmp/shared-core turbo-ext/bin/shared-core/check-linux.sh
 ```
 
 compiles every source against each supported version's headers (thread-safe
-8.6 included) in the CI build images with clang, and fails on any function,
-data section or relocation that differs outside the four version-specific
-objects. It runs as `turbo-shared-core-gate` in `phar.yml`, and the commit
-job waits for it. clang, not the GCC the Linux binaries ship with: the
+builds included) in the CI build images with clang, and fails on any
+function, data section or relocation that differs from the build against the
+version the cores are compiled against (8.5, `GATE_REFERENCE`) outside the
+four version-specific objects. It is two halves: `fingerprint-linux.sh`
+compiles for the versions in `GATE_VERSIONS` and writes a fingerprint per
+version (`gate.py fingerprint`: a hash per function, data section and
+relocation table), and `gate.py compare` compares them. In `phar.yml` the
+first half runs as one `turbo-shared-core-gate` leg per version — a leg
+reuses the fingerprint of an earlier run with identical build inputs, like a
+core — and the second as `turbo-shared-core-gate-compare`. The commit job
+does not wait for them: like the other test jobs, a failure is reported
+loudly but does not stop publication. clang, not the GCC the Linux binaries ship with: the
 property proven is the source's (does it mean the same against every header
 set?), and GCC's register allocation drifts with incidental header spelling
 (a `const` added to a helper's parameter) where clang's output does not.
@@ -492,7 +500,8 @@ A failure lists the functions. Find the cause with the tools next to it —
 `gate-linux.sh` (any image/compiler, `GATE_VERSIONS`, `GATE_MAKE_ARGS`;
 `PGO_FLAGS='-fno-inline -fno-ipa-sra -fno-ipa-cp -fno-partial-inlining'`
 keeps each inline helper a function of its own, so the differing one shows
-up by name), `compare-functions.py`, `compare-data-linux.sh`, and their
+up by name), `compare-functions.py` (the per-function comparison
+`gate.py` builds on, with a size summary per object), and their
 macOS counterparts (`gate-mac.sh`, `compare-data-mac.sh`,
 `link-split-mac.sh`, with Homebrew's `php@8.x`); `link-split-linux.sh` links
 a core from one version's objects with every version's extension and runs
@@ -505,8 +514,10 @@ Windows legs' tests are their check.
 ### Supporting a new PHP version
 
 1. **Gate it.** Build the new version's CI image (`.github/turbo-build`),
-   add the version to `VERSIONS` in `bin/shared-core/check-linux.sh` and run
-   it. Every function it reports reads something that changed; for each,
+   add the version (and its `-zts` twin) to `VERSIONS` in
+   `bin/shared-core/check-linux.sh` and `bin/shared-core/fingerprint-linux.sh`
+   and to the `turbo-shared-core-gate` matrix in `phar.yml`, and run
+   `check-linux.sh`. Every function it reports reads something that changed; for each,
    find what (diff the headers' struct layouts — `sizeof`/`offsetof` of the
    structs the function touches — macro values, enum numbering and
    function declarations between the old and the new version) and route it
@@ -531,7 +542,10 @@ Windows legs' tests are their check.
    `turbo-compile-core`'s targets, `["8.3"]` with the vs16 toolset in
    `turbo-compile-core-windows`). Run the smoke test of every version
    against one core (`link-split-linux.sh`); in CI the
-   `turbo-shared-core-gate` job reruns step 1.
+   `turbo-shared-core-gate` legs rerun step 1. When the version the cores
+   are compiled against moves, move the reference with it (`GATE_REFERENCE`
+   in `check-linux.sh`, the version passed to `gate.py compare` in
+   `turbo-shared-core-gate-compare`).
 5. **Dropping a version** is the reverse of step 4 plus
    `TURBO_RETIRED_BINARIES` for the files phpstan/phpstan should delete. If
    it is the version a core is pinned to, move that pin to a version still

turbo-ext/bin/shared-core/check-linux.sh

@@ -1,68 +1,32 @@
 #!/bin/bash
-# The shared-core gate (phar.yml turbo-shared-core-gate, and locally): the
-# core is compiled once per platform and runs on every supported PHP, so
-# every source but the version-specific ones (the Makefile's SHIM_SOURCES)
-# must compile to the same code against each version's headers — thread-safe
-# ones included. Builds them all in the CI images with clang (whose output
-# does not drift with header spelling the way GCC's register allocation
-# does; the property proven is the source's, and the GCC build ships it),
-# then fails on any function, data section or relocation that differs
-# outside Abi.o, Shadow.o, TrustedTypes.o and main.o.
+# The shared-core gate, locally: the core is compiled once per platform and
+# runs on every supported PHP, so every source but the version-specific ones
+# (the Makefile's SHIM_SOURCES) must compile to the same code against each
+# version's headers — thread-safe ones included. Builds them all in the CI
+# images with clang (whose output does not drift with header spelling the
+# way GCC's register allocation does; the property proven is the source's,
+# and the GCC build ships it), then fails on any function, data section or
+# relocation that differs from the reference version's build outside Abi.o,
+# Shadow.o, TrustedTypes.o and main.o.
+#
+# phar.yml's turbo-shared-core-gate runs the same two halves on separate
+# machines: one fingerprint-linux.sh leg per version, then gate.py compare.
+# The reference is the version the cores are compiled against
+# (turbo-compile-core).
 #
 # Needs docker and an llvm-objdump (LLVM_OBJDUMP, else the newest found).
-# Usage: SHARED_CORE_WORK_DIR=<scratch directory> check-linux.sh
+# Usage: SHARED_CORE_WORK_DIR=<scratch directory> [GATE_VERSIONS=...] [GATE_REFERENCE=8.5] check-linux.sh
 set -euo pipefail
 
 S=${SHARED_CORE_WORK_DIR:?set SHARED_CORE_WORK_DIR to a scratch directory}
 HERE=$(cd "$(dirname "$0")" && pwd)
-VERSIONS=${GATE_VERSIONS:-8.3 8.4 8.5 8.6 8.6-zts}
-SHIM_OBJECTS='src/(Abi|Shadow|TrustedTypes|main)\.o'
-if [ -z "${LLVM_OBJDUMP:-}" ]; then
-	LLVM_OBJDUMP=$(command -v llvm-objdump || { ls /usr/bin/llvm-objdump-* /opt/homebrew/opt/llvm/bin/llvm-objdump 2>/dev/null || true; } | sort -V | tail -1)
-fi
-export LLVM_OBJDUMP
-[ -x "$LLVM_OBJDUMP" ] || { echo "no llvm-objdump found (set LLVM_OBJDUMP)"; exit 2; }
-
-# the CI build images plus clang, built once per run
-for v in $VERSIONS; do
-	printf 'FROM ghcr.io/phpstan/turbo-build:gnu-php%s\nRUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq clang-14 > /dev/null && rm -rf /var/lib/apt/lists/*\n' "$v" \
-		| docker build -q ${GATE_PLATFORM:+--platform "$GATE_PLATFORM"} -t "turbo-build-clang:gnu-php$v" - > /dev/null
-done
-
-# -fgnuc-version=4.3: the headers before 8.6 enable ZEND_COLD for GCC 4.3+
-# only and clang reports 4.2 by default; -U_FORTIFY_SOURCE: glibc's fortify
-# wrappers then expect GCC builtins clang lacks. Neither matters to the
-# property — they keep the header sets' compiler-identity branches alike.
-GATE_IMAGE=turbo-build-clang \
-GATE_VERSIONS="$VERSIONS" \
-GATE_MAKE_ARGS="CXX=clang++-14 PGO_FLAGS='-fgnuc-version=4.3 -U_FORTIFY_SOURCE'" \
-	"$HERE/gate-linux.sh"
+REFERENCE=${GATE_REFERENCE:-8.5}
+VERSIONS=${GATE_VERSIONS:-8.3 8.4 8.5 8.6 8.3-zts 8.4-zts 8.5-zts 8.6-zts}
+case " $VERSIONS " in
+	*" $REFERENCE "*) ;;
+	*) VERSIONS="$REFERENCE $VERSIONS" ;;
+esac
 
-cd "$S"
-first=$(echo "$VERSIONS" | awk '{print $1}')
-failed=0
-for v in $VERSIONS; do
-	[ "$v" = "$first" ] && continue
-	python3 -I "$HERE/compare-functions.py" "lx-$first" "lx-$v" > /dev/null
-	functions=$(grep -v -E "$SHIM_OBJECTS" "diff-lx-$v.txt" || true)
-	data=$("$HERE/compare-data-linux.sh" "lx-$first" "lx-$v" | grep -v -E "$SHIM_OBJECTS" || true)
-	if [ -n "$functions$data" ]; then
-		failed=1
-		echo "::error::shared code compiles differently against PHP $first and PHP $v headers"
-		[ -n "$functions" ] && echo "$functions" | while read -r size object name; do
-			echo "  $object: $(echo "$name" | c++filt) ($size bytes)"
-		done
-		[ -n "$data" ] && echo "$data" | sed 's/^/  /'
-	else
-		echo "PHP $first vs PHP $v: shared code identical"
-	fi
-done
-if [ "$failed" = 1 ]; then
-	cat <<'EOF'
-A source outside the version-specific ones reads something whose layout,
-value or signature differs between PHP versions. Route it through pt_abi
-(abi.h, filled by Abi.cpp) or move the code into a version-specific source;
-turbo-ext/README.md ("Shared core") lists the mechanisms.
-EOF
-fi
-exit "$failed"
+rm -rf "$S/fingerprints"
+GATE_VERSIONS="$VERSIONS" "$HERE/fingerprint-linux.sh"
+python3 -I "$HERE/gate.py" compare "$REFERENCE" "$S/fingerprints"

turbo-ext/bin/shared-core/compare-data-linux.sh

@@ -108,4 +108,5 @@ def main():
 		print(f'identical across all {len(dirs)}: {allsame/1048576:.2f} MB ({100*allsame/total:.1f}%)')
 
 
-main()
+if __name__ == '__main__':
+	main()