compat: test against what the polyrepos actually shipped

.github/workflows/ci.yml

@@ -70,6 +70,8 @@ jobs:
         run: .github/scripts/check-goreleaser-ldflags.sh
       - name: the release version assertion fails on a dead ldflag, and passes on a live one
         run: .github/scripts/check-assert-released-version.sh
+      - name: the compat baseline resolver refuses rather than reporting no releases
+        run: .github/scripts/check-compat-baselines.sh
       - name: gofmt
         run: |
           unformatted=$(gofmt -s -l .)
@@ -95,7 +97,7 @@ jobs:
           - service: ingot
           - service: sprue
           - service: smelt
-            tagged_suites: e2e
+            tagged_suites: e2e,compat
           - service: delegator
           - service: piri-signing-service
           - service: swarf
@@ -161,6 +163,9 @@ jobs:
       # import path in them stays green until someone runs the suite by hand.
       # Compiling them (not running them — they need Docker and a live stack)
       # is what keeps that from happening.
+      #
+      # The tag list is hand-maintained on purpose: a suite is added about
+      # never, so a guard deriving it would cost more than it saves.
       - name: vet tagged suites
         if: matrix.tagged_suites != ''
         run: go vet -tags ${{ matrix.tagged_suites }} ./...

.github/workflows/compat-refresh.yml

@@ -0,0 +1,124 @@
+name: compat-refresh
+
+# Keep every open release pull request's compat answer current.
+#
+# HOW: a scheduled run -- so it runs on `main`, the only ref a schedule can run
+# on -- dispatches compat.yml at each open `release/*` branch. A dispatched run
+# records its check runs against that branch's head commit, which is the pull
+# request's head commit, so the fresh answer replaces the check where the merge
+# decision is made.
+#
+# WHY THIS EXISTS AT ALL. compat.yml asks "does the release proposed here still
+# work against what is already deployed?" -- and the answer changes without the
+# pull request changing, because the deployed side is read live from the
+# registry. An upstream release lands or a `:main` moves, the baselines move
+# with them, and a release pull request that was green yesterday is no longer
+# known to be green. Nothing about the pull request triggers anything, so
+# without this its check is simply old.
+#
+# WHY NOT A SCHEDULE ON compat.yml ITSELF. That version would test `main`, and a
+# result about `main` has no decision attached to it: `main` is not deployed, so
+# a red there leaves you working out which pending release it implicates --
+# which is the work the release pull request was supposed to have done for you.
+#
+# WHAT IT GIVES UP: nothing runs while no release pull request is open. An
+# upstream incompatibility that lands in a quiet week is found when the next
+# release is proposed rather than when it appeared. Deliberate -- the earlier
+# signal had nobody to act on it -- but if the lead time turns out to matter,
+# the answer is a scheduled run that opens or annotates a release pull request,
+# not one that tests `main`.
+
+on:
+  schedule:
+    # 07:00 UTC daily -- after the US evening merges, before the EU morning.
+    - cron: '0 7 * * *'
+  workflow_dispatch:
+
+# `actions: write` is what `POST /actions/workflows/{id}/dispatches` requires,
+# and dispatching compat.yml is this workflow's whole job. It is also the scope
+# that would let this token start release.yml: `workflow_dispatch` is one of the
+# two events exempt from the rule that GITHUB_TOKEN-triggered events start no
+# run. GitHub has no narrower scope to ask for; what bounds this is that the
+# step below dispatches one named workflow at branches it derived by reading.
+# https://docs.github.com/en/actions/concepts/security/github_token
+permissions:
+  contents: read
+  pull-requests: read
+  actions: write
+
+concurrency:
+  group: ${{ github.workflow }}
+  cancel-in-progress: false
+
+jobs:
+  refresh:
+    name: Dispatch compat on every open release pull request
+    runs-on: ubuntu-24.04
+    timeout-minutes: 10
+    steps:
+      - name: Dispatch
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+        run: |
+          set -euo pipefail
+
+          # DERIVED from the open pull requests, never a list kept here: a
+          # release branch this file had not been told about would otherwise go
+          # unrefreshed, which is the failure mode that looks exactly like
+          # success.
+          #
+          # ASSIGNED AND TESTED, not piped into mapfile, because "could not ask"
+          # and "nothing to refresh" are different answers and the pipe form
+          # cannot tell them apart -- `mapfile < <(...)` discards the subshell's
+          # status, so a gh that cannot reach the API produces an empty list and
+          # a green run that refreshed nothing.
+          if ! raw=$(gh pr list --repo "$REPO" --state open --limit 100 \
+                       --json number,headRefName,headRefOid \
+                       --jq '.[] | select(.headRefName | startswith("release/"))
+                                 | "\(.number)\t\(.headRefName)\t\(.headRefOid)"'); then
+            echo "::error::could not list pull requests. Refusing to report that" \
+                 "as 'nothing to refresh': every open release pull request would" \
+                 "keep a stale green check and nobody would know." >&2
+            exit 1
+          fi
+
+          # `grep .` drops a blank line rather than letting one become an
+          # element the tally would then count.
+          mapfile -t prs < <(printf '%s' "$raw" | grep . || true)
+
+          if [ "${#prs[@]}" -eq 0 ]; then
+            echo "No open release/* pull request. Nothing to refresh."
+            exit 0
+          fi
+
+          failed=0
+          for row in "${prs[@]}"; do
+            num=${row%%$'\t'*}; rest=${row#*$'\t'}
+            branch=${rest%%$'\t'*}; sha=${rest##*$'\t'}
+            echo "::group::#$num ($branch @ ${sha:0:8})"
+
+            # Dispatched at the BRANCH, which runs compat.yml as that branch has
+            # it and at whatever its tip is now -- so a head that moved between
+            # the listing above and here is tested as it currently stands, not
+            # as it was. compat.yml is written for this: its job condition is
+            # `event_name != 'pull_request' || head_ref starts with release/`,
+            # so a dispatch satisfies it on any ref.
+            #
+            # EACH IS INDEPENDENT: a refusal is counted and the loop carries on,
+            # because aborting here would leave every later pull request untried
+            # on account of one.
+            if ! gh workflow run compat.yml --repo "$REPO" --ref "$branch"; then
+              echo "::error::#$num ($branch): could not dispatch compat." \
+                   "Its compat check still shows an earlier answer against" \
+                   "today's fleet." >&2
+              failed=$((failed + 1))
+            fi
+            echo "::endgroup::"
+          done
+
+          echo
+          echo "${#prs[@]} open release pull request(s): $failed could not be dispatched."
+
+          # Non-zero on a failure, so a stale check has somewhere to be seen.
+          [ "$failed" -eq 0 ]

.github/workflows/compat.yml

@@ -0,0 +1,200 @@
+name: compat
+
+# Does what we are about to ship still work against what is already deployed?
+#
+# `ci`, `itest` and `e2e` all boot this commit against itself, which is the
+# guarantee the monorepo gives for free and is a development-time one. The
+# field is a mix: we run sprue and hilt, third-party operators run piri and
+# ingot and upgrade on their own schedule. This workflow tests that skew.
+#
+# IT RUNS ON A RELEASE PULL REQUEST, AND NOWHERE ELSE ON ITS OWN. A pull request
+# from a `release/<svc>` branch is the one place the question has an answer
+# somebody acts on: red means do not merge this release. The same run against
+# `main` produces a result with no decision attached -- `main` is not deployed,
+# so a red there leaves you working out which pending release it implicates.
+#
+# It takes no service. Unlike release.yml it tests the whole stack against the
+# baseline set, so a release pull request for any service gets the same run;
+# the branch name is only the signal that a release is being proposed.
+#
+# THE ANSWER GOES STALE WITHOUT THE CODE CHANGING, which is why this file has no
+# `schedule:` and `compat-refresh.yml` does. The pinned side is read live from
+# the registry, so the same commit can face a different fleet tomorrow: an
+# upstream release lands or a `:main` moves, the baselines move with it, and a
+# release pull request that was green is no longer known to be. That workflow
+# dispatches THIS one at each open release branch, daily. A dispatched run
+# records its check runs against the branch's head commit, which is the pull
+# request's head commit, so the fresh answer lands where the merge decision is
+# made -- measured on run 35760677003, whose 11 check runs all landed on the
+# dispatched commit. It used to RE-RUN the pull request's own run instead,
+# which put the answer in the same place but expired 30 days after the
+# original; a dispatch has no such ceiling.
+#
+# Also on demand.
+#
+# THE PINNED SIDE IS THE POLYREPOS' IMAGES -- ghcr.io/fil-forge/<svc>. Those
+# are what ships to the network today, so they are what "already deployed"
+# means. This repository publishes no images (images.yml runs with push: false)
+# and carries no service-prefixed tags; neither is a prerequisite here, which
+# is why this can run before the release-ownership question is settled.
+#
+# THE BASELINE SET COMES FROM THE REGISTRY, not from git tags, because the two
+# disagree: the polyrepos tag git `v0.2.4` and publish the image `0.2.4`.
+# Reading the registry asks the question the test actually has -- "what can I
+# pin against?" -- and cannot drift from it.
+#
+# FIVE OF THE EIGHT HAVE CUT NO RELEASE, so their baseline is `:main` resolved
+# to a digest. This job used to skip when any service lacked one, which made
+# the check GREEN IN 0.126s HAVING BOOTED NOTHING -- the job ran because SOME
+# service could be pinned, the test needed EVERY service pinned, and a skipped
+# Go test exits 0. There is now no skip: the script produces a complete
+# baseline set or fails, and compat-baselines.sh's own header says what a
+# floating baseline is worth and what it is not.
+
+on:
+  # No `branches:` filter would help: it matches the BASE branch, and what
+  # selects a release here is the HEAD branch. The job condition below does it.
+  pull_request:
+  # No inputs. There was a `window` input, for a pinned-peer test that has
+  # since been cut; nothing read it.
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+# KEYED BY REF, which it was not until this trigger arrived. A group of just
+# `${{ github.workflow }}` was harmless while this ran only on a schedule and on
+# demand; with `pull_request:` it is not. GitHub holds one run per group and
+# cancels any PREVIOUSLY PENDING one when a newer arrives, so with a global key
+# a release pull request's hour-long compat run would make every other pull
+# request's compat run pend behind it -- and the third arrival would cancel the
+# second outright. A release pull request could have its compat check cancelled
+# by an unrelated pull request, which is the one thing this workflow must not do.
+# https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/control-the-concurrency-of-workflows-and-jobs
+#
+# `head_ref` for a pull request, `ref` for a dispatch; a re-run reuses its
+# original event, so it lands back in the same group.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  baselines:
+    name: Resolve the baselines
+    # A release pull request, or a dispatch. Every other pull request skips
+    # this job, and `compat` needs it, so both skip together -- measured on
+    # run 35788998286, an ordinary pull request, where both report `skipped`.
+    # That matters because the compat job below is capped at an hour.
+    #
+    # NOT "costs an ordinary pull request nothing", which this said until the
+    # concurrency note above was written: the run is still created and still
+    # joins the group. What it costs is a queue slot, which is why that group
+    # is now keyed by ref.
+    if: >-
+      github.event_name != 'pull_request' ||
+      startsWith(github.head_ref, 'release/')
+    runs-on: ubuntu-latest
+    # TWO JSON OBJECTS, not one entry per service. A hand-written list here was
+    # wrong in both available directions at once: it named `base_sprue`, which
+    # the test read as COMPAT_BASELINE_UPLOAD and never saw, and it omitted the
+    # four services the fleet gained. `outputs:` cannot be generated, so the
+    # thing that varies moves inside a value instead.
+    outputs:
+      baselines: ${{ steps.resolve.outputs.baselines }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Resolve baselines from the registry
+        id: resolve
+        run: .github/scripts/compat-baselines.sh >> "$GITHUB_OUTPUT"
+
+  compat:
+    name: Version skew
+    needs: baselines
+    # NO CONDITION, deliberately. There was one -- `pinnable == 'true'`, true
+    # when ANY service could be pinned -- and the test needs EVERY service
+    # pinned, so the job ran and the test skipped and the check went green
+    # having booted nothing. Two conditions that differ by one quantifier, one
+    # in YAML and one in Go, with a green check in the gap.
+    #
+    # Now the baselines job either produces a complete set or fails, so there
+    # is one place that decides and it is the one that reads the registry.
+    runs-on: ubuntu-24.04
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: go.work
+          cache-dependency-path: |
+            piri/go.sum
+            hilt/go.sum
+            sprue/go.sum
+            ingot/go.sum
+            smelt/go.sum
+
+      # Expanded here rather than listed as `env:` entries, for the reason the
+      # `outputs:` comment gives. The key is already the package name the test
+      # asks for, so this adds no mapping of its own -- it writes exactly what
+      # the registry answered.
+      - name: Expand the resolved versions
+        env:
+          BASELINES: ${{ needs.baselines.outputs.baselines }}
+        run: |
+          set -euo pipefail
+          python3 - <<'EOF' >> "$GITHUB_ENV"
+          import json, os, re
+          for k, v in json.loads(os.environ["BASELINES"]).items():
+              # The key comes from a package name the registry confirmed, so it
+              # cannot carry a newline -- but $GITHUB_ENV is parsed line by
+              # line, and a check here costs nothing.
+              assert re.fullmatch(r"[A-Z0-9_]+", k), k
+              assert "\n" not in v, v
+              print(f"COMPAT_BASELINE_{k}={v}")
+          EOF
+          echo "--- resolved ---"; grep '^COMPAT_' "$GITHUB_ENV"
+
+      # ASSERTED, not assumed. The line below says GOWORK stays on, and
+      # nothing checked it: with it off, workspace.Detect() fails and the
+      # suite used to skip green. It is a hard failure in the test now, but
+      # failing here says which knob rather than which symptom.
+      #
+      # TWO CONDITIONS, because `go env GOWORK` prints the LITERAL STRING
+      # `off` when GOWORK=off -- it is empty only when no go.work was found at
+      # all. A `[ -n "$w" ]` on its own therefore passes on exactly the case
+      # this step is written about, which is rule 5's partial guard that reads
+      # as a total one. pkg/workspace tests both (`gowork == "" || gowork ==
+      # "off"`); this now matches it.
+      - name: the workspace is active
+        run: |
+          set -euo pipefail
+          w=$(go env GOWORK)
+          if [ -z "$w" ] || [ "$w" = off ]; then
+            echo "::error::go env GOWORK is '${w:-<empty>}', so pkg/workspace has no fleet to build from HEAD and this suite cannot compare anything."
+            exit 1
+          fi
+          echo "GOWORK=$w"
+
+      # GOWORK stays ON: the services NOT pinned to a released image are built
+      # from HEAD by pkg/workspace, which is the whole point of the comparison.
+      - name: Run compatibility suite
+        working-directory: smelt
+        run: go test -tags compat -v -timeout 55m -parallel 1 ./tests/compat/...
+
+      - name: Dump container state on failure
+        if: failure()
+        run: |
+          mkdir -p /tmp/container-logs
+          for c in $(docker ps -a --format '{{.Names}}'); do
+            docker logs "$c" > "/tmp/container-logs/$c.log" 2>&1 || true
+            echo "::group::$c"; tail -200 "/tmp/container-logs/$c.log"; echo "::endgroup::"
+          done
+
+      - name: Upload container logs
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: compat-container-logs
+          path: /tmp/container-logs/
+          if-no-files-found: ignore
+          retention-days: 7

AGENTS.md

@@ -46,9 +46,17 @@ Four places, and putting something in the wrong one is how it gets lost:
 | **wiki → Needs Human Work** | what is blocked on a person, and decisions taken on their behalf awaiting confirmation | items leave via *Recently cleared* |
 | **`MAJOR_DECISIONS.md`** | decisions with a rationale someone will otherwise re-litigate — what is deliberately *out* of the monorepo, and why | permanent |
 | **`MONOREPO_TODO.md`** | questions only answerable once the monorepo is whole, and bugs found in imported code that are not the migration's to fix | until answered |
+| **`RELEASE.md`** | how a change becomes a release: the pull request path, versions, tagging, artifacts | permanent, and **kept tight** |
 
 Ordinary unfinished work belongs in issues, not in any of these.
 
+`RELEASE.md` is the one a reader is most likely to act on directly, so it earns
+a second rule: **it stays current and it stays short.** Change the release
+path, the version scheme, the tag scheme or what gets published, and update it
+in the same pull request. Resist growing it — reasoning about why the flow has
+its shape belongs in the workflow files, which already carry it; `RELEASE.md`
+says what to do.
+
 ## Update the wiki in the same turn as the thing it describes
 
 **Not "keep it current".** That phrasing has failed here more than once: the
@@ -191,11 +199,18 @@ GOWORK=off go build ./...  # inside a module: what CI does, standalone
 go build ./...             # workspace mode, across go.work
 ```
 
-CI is four workflows — `ci` (per-module build/vet/staticcheck/tidy/test plus
-the `guards` job), `images`, `e2e`, `itest`. A fifth, `release`, exists but is
-**dispatch-only and never runs on a push**, so it is not part of what a change
-costs. **Nothing is filtered by path, on purpose**: `ci.yml`'s header says why,
-and `MONOREPO_TODO.md` carries the question of whether that should change. A
+Seven workflows, of which **four run jobs on every pull request** — `ci`
+(per-module build/vet/staticcheck/tidy/test plus the `guards` job), `images`,
+`e2e`, `itest`. Of the other three, two cost an ordinary pull request nothing
+at all: `release` is **dispatch-only**, and `compat-refresh` runs on a
+schedule. `compat` is the one in between — it triggers on `pull_request:`, so
+a run is created and joins the concurrency group, but its first job is gated
+on a `release/*` head branch and its second `needs:` the first, so both report
+*skipped*. What that costs is a queue slot, which is why the group is keyed by
+ref; `compat.yml` says so at length.
+
+**Nothing is filtered by path, on purpose**: `ci.yml`'s header says why, and
+`MONOREPO_TODO.md` carries the question of whether that should change. A
 documentation-only change therefore costs a full run; that is known, not an
 oversight.