Shard itest ingot across three runners, and record what it does not fix

[Peeja]

⏭️ Safe to merge without waiting for CI — all of it, at this head

Refreshed on every push. Checks are not required yet, so this is advice, not a gate.

check	why this diff cannot reach it
itest ×4	Already proven green on f4c5c21f (run 35240022926, 4/4). git diff f4c5c21f HEAD -- .github/ is empty — every commit since changes only MONOREPO_TODO.md.
ci (guards + 13 unit)	Reads neither changed file. No Go package imports itest.yml; no guard script parses it.
images ×8	Builds from Dockerfiles and images.yml; touches neither changed file.
e2e	Has its own e2e.yml; does not read itest.yml.

Derived, not asserted. grep -rn 'itest.yml\|MONOREPO_TODO' --include='*.yml' --include='*.sh' --include='*.go' --include=Makefile . returns three hits, all of them prose in comments (e2e.yml:136, check-base-images.sh:12, forge_encryption_test.go:600). Nothing consumes either file.

What this costs if it's wrong: the checks still run; we just don't wait. If an ignored check goes red after merge, main is red. That is the trade, and it is why the reasoning is re-derived per push rather than encoded as a paths: list that goes stale.

---

itest ingot was ~28 minutes and had become a drag on every PR. Two files, based on main, stacked on nothing.

Does not need rebasing despite main having moved to 586738ed: this touches only itest.yml and MONOREPO_TODO.md, and #17 and #20 touched neither.

✅ It ran, and it did less than I predicted

Run 35240022926 is green on all four jobs. And #19's unsharded run finished four minutes later on the same runner pool, so this is a real A/B rather than a comparison against last night's number:

	unsharded (#19, run 35239716032)	sharded (this, run 35240022926)
itest workflow, end to end	30m43s	23m07s
slowest job	29m29s itest ingot	17m07s itest ingot 1/3
test binary	ok …/ingot/itest 1267.853s	628.841s + 462.731s + 204.654s
runner-minutes, ingot side	29m29s	42m04s

~25% off the wall clock, not the "roughly half" I claimed when I opened this. 6fb033cd replaces the estimates in MONOREPO_TODO.md with the measurement; the workflow itself is unchanged from the green run.

Total test work is unchanged: 1296.2s against 1267.9s, +2.2%. That is the point rather than a disappointment — nothing got cheaper, it got spread — but it bounds every remaining option.

Where the other half went

1. The shards are unbalanced — 629s / 463s / 205s — and the critical path is the slowest, not the mean. The round-robin splits by test name order, which silently assumes uniform cost. Shard 1 drew TestForgeVersity (hundreds of subtests, dozens of them 3-second object-lock retention waits) plus TestForgeMultipartExpiryShred and TestForgeReadAfterEviction, both TTL-bound. Shard 3 drew four cheap ones. Perfect balance would be 432s, so imbalance alone costs 3m17s.
2. Queue wait of 2m25s–4m47s per shard, 4m47s on the critical one. Four concurrent jobs where there were two, and they did not all get runners at once. Pure loss, and it grows with shard count.
3. The ~6 minute image build is per job and did not move — sharding triplicated it.

It also falsifies the boot arithmetic I gave you twice

Shard 3 ran four tests, so four boots, in 204.654s. A boot is therefore at most ~51s, not the ~80s I derived from 1257/13 — so the "1040s booting, 217s working" decomposition was wrong, and the shared stack is worth less than the 3–4 minutes I last estimated, not more. Parked: it changes test isolation in the one job that exists to catch flakiness, for less than was claimed.

Two levers nobody had looked at (e0205346)

Both found by reading source after the measurement pointed at them:

• itest and e2e have no Docker layer cache, and never decided not to. itest.yml:143 and e2e.yml:118 shell out to plain docker build, which cannot use --cache-from type=gha at all; only images.yml uses buildx. The "No caching, deliberately (2026-09-11)" comment is in images.yml and reasons about that workflow — "the point is to provoke build failures, not to avoid them". That does not obviously carry where the build is a means to running tests, and images.yml already proves the cold build on the same commit on every PR. Keep images.yml uncached as the canary, cache the other two: up to ~6 min off each of four jobs, for a handful of lines.
• lockWaitTime is 3s in our own versitygw fork and is self-imposed. tests/integration/utils.go:2654; cleanupLockedObjects sets RetainUntilDate: now + lockWaitTime and then sleeps that long waiting for the lock it just created. 38 call sites ≈ 114s of pure sleep, inside TestForgeVersity, the test that bounds the job. 3s → 1s saves ~76s; 1s is the floor until someone checks sub-second retention round-trips. Lands in versitygw, arrives here as a pin bump.

Then, in order: build-once-and-load (same ~6 min from the other side, 1–2 GB of unmeasured round-trip — try the cache first); balance the shards by measured duration (~3m17s, derived from prior timings, never hand-grouped).

The shards derive their own tests

all=$(go test -list '^Test' ./... | grep '^Test' | sort)
mine=$(printf '%s\n' "$all" | awk -v k="$SHARD" -v s="$SHARDS" 'NR % s == k % s')

That distinction is the whole design. A hand-written -run regex per shard is a list; a test added later falls out of it, and it falls out silently — a green job that ran nothing looks exactly like a green job that ran everything. Deriving it means a new test lands in a shard by itself. An empty selection fails the job loudly rather than passing.

shards: 1 selects everything (NR % 1 is always 0), so hilt takes the same code path unsharded — no second branch to keep correct.

Separate jobs also give separate Docker hosts, which stack.CleanupLeaked requires: it removes every smeltery- container, so two suites sharing a host tear each other down. That constraint is why this is more jobs rather than more parallelism inside one.

⚠️ This changes check names

itest ingot becomes itest ingot 1/3, 2/3, 3/3. A branch protection rule naming the old one will wait on a check that never reports — the same edge the replaces → guards rename had. itest hilt is unchanged.

The trade, stated plainly

~43% more runner-minutes for ~25% less wall clock. Right while pull requests are the bottleneck; wrong if runner-minutes ever become the constraint. Every service brought in-repo adds another image build to the fixed half.

Verified

Three shards split 13 tests 5/4/4; union equals the full list; no test in two shards; shards: 1 selects all 13; actionlint clean. Checked by running the real go test -list output through the same awk — and now confirmed by a green run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt

[Peeja]

LGTM; merging when green.

[Peeja]

Doesn't seem to be the big win I'd hoped. #22 will suffice for now. We may return to this later, or try a different approach.