Skip to content
This repository was archived by the owner on Sep 17, 2026. It is now read-only.

Record CI wall clock, forgectl's mainnet metrics, and Phase 1's missing release flow - #23

Merged
Peeja merged 1 commit into
mainfrom
claude/todo-ci-speed
Sep 17, 2026
Merged

Peeja merged 1 commit into
mainfrom
claude/todo-ci-speed

Conversation

@Peeja

@Peeja Peeja commented Sep 17, 2026

Copy link
Copy Markdown

⏭️ Safe to merge without waiting for CI — all of it

Refreshed on every push. Checks are not required yet, so this is advice, not a gate.

One file changed: MONOREPO_TODO.md. Nothing in the repository reads it.

$ git diff --name-only origin/main HEAD
MONOREPO_TODO.md
$ grep -rn "MONOREPO_TODO" --include='*.yml' --include='*.sh' --include='*.go' --include=Makefile .
.github/scripts/check-base-images.sh:12:# not installed yet (MONOREPO_TODO.md), so that half is inert. This check is

One hit, and it is prose inside a comment — no workflow, guard, Go package or Makefile target consumes the file. ci, images, e2e and itest will all pass and none of them could have done otherwise.

What this costs if it's wrong: the checks still run; we just don't wait. If an ignored check goes red after merge, main is red.


Three entries. The first re-homes work that would otherwise exist only on a closed branch; the other two are findings from reading main while working out what comes next.

1. CI wall clock — the measurement survives the PR that produced it

#21 built test sharding, ran it green, and was closed for simplicity. Its A/B is the evidence base for every remaining option, so it belongs here rather than only on a closed branch and the wiki:

unsharded sharded (closed)
itest workflow 30m43s 23m07s
test binary 1267.853s 628.841 + 462.731 + 204.654s
runner-minutes 29m29s 42m04s

Total test work unchanged at +2.2% — nothing got cheaper, it got spread.

The live options are ordered, and each says what is measured versus assumed: the layer cache (open separately; needs re-measuring after one warm run, since the first run post-merge is cold by construction), versitygw's self-imposed 3s lockWaitTime across 38 call sites, build-once-and-load, and — if sharding is revived — bin-packing against measured durations rather than name order.

The shared stack is parked, and the arithmetic that made it look attractive is recorded as wrong: shard 3 ran four tests, so four boots, in 204.654s, which puts a boot at ≤~51s rather than the ~80s taken from 1257/13.

2. ⚠️ forgectl's mainnet metrics stop when the polyrepo is archived

forgectl is the only imported service that had genuinely scheduled workflows, and they are operational rather than CI:

workflow schedule runs
metrics-payments.yaml every 30 min forgectl metrics payments --payer … --otlp-endpoint …
metrics-faults.yaml every 12 h forgectl metrics faults …

Both target environment: mainnet and push to an OTLP endpoint, taking the payer address and endpoint from repository secrets.

Nothing is broken today — they run in fil-forge/forgectl, which still exists; the copies that came in with the subtree never ran. The hazard is the archival step: whenever that repo is archived or its workflows disabled, mainnet fault and payment metrics stop silently — a dashboard goes flat and nothing fails. Reproducing them here needs a mainnet environment and two secrets configured on this repository, which is a person's job. Sequence it with the archival, not after.

Checked: the seven services pruned earlier carry no equivalent risk. The only schedule: keys among every file that prune deleted were dependabot intervals.

3. Phase 1 needs machinery this repository does not have

The plan says to cut tags "via the existing release.yml flow". There is no release.yml — main has ci, e2e, images, itest, and none tags, releases or publishes. The plan treated old forge at f60dd59 as a starting point that already had that apparatus; this repo was built from subtree imports instead.

What survived is uneven: version.json in 8 of 10 (not forgectl, not smelt), .goreleaser.yaml in 4, Dockerfile.release in 4.

The Dockerfile.release files are goreleaser-shaped, not repo-shaped — COPY <svc> /usr/bin/<svc> copies a pre-built binary from goreleaser's dist/ context. So "nothing builds them" cannot be fixed by adding a CI job; the release flow has to exist first. hilt and swarf have one with no .goreleaser.yaml at all, so they're doubly orphaned.

And a trap worth seeing before it bites: at the monorepo root, hilt is a directory. COPY hilt /usr/bin/hilt is unambiguous in dist/, and copies an entire source tree if that file is ever built with the repository root as context — which is the context every other Dockerfile here uses.

The sequencing constraint the plan doesn't state

A submodule tag has to be <svc>/vX.Y.Z in the repository the module path names. Module paths are already github.com/fil-forge/forge/*, so a tag pushed to fil-forge/forge-2 sits where no module path resolves — go get github.com/fil-forge/forge/piri@piri/v1.2.3 looks in fil-forge/forge, the old repo, and finds nothing.

Any tag cut before the rename has to be cut again after it. The rename gates the tagging half of Phase 1. The tag scheme, the workflow itself and compat.yml do not wait on it.

Worth your eye

The forgectl metrics item is the one with a real-world consequence and a deadline attached to someone else's action, so it's the one I'd most want you to confirm I've read correctly. The rest is reconnaissance.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt


Generated by Claude Code

…ng flow

The first re-homes work that was born on #21 and would otherwise exist only on
a closed branch: the itest A/B measurement, and the two levers it turned up that
need no sharding at all. The other two are findings from reading main while
working out what comes next.

CI wall clock. The measurement is the evidence base for every remaining option,
so it stays even though the sharding that produced it was declined. The live
options are ordered and each says what is known versus assumed: the layer cache
(done, needs re-measuring after one warm run), versitygw's self-imposed 3s
lockWaitTime across 38 call sites, build-once-and-load, and -- if sharding is
ever revived -- bin-packing against measured durations rather than name order.
The shared stack is parked, and the arithmetic that made it look good is
recorded as wrong: shard 3 ran four tests in 204.654s, so a boot is at most
~51s, not the ~80s taken from 1257/13.

forgectl's mainnet metrics. It is the only imported service that had real
scheduled workflows -- payments every 30 minutes, faults every 12 hours, against
environment: mainnet, pushing to an OTLP endpoint. They still run in the
polyrepo, so nothing is broken now. Archiving fil-forge/forgectl stops them
silently, and reproducing them here needs an environment and secrets a person
has to configure. Sequenced with the archival rather than after it.

Phase 1. The plan says to cut tags "via the existing release.yml flow"; there is
no release.yml, and the four workflows here neither tag nor publish. What
survived is uneven -- version.json in 8 of 10, .goreleaser.yaml in 4,
Dockerfile.release in 4, and those four are goreleaser-shaped (COPY <svc>
/usr/bin/<svc> from a dist/ context), so no CI job can build them before the
release flow exists. hilt and swarf have one with no goreleaser config at all.

And the sequencing the plan does not state: a submodule tag must be <svc>/vX.Y.Z
in the repository the module path names, so a tag cut in forge-2 sits where no
module path resolves and has to be cut again after the rename. The rename gates
the tagging half; the tag scheme, the workflow itself and compat.yml do not wait
on it.

Also notes the trap that at the monorepo root `hilt` is a directory, so COPY
hilt /usr/bin/hilt copies a source tree if that file is ever built with the
repository root as context -- which is the context every other Dockerfile uses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt
Peeja pushed a commit that referenced this pull request Sep 17, 2026
…deadline

#23 re-homes the CI wall-clock measurement that would otherwise live only on
closed #21, plus two findings from reading main: Phase 1 has no release.yml to
use, and tags are gated on the rename because a submodule tag must sit in the
repository its module path names. #24 removes the last two per-service .github
directories. #25 layer-caches the image builds in itest and e2e while images.yml
stays cold as the canary -- the only one of the five open PRs that can fail.

The hazard is on Needs Human Work because it needs a person: forgectl's
metrics-payments and metrics-faults run against environment: mainnet every 30
minutes and 12 hours, pushing to an OTLP endpoint from repository secrets. They
run in the polyrepo today. Archiving fil-forge/forgectl stops them silently, and
reproducing them here needs an environment and secrets configured on this
repository.

Also records what was deliberately not attempted pre-rename and why: the release
workflow (unverifiable until tags can be cut, and an unrunnable workflow is the
silent-green shape this repo keeps deleting), compat.yml (needs published images
to test against), the tag scheme (long-lived, hard to reverse, a recommendation
rather than an agent's decision), and building the Dockerfile.release files
(goreleaser-shaped, they need the release flow first).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt
@Peeja
Peeja merged commit 290b5f5 into main Sep 17, 2026
22 checks passed
@Peeja
Peeja deleted the claude/todo-ci-speed branch September 17, 2026 16:46
Peeja pushed a commit that referenced this pull request Sep 17, 2026
TestUploadAndRetrieve/filesystem has failed 2 of 40 e2e runs, and both are the
same shape: compose up reports a dependency unhealthy, a different container
each time (upload-1 on run 160, plc-1 on run 182), always the filesystem
permutation and never s3. plc-1's log says connect ECONNREFUSED to its postgres,
one restart, then the healthcheck deadline. That is a startup race, not a broken
image.

Run 160 is on main at ff2f794, two and a half hours before any of today's
branches existed, which is what rules #25 out -- it changes how images are built,
so the red looked exactly like a regression until the main failure was found.
The two permutations are t.Parallel(), so ~40 containers start on one runner at
once; both observed failures being filesystem is suggestive of something
systematic, but two data points is not proof.

One re-run queued on #25, the single re-run allowed for establishing that a
failure is not the PR's. The honest fix is compose healthcheck ordering, not a
retry.

Also: main is 144b316 -- Petra merged #22, #23 and #24 while this was running.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt
Peeja pushed a commit that referenced this pull request Sep 17, 2026
Petra merged #20, #17, #22, #19, #23 and #24; main is 144b316 and the open
list is #25 and #26. Dropped the merged rows from both tables rather than
letting them accumulate.

#25's e2e red turned out to be the pre-existing filesystem flake, and the one
re-run that cleared it was warm -- so it produced the measurement as well as the
green: 8 images in 23s against a 7m34s baseline, and the e2e job from 13m10s to
5m29s. #26 puts that in MONOREPO_TODO.md with the caveats attached rather than
the headline alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants