Make pg_isready probe TCP, which is what dependents connect to

[Peeja]

⏭️ Safe to merge without waiting for CI — no. Watch e2e, and ci's guards.

Refreshed on every push. Checks are not required yet, so this is advice, not a gate.

check	verdict
e2e	Load-bearing. This changes when containers in the stack are considered healthy. It is the suite the bug was found in.
ci → guards	Load-bearing. Runs the new check-pg-healthchecks.sh for the first time in CI.
ci → 13 unit jobs	unit smelt is load-bearing (smelt/pkg/generate is changed). The other 12 are skippable — go list -deps reaches no other module, and no other module's packages are touched.
itest	Skippable. itest hilt/ingot boot their own stacks, but neither hilt's nor ingot's compose postgres is on their critical path in a way this changes — it only makes a healthcheck stricter, so the worst case is a slower boot, not a failure. The weakest claim in this block; if itest goes red, believe it over me.
images ×8	Skippable. No Dockerfile changed.

What this costs if it's wrong: the checks still run; we just don't wait. If an ignored check goes red after merge, main is red.

---

TestUploadAndRetrieve/filesystem has failed 2 of the last 40 e2e runs, naming a different container each time — upload-1 on run 160 (main ff2f794d), plc-1 on run 182. I called that a generic startup race this morning. It isn't. It's one bug with a precise mechanism, and plc-postgres's own log against plc's crash shows it:

16:55:27.593  temp server: listening on Unix socket ONLY
16:55:27.633  temp server: ready to accept connections   ← healthcheck goes GREEN
16:55:29.441  temp server: shut down
16:55:29.887  real server: listening on IPv4 0.0.0.0:5432 ← TCP finally exists
16:55:30.531  plc exits: ECONNREFUSED 172.18.0.6:5432

The healthcheck was green 2.25 seconds before the port existed.

Why

test: ["CMD-SHELL", "pg_isready -U plc -d plc"]      # no -h → Unix socket

Without -h, pg_isready probes the Unix socket. The official postgres image runs initdb against a temporary server started with listen_addresses='' — socket up, TCP refused by design — so the healthcheck passes during initialisation. A dependent with condition: service_healthy then starts into a window where the port it connects to doesn't exist.

plc already had depends_on: plc-postgres: {condition: service_healthy}. The ordering was never wrong; the readiness signal was. That's also why the victim differs run to run — whichever dependent happens to boot inside the window.

The fix, and its scope

-h 127.0.0.1 on all six sites — five compose files plus smelt/pkg/generate/compose.go, whose output generated/compose/piri.yml is gitignored and regenerates. I verified the generator actually emits it, not merely that the source changed:

generated/compose/piri.yml:105:  - pg_isready -U piri -d postgres -h 127.0.0.1

Postgres was the only member of this class, checked rather than assumed — every other healthcheck in the tree is HTTP over localhost/127.0.0.1 or redis-cli ping, all TCP by construction, so none can have the socket-versus-port gap.

The guard is the part that lasts

check-pg-healthchecks.sh, wired into the existing guards job. No list in it: every pg_isready either names a host or is a bug, so nothing needs updating when a service is added. It covers *.go as well as *.yml because the generator builds one of these strings.

It also fails when it finds nothing at all — a check that silently guards an empty set is the exact failure mode it exists to prevent.

Verified in both directions, not just the green one: 6 FAILs against the tree before the fix, 6 oks after.

⚠️ What I have not verified

That e2e now passes reliably. A 5% flake cannot be shown fixed by one green run, and there's no Docker here to exercise it. The mechanism is proven from the logs; the frequency is not. If it recurs, the log to check is whether the failing dependent's postgres shows the temp-server/real-server split again — if it does, this fix didn't take and the next suspect is start_period rather than the probe.

smelt/pkg/generate tests pass. The three pkg/stack failures in my sandbox are no such file or directory: /var/run/docker.sock and reproduce identically on unmodified main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt

---

Generated by Claude Code

[Peeja]

[From Claude:] guards went red, and it was the guard failing on its own step name. Fixed in a8a6040b.

FAIL ./.github/workflows/ci.yml:59 pg_isready with no -h probes the Unix socket

ci.yml:59 is - name: every pg_isready healthcheck probes TCP. The grep matched the bare word, so the check reported a workflow file as an unhosted postgres healthcheck. A guard that cannot tell running a thing from naming it is not guarding the thing.

The fix

Anchored to a preceding quote, so the name has to start a quoted command string. Deliberately not done by excluding .github/: an Actions services: block can legitimately carry --health-cmd "pg_isready …", so excluding the directory would have bought a real blind spot in exchange for dodging a false positive.

The gap that remains is named in the script's header rather than papered over: an unquoted invocation, e.g. --health-cmd pg_isready. Nothing in this repository writes one, and the alternative — matching every mention — is exactly what just misfired.

How I missed it, which is the part worth keeping

I verified the guard in both directions, then wired it into ci.yml, and did not run it again. So what I verified was a tree that no longer existed by the time I pushed — and the thing that broke it was the very line that wires it in.

Rule 5 already says a guard proves nothing until you have seen it fail on the defect it is for. It now also has to say: verify it against the tree you are actually pushing. That is on the wiki.

Re-verified on three cases this time, against the final tree:

-h dropped from plc	FAIL … compose.yml:51, exit 1
a prose mention added to ci.yml	exit 0 — no longer trips
the tree as pushed	exit 0, 6 oks

Unchanged

The postgres fix itself is untouched — six sites, -h 127.0.0.1, generator output checked. And still unverified: that the flake is gone. One green e2e cannot show a ~5% failure rate fixed. The mechanism is proven from plc-postgres's log; the frequency is not.

---

Generated by Claude Code