feat(events): cross-machine event bus via Postgres NOTIFY/LISTEN

[AquiGorka]

Why

Provider-platform runs multi-machine on Fly. The previous in-process EventBus only delivered events to subscribers on the same machine as the emitter. Fly proxy load-balances HTTP requests across machines, so a bundle POST lands on machine A while the WS subscriber for that PP is held on machine B — A's local emit goes to A's bus only, B's WS subscriber never sees it.

Diagnosed in the add-events-capture-framework-1 thread (2026-06-09) with verbatim Fly logs from PRs #115 + #116:

For a withdraw bundle, events 1–8 (deposit + send, processed on the same machine as the WS) all delivered with event sent to WS log lines. Events 9–12 (withdraw, processed on the other machine) had EventBus.emit log lines on the other machine but ZERO matching event sent or event dropped lines anywhere — the listener literally doesn't exist on the emitter's process.

Workaround currently in place: testnet's second machine has autostart disabled (flyctl machine update --autostart=false), forcing single-machine. Mainnet is also single-machine today. This PR makes that workaround unnecessary so capacity scaling can land later.

Design

Replace the in-process bus with a Postgres NOTIFY/LISTEN-backed implementation:

• Single channel provider_events for the entire ProviderEvent taxonomy. Receivers deserialize and dispatch by kind. No per-event-type channels, no event persistence table.
• Single-path delivery: emit() ONLY fires NOTIFY provider_events <payload> via the existing drizzle Postgres pool. Even local emitters round-trip through Postgres — adds ~5–20 ms latency but means there is one delivery path to debug. No dedup needed, no "did this fire twice" question.
• Dedicated listener task per machine: startPgListener(deps) boots a separate postgres() client (NOT the drizzle pool — LISTEN reserves its connection for the subscription's lifetime), calls sql.listen('provider_events', onNotify, onListen), and on every notification parses the JSON and calls into the bus's internal publishLocal() — driving local subscribers without re-triggering NOTIFY.
• Payload-size guard: defensive payload.length < 7900 bytes check at emit time (Postgres NOTIFY cap is 8000). Oversize events are logged + dropped rather than thrown back into the caller's already-committed transaction. Worst-case audited payload (verifier.bundle_completed with 100 sha256-hex bundle IDs) is ~7,060 bytes — under the guard with ~12% headroom. Defensive guard captures whatever future bundle-ID format pay-platform may send.
• Reconnect: postgres@^3.4.7's sql.listen ships its own reserved-connection reconnect logic. The onlisten callback fires on every (re-)subscribe so we log each recovery. Start-from-now semantics during disconnect windows — subscribers backfill themselves if needed (out of scope for the transport).
• Boot order: pgListener subscribes first, setNotifier() wires the NOTIFY transport on the bus second, then mempool / event-watcher / HTTP server start. Until setNotifier() is called, the bus falls back to in-process loopback — used in tests and during boot before transport is up.
• No call-site changes: the bus's public emit + subscribe surface is preserved byte-for-byte. emit-helpers.ts, ws-handler.ts, and the integration test only update their import paths.

Files

Added:

• src/core/service/events/pg-notify-event-bus.ts — new bus impl with emit + subscribe + internal publishLocal + setNotifier.
• src/core/service/events/pg-listener.ts — long-lived LISTEN task with shutdown hook.
• src/core/service/events/pg-notify-event-bus.test.ts — 9 unit tests covering subscribe/publishLocal, loopback fallback, single-path-delivery invariant, oversize-drop, NOTIFY-rejection swallow, and the simulated cross-machine round-trip.
• tests/integration/events/cross-machine-events.test.ts — 3 integration tests using PGlite's native LISTEN/NOTIFY API to exercise the same wire path (serialize → NOTIFY → LISTEN → parse → publishLocal → subscriber) across separate connections to a shared backend.

Removed:

• src/core/service/events/event-bus.ts — direct replacement, no feature flag, no parallel old + new.

Modified:

• src/core/service/events/index.ts — re-export PgNotifyEventBus, startPgListener, PROVIDER_EVENTS_CHANNEL.
• src/core/service/events/emit-helpers.ts, src/http/v1/events/ws-handler.ts, tests/integration/http/events-ws.test.ts — import path only.
• src/main.ts — boot order: startPgListener → setNotifier(pgClient.notify) → mempool/event-watcher/HTTP. Shutdown handler awaits listener teardown.
• src/persistence/drizzle/config.ts — export the raw postgres-js client so the bus's NOTIFY uses the existing pool.

Local multi-machine smoke

Ran a narrowed smoke against an isolated Postgres (port 5443, not touching any other local stack) with two provider-platform processes from this branch HEAD (ports 3110 + 3111, same DATABASE_URL).

Boot order on each instance:

EVT -starting pgListener
EVT -LISTEN provider_events subscribed
EVT -pgListener started
EVT -NOTIFY transport wired
EVT -server running on http://localhost:31XX

Cross-machine delivery via psql pg_notify:

• Fired from a third connection: SELECT pg_notify('provider_events', '<mempool.bundle_added JSON>');
• Instance A receipt: 13:14:59.600 DBG kind: mempool.bundle_added (main.PgNotifyEventBus)
• Instance B receipt: 13:14:59.600 DBG kind: mempool.bundle_added (main.PgNotifyEventBus)
• Same timestamp, 1 ms processing on each instance.

Production transport via postgres@3.4.7 sql.notify:

• Standalone script: await sql.notify('provider_events', JSON.stringify(event)) against the same DB.
• Instance A: 13:16:08.085 DBG kind: executor.transaction_submitted
• Instance B: 13:16:08.084 DBG kind: executor.transaction_submitted
• 1 ms cross-instance delivery via the exact wire format production uses.

Clean shutdown via SIGTERM:

EVT -shutting down server
EVT -stopping pgListener
EVT -pgListener stopped       (3 ms after stop)

Two-machine local-dev/testnet/run-local.sh all was NOT run against the full local-dev stack — PM-acked, deferred to post-merge verification.

Out of scope

• network-dashboard-platform — uses the same pattern but its bus.ts docstring explicitly calls itself "single-instance per env by design". Separate future thread.
• council-platform, pay-platform — audited, no EventBus / WS handler.
• Re-enabling provider-platform's second-machine autostart on testnet/mainnet — follow-up PM action once this lands.

Test plan

• deno fmt --check
• deno lint
• deno check src/main.ts
• deno task test:unit — 74 passed (9 new on PgNotifyEventBus)
• deno task test:integration — 99 passed (3 new on cross-machine delivery)
• deno task test (full) — 116 passed
• Local two-instance smoke: psql NOTIFY + postgres-js sql.notify both deliver to both instances within 1 ms; SIGTERM shuts pgListener cleanly.
• Two-machine local-dev/testnet/run-local.sh all against the full local-dev stack — PM-deferred to post-merge.