refactor(sync): event-driven topology — sync-at-boot + listeners (mirroring provider-platform watcher pattern)#5
Merged
Conversation
The hourly `refreshTopology` re-sync (HOURLY_RESYNC_MS = 60 min) was the only place a freshly-deployed council could land in the watcher's contractId filter — the auto-discovery fallback was gated on a GitHub-sourced WASM hash registry that returns HTTP 403 in any environment without authenticated access to Moonlight-Protocol/soroban-core. Move to provider-platform's pattern: sync at boot, set listeners, update state as events come in. - scheduler.ts: drop the hourly timer; keep the 60s rolling-window sweep (memory cap on the 24h counter window). - main.ts: drop the boot-time `refreshWasmRegistry` call + the "step 1 WASM registry" comment block. Re-number the bootstrap order. - wasm-registry.ts: deleted. `isKnownChannelAuthHash` / `isWasmRegistryReady` had no other callers after the contract-init listener was ungated.
The contract-init listener no longer requires a populated WASM hash registry. The watcher always polls for `contract_initialized` events from unknown contracts, registers each with its observed-at ledger, and debounces topology refreshes through `drainPendingAdoptions` (single-flight, runs once per poll tick across the whole batch). Two follow-on changes were needed for the events surface to stay clean: 1. Retry-with-TTL on pending adoptions. `contract_initialized` fires several seconds before council-platform learns about the new council, so a single-tick adopt-or-cache-as-notMoonlight check would permanently reject every legitimate new council. We now keep contracts pending for PENDING_TTL_MS (120s) and re-attempt on every subsequent drain pass before giving up. 2. Back-fill on adoption. Forward polling resumes from `lastLedgerSeen + 1`, which means any chain event between the deploy ledger and the adoption tick (typically `provider_added` and the first SAC transfer) is missed. The new `backfillFromLedger` scan in soroban-watcher walks every watched contract from the earliest adopted observed-at ledger to head, maps + publishes each event via the bus. Dedup via `networkState.recordEvent` keeps the forward poll's later observation of the same events from double-publishing. `topology-refresh.ts` no longer co-fetches the WASM registry — the single-flight refresh is just `fetchCouncilTopology` + `replaceTopology` + `rescanRollingWindow` now.
…dleTimeout Two race conditions surfaced when the events-capture harness was wired through testnet/run-local.sh against the live local stack — both root causes silently lost `council_formed` and `provider_added` events between `bus.publish` and the WS subscriber: 1. `refreshTopology` → `rescanRollingWindow` → `coldStartScan` → `seedRecent` REPLACED `networkState.recent` with cold-scanned events. The contract-init-listener's subsequent `backfillFromLedger` then called `publishMappedEvent` whose `recordEvent` dedup checks `recent.some(e => e.id === event.id)` — finding the cold-scanned entry, returning `wasNew=false`, and skipping `bus.publish`. The back-fill silently no-op'd for every event that the cold-start scan had just seeded. Fix: `refreshTopology` now only does `fetchCouncilTopology + replaceTopology`. The contract-init-listener caller is already followed by `backfillFromLedger` (the explicit catch-up path), which fans out the historical-but-newly-relevant events live via the bus. 2. `IDLE_TIMEOUT_SECONDS = 30` in network-ws.ts was too aggressive for programmatic subscribers (the events-capture harness) on flows where chain events don't fire for >30 s after `onopen`. The SPA's reconnect-on-close masked this in production; the harness has its own reconnect now, but the wider idle window stays in place so we don't lean on either side alone. Eight back-to-back smoke runs through ./testnet/run-local.sh (testnet-main × 5 + lifecycle × 3) on the corrected stack all assert 12/12 perPp + 5/5 network green, no flake.
AquiGorka
added a commit
to Moonlight-Protocol/local-dev
that referenced
this pull request
Jun 5, 2026
…-main + lifecycle-testnet-verify (#112) ## Summary Adds a small framework that asserts the WebSocket event surfaces of `provider-platform` (`/api/v1/providers/:pp/events/ws`) and `network-dashboard-platform` (`/api/v1/network/ws`) against a strict-order, strict-value declaration inlined at the top of each testnet flow script. Two scripts produce events in the canonical `testnet/run-all.sh` runner — `testnet/main.ts` (suite 1, payment flow) and `lifecycle/testnet-verify.ts` (suite 3, UC2 governance). Each script now declares an `EXPECTED_EVENTS` const at the top. The harness subscribes to both WS surfaces before the script's chain interactions, runs the script's exported `main()` in-process, tails for `EVENTS_CAPTURE_TAIL_MS` (default 5000), then diffs captured against expected and writes a structured run report. Suite 2 and suite 4 (OTEL trace verifiers) stay as-is — they do no chain ops, so there's nothing to capture. ## Cross-PR dependency — read this first **This PR depends on `Moonlight-Protocol/network-dashboard-platform#5`.** That PR refactors network-dashboard-platform from hourly periodic resync to event-driven topology + back-fill on adoption, which is what makes `council_formed` (the first network event in both scripts' `EXPECTED_EVENTS`) fire deterministically. The local-stack smoke in this PR was validated against that refactor; without it, the network-side assertion would still fail on the first event. **Merge order: n-d-p#5 first, then this.** ## What's in this PR - `testnet/events-capture/types.ts` — shared types + the `ALWAYS_SKIP_FIELDS` set (deep-stripped from both expected and captured before equality check). Covers `ts`, `id`, `occurredAt`, `ledger`, `bundleId/bundleIds`, `txHash/txId`, contract-id family, `councilName` (timestamp-embedded literal in both scripts), `reason` (free-form error.message at the producer site). - `testnet/events-capture/subscribe.ts` — per-PP + network-wide WS clients. Per-PP polls `GET /api/v1/dashboard/pp/list` until the operator's PP appears in the DB, then upgrades with `Sec-WebSocket-Protocol: bearer.<jwt>`. Network is public, snapshot frame is dropped (PM-acked: state-at-connect-time depends on prior runs), every subsequent `LiveFrame` is captured. Uses Deno's built-in `WebSocket` — no new dep. - `testnet/events-capture/assert.ts` — strict-order, strict-value diff with the always-skip set deep-stripped. Returns a structured per-index per-subscriber result; one subscriber failing doesn't mark another as failing. - `testnet/events-capture/report.ts` — stdout summary (PASS line per subscriber, or full mismatch diff with expected/captured per index) + JSON dump under `runs/<run-id>/report.json`. `runs/` is gitignored. - `testnet/events-capture/harness.ts` — CLI orchestrator. Derives the operator/alice/bob keypairs from `MASTER_SECRET` (matching the scripts' own `deriveKeypair(ROLES.PP / ROLES.ALICE / ROLES.BOB, 0)`), acquires the dashboard JWT, opens both subscribers, dynamic-imports the script module, runs `main()`, tails, closes, asserts, writes report, exits 0/1/2. - `testnet/main.ts` and `lifecycle/testnet-verify.ts` — inlined `EXPECTED_EVENTS`. Each script's `main()` is now exported and the bottom-of-file invocation is guarded with `if (import.meta.main)` so direct `deno run` still works. - `testnet/run-all.sh` and `testnet/run-local.sh` — suites 1 + 3 now invoke the harness instead of `run-tempo.sh` / direct `deno run`. `run-local.sh` exports defaults for `NETWORK_DASHBOARD_PLATFORM_URL` and `MASTER_SECRET` when not set by the caller. ## EXPECTED_EVENTS values were derived from Phase 0 audit + Phase 3 smoke - Per-PP: 12 events per script. `weight` values come from the bundle-operation counts in `lib/client/{deposit,send,withdraw}.ts` × the mempool defaults (`MEMPOOL_EXPENSIVE_OP_WEIGHT=10` × `MEMPOOL_CHEAP_OP_WEIGHT=1`). `amount` strings come from the deposit / send / withdraw amount + fee math; `entityName` / `jurisdictions` / `depositorAddress` / `recipientAddress` from the script's literals + the `MASTER_SECRET`-derived keypairs. - Network: 5 events per script. The `processRawEventBatch` dedup in network-dashboard-platform drops `channel_bundle` whenever the tx also produced `channel_deposit` or `channel_settlement`, so the visible sequence is `council_formed → provider_added → channel_deposit → channel_bundle (send) → channel_settlement`. Validated empirically in the Phase 3 smoke (after the n-d-p refactor lands). ## Test plan - [x] `deno fmt --check` over the changed files (`testnet/events-capture/`, `testnet/main.ts`, `lifecycle/testnet-verify.ts`, `testnet/run-all.sh`, `testnet/run-local.sh`) — clean. - [x] `deno lint` over the same set — clean. (Repo-wide `deno lint` surfaces ~23 pre-existing problems in unrelated files such as `setup-pp.ts:35` and `playwright/tests/full-flow.spec.ts:44`. Same out-of-PR-scope situation as the prior PR's pre-existing test debt.) - [x] Local-stack smoke against the harness invocation for both scripts: ``` testnet-main — perPp: 12/12 network: 5/5 PASS lifecycle-testnet-verify — perPp: 12/12 network: 5/5 PASS ``` Run artifacts: `testnet/events-capture/runs/testnet-main-2026-06-05T16-31-48-970Z/report.json` and `testnet/events-capture/runs/lifecycle-testnet-verify-2026-06-05T16-33-57-529Z/report.json`. No testnet RPC / contract / signer was touched — outbound calls only to `localhost:8000` (Stellar quickstart), `localhost:4015` (council), `localhost:4010` (provider), `localhost:4035` (network-dashboard). - [ ] Reviewer pass on the `EXPECTED_EVENTS` field selections (which keys are strict-asserted vs deep-stripped — see `ALWAYS_SKIP_FIELDS` in `types.ts`). - [ ] Reviewer pass on the harness's per-PP open loop (`GET /api/v1/dashboard/pp/list` because no single-PP GET endpoint exists today).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactors the topology / event-discovery surface to mirror
provider-platform'sevent-watcherpattern: sync at boot, set listeners, update state as events come in. The previous hourly periodic re-sync is gone; the new-council fallback that depended on a GitHub-sourced WASM hash registry is gone; what's left is a tight chain-event-driven loop that publishes within ~5 s of a new council going on-chain.Validated by a local-stack end-to-end smoke (
/Users/theahaco/repos/tmp/add-events-capture-framework-1/) where two flow scripts deploy a fresh council mid-test, and the network-dashboard WS surfaces the resultingcouncil_formed → provider_added → channel_deposit → channel_bundle → channel_settlementsequence in real time.The four architectural changes
Remove the hourly topology resync (
src/core/sync/scheduler.ts). The scheduler now does only the 60-second rolling-window sweep (memory cap on the 24h counter window). Topology arrives at boot viafetchCouncilTopologyand updates via the contract-init listener.Drop the WASM hash registry (
src/core/sync/wasm-registry.tsdeleted;src/main.ts,src/core/sync/topology-refresh.tsupdated). The registry was sourced fromMoonlight-Protocol/soroban-core's GitHub releases and returnedHTTP 403for any unauthenticated fetch — which meant the new-council fallback was effectively a permanent no-op in local-dev.isKnownChannelAuthHash/isWasmRegistryReady/knownHashCounthad no callers after change feat: real-time discovery + expanded aggregations + live counters #3.Ungate the contract-init listener (
src/core/sync/contract-init-listener.ts,src/core/sync/soroban-watcher.ts). The watcher always pollsgetEventsforcontract_initializedfrom unknown contracts (no moreisContractInitListenerEnabled()gate). Each unknown is held inpendingAdoptionwith anobservedAtLedgerfor back-fill (see feat: migrate to new backend logging convention #4) and a TTL ofPENDING_TTL_MS = 120 s— long enough that a council registered AFTER its on-chain deploy still gets adopted on a subsequent tick. Past the TTL the contract is cached asnotMoonlightso a chatty unrelated contract can't drag us into infinite topology refreshes.Back-fill on adoption (
src/core/sync/soroban-watcher.ts:backfillFromLedger, threaded viadrainPendingAdoptions's newbackfillFromLedgerdep). Forward polling resumes fromlastLedgerSeen + 1, so any chain event between the deploy ledger and the adoption tick (most notablyprovider_addedand the first SACtransfer) would otherwise be lost. The back-fill scan walks every currently-watched contract from the earliest adoptedobservedAtLedgerto head, maps each event, and publishes via the bus.networkState.recordEvent's dedup makes this safe to run concurrent withpollTick.The hot-path behaviour now matches
provider-platform/src/core/service/event-watcher/index.ts:initFromDb→ensureWatcherper ACTIVE membership.emitForPp/emitForBundles.addCouncilWatcher/addProviderAddressexported for API handlers to call when new entities are created.For network-dashboard the analog is now:
fetchCouncilTopologyat boot, then chain-event-driven adoption viacontract_initialized→refreshTopology→backfillFromLedger.Cross-PR dependency
This PR must merge before Moonlight-Protocol/local-dev#112. That PR adds a strict-order, strict-value assertion harness whose
EXPECTED_EVENTS.networkbegins withcouncil_formed— which only fires deterministically after this refactor'sbackfillFromLedgeris in place. Without this PR, the local-dev assertion would still fail on the first event index.Test plan
deno fmt --check— clean (32 files)deno lint— clean (21 files)deno task test— 17 / 0council_formedlanding first via the back-fill path.notMoonlightTTL choice (PENDING_TTL_MS = 120 s) — that's the upper bound between a council's on-chain deploy and council-platform'sPUT /council/metadatain the testnet scripts.backfillFromLedgerscans every watched contract fromminAdoptedLedger, not only the newly-adopted ones, because cross-council SAC events (XLMtransfer/fee) are mapped via the linkage table thatrefreshTopologyjust installed.