feat(testnet): events-capture framework + EXPECTED_EVENTS for testnet-main + lifecycle-testnet-verify#112
Merged
Merged
Conversation
… orchestrator A small framework for asserting the WS event surfaces of provider-platform (`/api/v1/providers/:pp/events/ws`) and network-dashboard-platform (`/api/v1/network/ws`) against a strict-order, strict-value declaration inlined at the top of each testnet flow script. - subscribe.ts: per-PP + network-wide WebSocket clients using Deno's built-in `WebSocket`. Per-PP polls `GET /api/v1/dashboard/pp/list` until the operator's PP appears in the DB (no single-PP endpoint exists), then upgrades with `Sec-WebSocket-Protocol: bearer.<jwt>`. Network is public. - assert.ts: deep-strips the ALWAYS_SKIP set (`ts`, `id`, `occurredAt`, `ledger`, contract-id family, `councilName`, `reason`, etc.) from both expected and captured, then walks each subscriber's events in lockstep and emits a structured per-index diff. - report.ts: stdout summary + JSON dump under `events-capture/runs/<run-id>/report.json` (run dir is gitignored). - harness.ts: CLI entrypoint. Derives the operator/alice/bob keypairs from `MASTER_SECRET` (matching the scripts' own derivation), acquires the dashboard JWT, opens both subscribers, runs the script's exported `main()`, tails for `EVENTS_CAPTURE_TAIL_MS` (default 5_000), closes the sockets, asserts, writes the report, exits 0 / 1 / 2.
…verify.ts Each flow script now declares its expected per-PP and network-wide event streams at the top of the file. The harness reads `EXPECTED_EVENTS` verbatim and asserts captured == expected after stripping the nondeterministic skip set. Values for the strict-equality fields (`weight`, `entityName`, `jurisdictions`, `amount`, `ppLabel`, `depositorAddress`, `recipientAddress`, `newSlot`, `kind`, `scope.ppPublicKey`) come from a Phase 3 local-stack smoke that exercised both scripts end-to-end against a fresh `MASTER_SECRET`-derived deploy. Per-PP count for both scripts is 12 events: channel.provider_added → deposit cycle (mempool / executor / verifier / bundle.deposit_completed) → send cycle (3 events) → withdraw cycle (4 events incl. bundle.withdraw_completed). Network-wide count for both scripts is 5 events (after the network-dashboard's `processRawEventBatch` dedup which drops `channel_bundle` whenever the tx also emitted a money-flow event): council_formed → provider_added → channel_deposit → channel_bundle (send fee) → channel_settlement. `council_formed` was provisionally OMITted in the Phase 0 catalogue (item B) because the cold-start lookback race was unclear. The Phase 3 smoke showed it landing deterministically once network-dashboard-platform's contract-init listener adopts a new council and back-fills from the deploy ledger, so it's reinstated. Each script's existing `main()` is now `export async function main()`, and its bottom-of-file invocation is guarded with `if (import.meta.main)` so direct `deno run ...` still works AND the harness can dynamic-import the module to read `EXPECTED_EVENTS` and call `main()` in-process.
…r suites 1+3 `testnet/run-all.sh` and `testnet/run-local.sh` now invoke `events-capture/harness.ts` instead of `run-tempo.sh` (suite 1) and the direct `lifecycle/testnet-verify.ts` `deno run` (suite 3). Suites 2 and 4 (the OTEL trace verifiers) stay as-is — no chain ops, nothing to assert. `run-local.sh` additionally exports defaults for `NETWORK_DASHBOARD_PLATFORM_URL` (the harness's network-wide WS target) and `MASTER_SECRET` (so the harness's keypair derivation matches the flow scripts') when not set by the caller.
6 tasks
…fault tail Two robustness fixes for the network subscriber, surfaced by running the harness through testnet/run-local.sh against the live local stack: 1. Auto-reconnect on close (network + per-PP). The previous one-shot `new WebSocket(...)` would silently lose every subsequent frame when the connection dropped — e.g. on the server-side idleTimeout that fires when chain events don't arrive for tens of seconds after `onopen`. The network subscriber also parses the new `SnapshotFrame.recent` on every reconnect (newest-first per the server's `seedRecent` contract) and replays any `NetworkEvent.id` it hasn't observed yet, recovering events that landed during the disconnect window. Backoff is 250 ms → 5 s, capped. 2. `DEFAULT_TAIL_MS` bumped from 5 000 to 30 000. The original 5 s default was tight enough for the per-PP-only smoke runs we tested with directly, but the network subscriber's first event (`council_formed`) lands 25-40 s after the back-fill scan picks up a freshly-deployed council — comfortably past 5 s. 30 s also matches the value used in the Phase 3 smoke runs that informed the EXPECTED_EVENTS catalogues. Override via `EVENTS_CAPTURE_TAIL_MS` per run. With these + the n-d-p `fix(sync): drop rescanRollingWindow from refreshTopology + raise WS idleTimeout` change, eight back-to-back smoke runs through ./testnet/run-local.sh (testnet-main × 5 + lifecycle × 3) assert 12/12 perPp + 5/5 network green with no flake.
AquiGorka
added a commit
to Moonlight-Protocol/network-dashboard-platform
that referenced
this pull request
Jun 5, 2026
…roring provider-platform watcher pattern) (#5) ## Summary Refactors the topology / event-discovery surface to mirror `provider-platform`'s `event-watcher` pattern: **sync at boot, set listeners, update state as events come in**. The previous hourly periodic re-sync is gone; the new-council fallback that depended on a GitHub-sourced WASM hash registry is gone; what's left is a tight chain-event-driven loop that publishes within ~5 s of a new council going on-chain. Validated by a local-stack end-to-end smoke (`/Users/theahaco/repos/tmp/add-events-capture-framework-1/`) where two flow scripts deploy a fresh council mid-test, and the network-dashboard WS surfaces the resulting `council_formed → provider_added → channel_deposit → channel_bundle → channel_settlement` sequence in real time. ## The four architectural changes 1. **Remove the hourly topology resync** (`src/core/sync/scheduler.ts`). The scheduler now does only the 60-second rolling-window sweep (memory cap on the 24h counter window). Topology arrives at boot via `fetchCouncilTopology` and updates via the contract-init listener. 2. **Drop the WASM hash registry** (`src/core/sync/wasm-registry.ts` deleted; `src/main.ts`, `src/core/sync/topology-refresh.ts` updated). The registry was sourced from `Moonlight-Protocol/soroban-core`'s GitHub releases and returned `HTTP 403` for any unauthenticated fetch — which meant the new-council fallback was effectively a permanent no-op in local-dev. `isKnownChannelAuthHash` / `isWasmRegistryReady` / `knownHashCount` had no callers after change #3. 3. **Ungate the contract-init listener** (`src/core/sync/contract-init-listener.ts`, `src/core/sync/soroban-watcher.ts`). The watcher always polls `getEvents` for `contract_initialized` from unknown contracts (no more `isContractInitListenerEnabled()` gate). Each unknown is held in `pendingAdoption` with an `observedAtLedger` for back-fill (see #4) and a TTL of `PENDING_TTL_MS = 120 s` — long enough that a council registered AFTER its on-chain deploy still gets adopted on a subsequent tick. Past the TTL the contract is cached as `notMoonlight` so a chatty unrelated contract can't drag us into infinite topology refreshes. 4. **Back-fill on adoption** (`src/core/sync/soroban-watcher.ts:backfillFromLedger`, threaded via `drainPendingAdoptions`'s new `backfillFromLedger` dep). Forward polling resumes from `lastLedgerSeen + 1`, so any chain event between the deploy ledger and the adoption tick (most notably `provider_added` and the first SAC `transfer`) would otherwise be lost. The back-fill scan walks every currently-watched contract from the earliest adopted `observedAtLedger` to head, maps each event, and publishes via the bus. `networkState.recordEvent`'s dedup makes this safe to run concurrent with `pollTick`. The hot-path behaviour now matches `provider-platform/src/core/service/event-watcher/index.ts`: - **Boot**: `initFromDb` → `ensureWatcher` per ACTIVE membership. - **Listeners**: per-contract Soroban poll, narrowly-scoped. - **State updates as events come in**: handler emits via `emitForPp` / `emitForBundles`. - **Hot-path additions**: `addCouncilWatcher` / `addProviderAddress` exported for API handlers to call when new entities are created. For network-dashboard the analog is now: `fetchCouncilTopology` at boot, then chain-event-driven adoption via `contract_initialized` → `refreshTopology` → `backfillFromLedger`. ## Cross-PR dependency This PR must merge **before** Moonlight-Protocol/local-dev#112. That PR adds a strict-order, strict-value assertion harness whose `EXPECTED_EVENTS.network` begins with `council_formed` — which only fires deterministically after this refactor's `backfillFromLedger` is in place. Without this PR, the local-dev assertion would still fail on the first event index. ## Test plan - [x] `deno fmt --check` — clean (32 files) - [x] `deno lint` — clean (21 files) - [x] `deno task test` — **17 / 0** - [x] Local-stack end-to-end against a fresh-deploy flow script (testnet/main.ts + lifecycle/testnet-verify.ts): 5/5 network events captured in correct order per script, both with `council_formed` landing first via the back-fill path. - [ ] Reviewer pass on the `notMoonlight` TTL choice (`PENDING_TTL_MS = 120 s`) — that's the upper bound between a council's on-chain deploy and council-platform's `PUT /council/metadata` in the testnet scripts. - [ ] Reviewer pass on the back-fill scope — `backfillFromLedger` scans every watched contract from `minAdoptedLedger`, not only the newly-adopted ones, because cross-council SAC events (XLM `transfer` / `fee`) are mapped via the linkage table that `refreshTopology` just installed.
5 tasks
AquiGorka
added a commit
that referenced
this pull request
Jun 8, 2026
…all/run-local (#113) ## Summary - The events-capture harness shipped in #112 required a wrapper-level `MASTER_SECRET` to derive operator/alice/bob keys ahead of subscriber bring-up. - When `run-local.sh all` runs suite 1 (`testnet-main`) and suite 3 (`lifecycle-testnet-verify`) back-to-back under the single root the wrapper exports, both suites derive the **same** PP operator key — suite 3's `POST /dashboard/council/join` then hits provider-platform's "PP already active for this council" check and 409s. `run-all.sh` has a related failure mode: it doesn't export `MASTER_SECRET` at all, so the harness throws at parse-args time. - Fix: layer one more SHA-256 stage on top of `masterSeedFromSecret` keyed on the `scriptName`, and fall back to a freshly-minted random root when `MASTER_SECRET` is unset. ## What changed in `testnet/events-capture/harness.ts` - `derivePerSuiteSecret(rootSecret, scriptName)` — `SHA-256(rootSeed || `suite/\${scriptName}`)` → 32 bytes → Stellar SK via `Keypair.fromRawEd25519Seed(...).secret()`. Matches the SHA-256-of-concatenation style already in `lib/master-seed.ts`. - Before invoking the script's `main()`, the harness re-exports `MASTER_SECRET` to the per-suite SK so the script's own `Deno.env.get("MASTER_SECRET") + masterSeedFromSecret(...)` derives matching role keys. - `parseArgs()` no longer throws when `MASTER_SECRET` is unset — it mints an ephemeral root, restoring the pre-harness shape of the underlying scripts and letting `run-all.sh` invoke the harness without wrapper changes. ## Properties - Caller's surface unchanged. Pin `MASTER_SECRET` once at the wrapper level → reproducible across re-runs; leave it unset → ephemeral per-invocation. - Same root + same `scriptName` ⇒ identical per-suite SK (reproducibility). - Different `scriptName` ⇒ disjoint role keys (suite 1 and suite 3 isolated). ## Test plan - [x] `deno fmt --check` clean (`testnet`) - [x] `deno lint` clean (`testnet`) - [x] `deno check events-capture/harness.ts` clean - [x] `./testnet/run-local.sh all` against local stack — full sweep PASS: - Suite 1 — payment flow: 12/12 per-PP events + 5/5 network events, flow in 76.4s - Suite 2 — OTEL verify (payment): 23/23 assertions - Suite 3 — lifecycle flow: 12/12 per-PP events + 5/5 network events, flow in 80.6s - Suite 4 — OTEL verify (lifecycle): 23/23 assertions - PP operator keys differ between suite 1 (`GAFWC4R3...`) and suite 3 (`GAX77SRV...`) — per-suite derivation confirmed working. - [ ] `./testnet/run-all.sh` against deployed testnet — to verify the deployed-Tempo path still works end-to-end with the no-`MASTER_SECRET` fallback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a small framework that asserts the WebSocket event surfaces of
provider-platform(/api/v1/providers/:pp/events/ws) andnetwork-dashboard-platform(/api/v1/network/ws) against a strict-order, strict-value declaration inlined at the top of each testnet flow script.Two scripts produce events in the canonical
testnet/run-all.shrunner —testnet/main.ts(suite 1, payment flow) andlifecycle/testnet-verify.ts(suite 3, UC2 governance). Each script now declares anEXPECTED_EVENTSconst at the top. The harness subscribes to both WS surfaces before the script's chain interactions, runs the script's exportedmain()in-process, tails forEVENTS_CAPTURE_TAIL_MS(default 5000), then diffs captured against expected and writes a structured run report.Suite 2 and suite 4 (OTEL trace verifiers) stay as-is — they do no chain ops, so there's nothing to capture.
Cross-PR dependency — read this first
This PR depends on
Moonlight-Protocol/network-dashboard-platform#5. That PR refactors network-dashboard-platform from hourly periodic resync to event-driven topology + back-fill on adoption, which is what makescouncil_formed(the first network event in both scripts'EXPECTED_EVENTS) fire deterministically. The local-stack smoke in this PR was validated against that refactor; without it, the network-side assertion would still fail on the first event.Merge order: n-d-p#5 first, then this.
What's in this PR
testnet/events-capture/types.ts— shared types + theALWAYS_SKIP_FIELDSset (deep-stripped from both expected and captured before equality check). Coversts,id,occurredAt,ledger,bundleId/bundleIds,txHash/txId, contract-id family,councilName(timestamp-embedded literal in both scripts),reason(free-form error.message at the producer site).testnet/events-capture/subscribe.ts— per-PP + network-wide WS clients. Per-PP pollsGET /api/v1/dashboard/pp/listuntil the operator's PP appears in the DB, then upgrades withSec-WebSocket-Protocol: bearer.<jwt>. Network is public, snapshot frame is dropped (PM-acked: state-at-connect-time depends on prior runs), every subsequentLiveFrameis captured. Uses Deno's built-inWebSocket— no new dep.testnet/events-capture/assert.ts— strict-order, strict-value diff with the always-skip set deep-stripped. Returns a structured per-index per-subscriber result; one subscriber failing doesn't mark another as failing.testnet/events-capture/report.ts— stdout summary (PASS line per subscriber, or full mismatch diff with expected/captured per index) + JSON dump underruns/<run-id>/report.json.runs/is gitignored.testnet/events-capture/harness.ts— CLI orchestrator. Derives the operator/alice/bob keypairs fromMASTER_SECRET(matching the scripts' ownderiveKeypair(ROLES.PP / ROLES.ALICE / ROLES.BOB, 0)), acquires the dashboard JWT, opens both subscribers, dynamic-imports the script module, runsmain(), tails, closes, asserts, writes report, exits 0/1/2.testnet/main.tsandlifecycle/testnet-verify.ts— inlinedEXPECTED_EVENTS. Each script'smain()is now exported and the bottom-of-file invocation is guarded withif (import.meta.main)so directdeno runstill works.testnet/run-all.shandtestnet/run-local.sh— suites 1 + 3 now invoke the harness instead ofrun-tempo.sh/ directdeno run.run-local.shexports defaults forNETWORK_DASHBOARD_PLATFORM_URLandMASTER_SECRETwhen not set by the caller.EXPECTED_EVENTS values were derived from Phase 0 audit + Phase 3 smoke
weightvalues come from the bundle-operation counts inlib/client/{deposit,send,withdraw}.ts× the mempool defaults (MEMPOOL_EXPENSIVE_OP_WEIGHT=10×MEMPOOL_CHEAP_OP_WEIGHT=1).amountstrings come from the deposit / send / withdraw amount + fee math;entityName/jurisdictions/depositorAddress/recipientAddressfrom the script's literals + theMASTER_SECRET-derived keypairs.processRawEventBatchdedup in network-dashboard-platform dropschannel_bundlewhenever the tx also producedchannel_depositorchannel_settlement, so the visible sequence iscouncil_formed → provider_added → channel_deposit → channel_bundle (send) → channel_settlement. Validated empirically in the Phase 3 smoke (after the n-d-p refactor lands).Test plan
deno fmt --checkover the changed files (testnet/events-capture/,testnet/main.ts,lifecycle/testnet-verify.ts,testnet/run-all.sh,testnet/run-local.sh) — clean.deno lintover the same set — clean. (Repo-widedeno lintsurfaces ~23 pre-existing problems in unrelated files such assetup-pp.ts:35andplaywright/tests/full-flow.spec.ts:44. Same out-of-PR-scope situation as the prior PR's pre-existing test debt.)testnet/events-capture/runs/testnet-main-2026-06-05T16-31-48-970Z/report.jsonandtestnet/events-capture/runs/lifecycle-testnet-verify-2026-06-05T16-33-57-529Z/report.json. No testnet RPC / contract / signer was touched — outbound calls only tolocalhost:8000(Stellar quickstart),localhost:4015(council),localhost:4010(provider),localhost:4035(network-dashboard).EXPECTED_EVENTSfield selections (which keys are strict-asserted vs deep-stripped — seeALWAYS_SKIP_FIELDSintypes.ts).GET /api/v1/dashboard/pp/listbecause no single-PP GET endpoint exists today).