Skip to content

feat(testnet): events-capture framework + EXPECTED_EVENTS for testnet-main + lifecycle-testnet-verify#112

Merged
AquiGorka merged 4 commits into
mainfrom
feat/events-capture-framework
Jun 5, 2026
Merged

feat(testnet): events-capture framework + EXPECTED_EVENTS for testnet-main + lifecycle-testnet-verify#112
AquiGorka merged 4 commits into
mainfrom
feat/events-capture-framework

Conversation

@AquiGorka

Copy link
Copy Markdown
Contributor

Summary

Adds a small framework that asserts the WebSocket event surfaces of provider-platform (/api/v1/providers/:pp/events/ws) and network-dashboard-platform (/api/v1/network/ws) against a strict-order, strict-value declaration inlined at the top of each testnet flow script.

Two scripts produce events in the canonical testnet/run-all.sh runner — testnet/main.ts (suite 1, payment flow) and lifecycle/testnet-verify.ts (suite 3, UC2 governance). Each script now declares an EXPECTED_EVENTS const at the top. The harness subscribes to both WS surfaces before the script's chain interactions, runs the script's exported main() in-process, tails for EVENTS_CAPTURE_TAIL_MS (default 5000), then diffs captured against expected and writes a structured run report.

Suite 2 and suite 4 (OTEL trace verifiers) stay as-is — they do no chain ops, so there's nothing to capture.

Cross-PR dependency — read this first

This PR depends on Moonlight-Protocol/network-dashboard-platform#5. That PR refactors network-dashboard-platform from hourly periodic resync to event-driven topology + back-fill on adoption, which is what makes council_formed (the first network event in both scripts' EXPECTED_EVENTS) fire deterministically. The local-stack smoke in this PR was validated against that refactor; without it, the network-side assertion would still fail on the first event.

Merge order: n-d-p#5 first, then this.

What's in this PR

  • testnet/events-capture/types.ts — shared types + the ALWAYS_SKIP_FIELDS set (deep-stripped from both expected and captured before equality check). Covers ts, id, occurredAt, ledger, bundleId/bundleIds, txHash/txId, contract-id family, councilName (timestamp-embedded literal in both scripts), reason (free-form error.message at the producer site).
  • testnet/events-capture/subscribe.ts — per-PP + network-wide WS clients. Per-PP polls GET /api/v1/dashboard/pp/list until the operator's PP appears in the DB, then upgrades with Sec-WebSocket-Protocol: bearer.<jwt>. Network is public, snapshot frame is dropped (PM-acked: state-at-connect-time depends on prior runs), every subsequent LiveFrame is captured. Uses Deno's built-in WebSocket — no new dep.
  • testnet/events-capture/assert.ts — strict-order, strict-value diff with the always-skip set deep-stripped. Returns a structured per-index per-subscriber result; one subscriber failing doesn't mark another as failing.
  • testnet/events-capture/report.ts — stdout summary (PASS line per subscriber, or full mismatch diff with expected/captured per index) + JSON dump under runs/<run-id>/report.json. runs/ is gitignored.
  • testnet/events-capture/harness.ts — CLI orchestrator. Derives the operator/alice/bob keypairs from MASTER_SECRET (matching the scripts' own deriveKeypair(ROLES.PP / ROLES.ALICE / ROLES.BOB, 0)), acquires the dashboard JWT, opens both subscribers, dynamic-imports the script module, runs main(), tails, closes, asserts, writes report, exits 0/1/2.
  • testnet/main.ts and lifecycle/testnet-verify.ts — inlined EXPECTED_EVENTS. Each script's main() is now exported and the bottom-of-file invocation is guarded with if (import.meta.main) so direct deno run still works.
  • testnet/run-all.sh and testnet/run-local.sh — suites 1 + 3 now invoke the harness instead of run-tempo.sh / direct deno run. run-local.sh exports defaults for NETWORK_DASHBOARD_PLATFORM_URL and MASTER_SECRET when not set by the caller.

EXPECTED_EVENTS values were derived from Phase 0 audit + Phase 3 smoke

  • Per-PP: 12 events per script. weight values come from the bundle-operation counts in lib/client/{deposit,send,withdraw}.ts × the mempool defaults (MEMPOOL_EXPENSIVE_OP_WEIGHT=10 × MEMPOOL_CHEAP_OP_WEIGHT=1). amount strings come from the deposit / send / withdraw amount + fee math; entityName / jurisdictions / depositorAddress / recipientAddress from the script's literals + the MASTER_SECRET-derived keypairs.
  • Network: 5 events per script. The processRawEventBatch dedup in network-dashboard-platform drops channel_bundle whenever the tx also produced channel_deposit or channel_settlement, so the visible sequence is council_formed → provider_added → channel_deposit → channel_bundle (send) → channel_settlement. Validated empirically in the Phase 3 smoke (after the n-d-p refactor lands).

Test plan

  • deno fmt --check over the changed files (testnet/events-capture/, testnet/main.ts, lifecycle/testnet-verify.ts, testnet/run-all.sh, testnet/run-local.sh) — clean.
  • deno lint over the same set — clean. (Repo-wide deno lint surfaces ~23 pre-existing problems in unrelated files such as setup-pp.ts:35 and playwright/tests/full-flow.spec.ts:44. Same out-of-PR-scope situation as the prior PR's pre-existing test debt.)
  • Local-stack smoke against the harness invocation for both scripts:
    testnet-main             — perPp: 12/12  network: 5/5  PASS
    lifecycle-testnet-verify — perPp: 12/12  network: 5/5  PASS
    
    Run artifacts: testnet/events-capture/runs/testnet-main-2026-06-05T16-31-48-970Z/report.json and testnet/events-capture/runs/lifecycle-testnet-verify-2026-06-05T16-33-57-529Z/report.json. No testnet RPC / contract / signer was touched — outbound calls only to localhost:8000 (Stellar quickstart), localhost:4015 (council), localhost:4010 (provider), localhost:4035 (network-dashboard).
  • Reviewer pass on the EXPECTED_EVENTS field selections (which keys are strict-asserted vs deep-stripped — see ALWAYS_SKIP_FIELDS in types.ts).
  • Reviewer pass on the harness's per-PP open loop (GET /api/v1/dashboard/pp/list because no single-PP GET endpoint exists today).

AquiGorka added 3 commits June 5, 2026 13:42
… orchestrator

A small framework for asserting the WS event surfaces of provider-platform
(`/api/v1/providers/:pp/events/ws`) and network-dashboard-platform
(`/api/v1/network/ws`) against a strict-order, strict-value declaration
inlined at the top of each testnet flow script.

- subscribe.ts: per-PP + network-wide WebSocket clients using Deno's
  built-in `WebSocket`. Per-PP polls `GET /api/v1/dashboard/pp/list` until
  the operator's PP appears in the DB (no single-PP endpoint exists), then
  upgrades with `Sec-WebSocket-Protocol: bearer.<jwt>`. Network is public.
- assert.ts: deep-strips the ALWAYS_SKIP set (`ts`, `id`, `occurredAt`,
  `ledger`, contract-id family, `councilName`, `reason`, etc.) from both
  expected and captured, then walks each subscriber's events in lockstep
  and emits a structured per-index diff.
- report.ts: stdout summary + JSON dump under
  `events-capture/runs/<run-id>/report.json` (run dir is gitignored).
- harness.ts: CLI entrypoint. Derives the operator/alice/bob keypairs
  from `MASTER_SECRET` (matching the scripts' own derivation), acquires
  the dashboard JWT, opens both subscribers, runs the script's exported
  `main()`, tails for `EVENTS_CAPTURE_TAIL_MS` (default 5_000), closes
  the sockets, asserts, writes the report, exits 0 / 1 / 2.
…verify.ts

Each flow script now declares its expected per-PP and network-wide event
streams at the top of the file. The harness reads `EXPECTED_EVENTS`
verbatim and asserts captured == expected after stripping the
nondeterministic skip set. Values for the strict-equality fields
(`weight`, `entityName`, `jurisdictions`, `amount`, `ppLabel`,
`depositorAddress`, `recipientAddress`, `newSlot`, `kind`,
`scope.ppPublicKey`) come from a Phase 3 local-stack smoke that exercised
both scripts end-to-end against a fresh `MASTER_SECRET`-derived deploy.

Per-PP count for both scripts is 12 events:
  channel.provider_added → deposit cycle (mempool / executor / verifier /
  bundle.deposit_completed) → send cycle (3 events) → withdraw cycle
  (4 events incl. bundle.withdraw_completed).

Network-wide count for both scripts is 5 events (after the
network-dashboard's `processRawEventBatch` dedup which drops
`channel_bundle` whenever the tx also emitted a money-flow event):
  council_formed → provider_added → channel_deposit →
  channel_bundle (send fee) → channel_settlement.

`council_formed` was provisionally OMITted in the Phase 0 catalogue
(item B) because the cold-start lookback race was unclear. The Phase 3
smoke showed it landing deterministically once
network-dashboard-platform's contract-init listener adopts a new council
and back-fills from the deploy ledger, so it's reinstated.

Each script's existing `main()` is now `export async function main()`,
and its bottom-of-file invocation is guarded with `if (import.meta.main)`
so direct `deno run ...` still works AND the harness can dynamic-import
the module to read `EXPECTED_EVENTS` and call `main()` in-process.
…r suites 1+3

`testnet/run-all.sh` and `testnet/run-local.sh` now invoke
`events-capture/harness.ts` instead of `run-tempo.sh` (suite 1) and the
direct `lifecycle/testnet-verify.ts` `deno run` (suite 3). Suites 2 and
4 (the OTEL trace verifiers) stay as-is — no chain ops, nothing to
assert.

`run-local.sh` additionally exports defaults for
`NETWORK_DASHBOARD_PLATFORM_URL` (the harness's network-wide WS target)
and `MASTER_SECRET` (so the harness's keypair derivation matches the
flow scripts') when not set by the caller.
…fault tail

Two robustness fixes for the network subscriber, surfaced by running the
harness through testnet/run-local.sh against the live local stack:

1. Auto-reconnect on close (network + per-PP). The previous one-shot
   `new WebSocket(...)` would silently lose every subsequent frame
   when the connection dropped — e.g. on the server-side idleTimeout
   that fires when chain events don't arrive for tens of seconds
   after `onopen`. The network subscriber also parses the new
   `SnapshotFrame.recent` on every reconnect (newest-first per the
   server's `seedRecent` contract) and replays any `NetworkEvent.id`
   it hasn't observed yet, recovering events that landed during the
   disconnect window. Backoff is 250 ms → 5 s, capped.

2. `DEFAULT_TAIL_MS` bumped from 5 000 to 30 000. The original 5 s
   default was tight enough for the per-PP-only smoke runs we tested
   with directly, but the network subscriber's first event
   (`council_formed`) lands 25-40 s after the back-fill scan picks up
   a freshly-deployed council — comfortably past 5 s. 30 s also
   matches the value used in the Phase 3 smoke runs that informed
   the EXPECTED_EVENTS catalogues. Override via
   `EVENTS_CAPTURE_TAIL_MS` per run.

With these + the n-d-p `fix(sync): drop rescanRollingWindow from
refreshTopology + raise WS idleTimeout` change, eight back-to-back smoke
runs through ./testnet/run-local.sh (testnet-main × 5 + lifecycle × 3)
assert 12/12 perPp + 5/5 network green with no flake.
AquiGorka added a commit to Moonlight-Protocol/network-dashboard-platform that referenced this pull request Jun 5, 2026
…roring provider-platform watcher pattern) (#5)

## Summary

Refactors the topology / event-discovery surface to mirror
`provider-platform`'s `event-watcher` pattern: **sync at boot, set
listeners, update state as events come in**. The previous hourly
periodic re-sync is gone; the new-council fallback that depended on a
GitHub-sourced WASM hash registry is gone; what's left is a tight
chain-event-driven loop that publishes within ~5 s of a new council
going on-chain.

Validated by a local-stack end-to-end smoke
(`/Users/theahaco/repos/tmp/add-events-capture-framework-1/`) where two
flow scripts deploy a fresh council mid-test, and the network-dashboard
WS surfaces the resulting `council_formed → provider_added →
channel_deposit → channel_bundle → channel_settlement` sequence in real
time.

## The four architectural changes

1. **Remove the hourly topology resync** (`src/core/sync/scheduler.ts`).
The scheduler now does only the 60-second rolling-window sweep (memory
cap on the 24h counter window). Topology arrives at boot via
`fetchCouncilTopology` and updates via the contract-init listener.

2. **Drop the WASM hash registry** (`src/core/sync/wasm-registry.ts`
deleted; `src/main.ts`, `src/core/sync/topology-refresh.ts` updated).
The registry was sourced from `Moonlight-Protocol/soroban-core`'s GitHub
releases and returned `HTTP 403` for any unauthenticated fetch — which
meant the new-council fallback was effectively a permanent no-op in
local-dev. `isKnownChannelAuthHash` / `isWasmRegistryReady` /
`knownHashCount` had no callers after change #3.

3. **Ungate the contract-init listener**
(`src/core/sync/contract-init-listener.ts`,
`src/core/sync/soroban-watcher.ts`). The watcher always polls
`getEvents` for `contract_initialized` from unknown contracts (no more
`isContractInitListenerEnabled()` gate). Each unknown is held in
`pendingAdoption` with an `observedAtLedger` for back-fill (see #4) and
a TTL of `PENDING_TTL_MS = 120 s` — long enough that a council
registered AFTER its on-chain deploy still gets adopted on a subsequent
tick. Past the TTL the contract is cached as `notMoonlight` so a chatty
unrelated contract can't drag us into infinite topology refreshes.

4. **Back-fill on adoption**
(`src/core/sync/soroban-watcher.ts:backfillFromLedger`, threaded via
`drainPendingAdoptions`'s new `backfillFromLedger` dep). Forward polling
resumes from `lastLedgerSeen + 1`, so any chain event between the deploy
ledger and the adoption tick (most notably `provider_added` and the
first SAC `transfer`) would otherwise be lost. The back-fill scan walks
every currently-watched contract from the earliest adopted
`observedAtLedger` to head, maps each event, and publishes via the bus.
`networkState.recordEvent`'s dedup makes this safe to run concurrent
with `pollTick`.

The hot-path behaviour now matches
`provider-platform/src/core/service/event-watcher/index.ts`:
- **Boot**: `initFromDb` → `ensureWatcher` per ACTIVE membership.
- **Listeners**: per-contract Soroban poll, narrowly-scoped.
- **State updates as events come in**: handler emits via `emitForPp` /
`emitForBundles`.
- **Hot-path additions**: `addCouncilWatcher` / `addProviderAddress`
exported for API handlers to call when new entities are created.

For network-dashboard the analog is now: `fetchCouncilTopology` at boot,
then chain-event-driven adoption via `contract_initialized` →
`refreshTopology` → `backfillFromLedger`.

## Cross-PR dependency

This PR must merge **before** Moonlight-Protocol/local-dev#112. That PR
adds a strict-order, strict-value assertion harness whose
`EXPECTED_EVENTS.network` begins with `council_formed` — which only
fires deterministically after this refactor's `backfillFromLedger` is in
place. Without this PR, the local-dev assertion would still fail on the
first event index.

## Test plan

- [x] `deno fmt --check` — clean (32 files)
- [x] `deno lint` — clean (21 files)
- [x] `deno task test` — **17 / 0**
- [x] Local-stack end-to-end against a fresh-deploy flow script
(testnet/main.ts + lifecycle/testnet-verify.ts): 5/5 network events
captured in correct order per script, both with `council_formed` landing
first via the back-fill path.
- [ ] Reviewer pass on the `notMoonlight` TTL choice (`PENDING_TTL_MS =
120 s`) — that's the upper bound between a council's on-chain deploy and
council-platform's `PUT /council/metadata` in the testnet scripts.
- [ ] Reviewer pass on the back-fill scope — `backfillFromLedger` scans
every watched contract from `minAdoptedLedger`, not only the
newly-adopted ones, because cross-council SAC events (XLM `transfer` /
`fee`) are mapped via the linkage table that `refreshTopology` just
installed.
@AquiGorka
AquiGorka merged commit acbecb1 into main Jun 5, 2026
7 of 8 checks passed
@AquiGorka
AquiGorka deleted the feat/events-capture-framework branch June 5, 2026 20:35
AquiGorka added a commit that referenced this pull request Jun 8, 2026
…all/run-local (#113)

## Summary

- The events-capture harness shipped in #112 required a wrapper-level
`MASTER_SECRET` to derive operator/alice/bob keys ahead of subscriber
bring-up.
- When `run-local.sh all` runs suite 1 (`testnet-main`) and suite 3
(`lifecycle-testnet-verify`) back-to-back under the single root the
wrapper exports, both suites derive the **same** PP operator key — suite
3's `POST /dashboard/council/join` then hits provider-platform's "PP
already active for this council" check and 409s. `run-all.sh` has a
related failure mode: it doesn't export `MASTER_SECRET` at all, so the
harness throws at parse-args time.
- Fix: layer one more SHA-256 stage on top of `masterSeedFromSecret`
keyed on the `scriptName`, and fall back to a freshly-minted random root
when `MASTER_SECRET` is unset.

## What changed in `testnet/events-capture/harness.ts`

- `derivePerSuiteSecret(rootSecret, scriptName)` — `SHA-256(rootSeed ||
`suite/\${scriptName}`)` → 32 bytes → Stellar SK via
`Keypair.fromRawEd25519Seed(...).secret()`. Matches the
SHA-256-of-concatenation style already in `lib/master-seed.ts`.
- Before invoking the script's `main()`, the harness re-exports
`MASTER_SECRET` to the per-suite SK so the script's own
`Deno.env.get("MASTER_SECRET") + masterSeedFromSecret(...)` derives
matching role keys.
- `parseArgs()` no longer throws when `MASTER_SECRET` is unset — it
mints an ephemeral root, restoring the pre-harness shape of the
underlying scripts and letting `run-all.sh` invoke the harness without
wrapper changes.

## Properties

- Caller's surface unchanged. Pin `MASTER_SECRET` once at the wrapper
level → reproducible across re-runs; leave it unset → ephemeral
per-invocation.
- Same root + same `scriptName` ⇒ identical per-suite SK
(reproducibility).
- Different `scriptName` ⇒ disjoint role keys (suite 1 and suite 3
isolated).

## Test plan

- [x] `deno fmt --check` clean (`testnet`)
- [x] `deno lint` clean (`testnet`)
- [x] `deno check events-capture/harness.ts` clean
- [x] `./testnet/run-local.sh all` against local stack — full sweep
PASS:
- Suite 1 — payment flow: 12/12 per-PP events + 5/5 network events, flow
in 76.4s
  - Suite 2 — OTEL verify (payment): 23/23 assertions
- Suite 3 — lifecycle flow: 12/12 per-PP events + 5/5 network events,
flow in 80.6s
  - Suite 4 — OTEL verify (lifecycle): 23/23 assertions
- PP operator keys differ between suite 1 (`GAFWC4R3...`) and suite 3
(`GAX77SRV...`) — per-suite derivation confirmed working.
- [ ] `./testnet/run-all.sh` against deployed testnet — to verify the
deployed-Tempo path still works end-to-end with the no-`MASTER_SECRET`
fallback.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant