Skip to content

Watch SAC events by topic filter with cursor-paged drains; restore periodic topology re-sync - #16

Merged
AquiGorka merged 1 commit into
mainfrom
fix/mainnet-watcher-sac-flood
Jul 31, 2026
Merged

Watch SAC events by topic filter with cursor-paged drains; restore periodic topology re-sync#16
AquiGorka merged 1 commit into
mainfrom
fix/mainnet-watcher-sac-flood

Conversation

@AquiGorka

Copy link
Copy Markdown
Contributor

Problem

The mainnet dashboard showed nothing from the 2026-07-30 T3 demo run (Councils Cheshire/Salem, Provider Orion, deposit/send/withdraw) despite a healthy watcher and correct council-platform registrations.

Root cause (all measured against mainnet):

  1. XLM SAC flood + cursor skip. The watcher subscribed to the raw XLM SAC contract, which emits 200+ events per ledger on mainnet. A 100-limit getEvents page never spanned even one ledger, and pollTick then set lastLedgerSeen = res.latestLedger — jumping the cursor ~15 minutes of ledgers per 5s tick and silently dropping every Moonlight event (provider_added, deposits, settlements). The cold-start/back-fill walks had the same flood problem: lastLedgerInPage + 1 stepping plus a 50-page cap bounded a "24h" walk to ~2 minutes of mainnet ledgers.
  2. Topology frozen. The hourly re-sync had been removed; refresh only fired at boot, on unknown contract_initialized (pending 120s, then permanently blacklisted), or piggybacked on provider_added — which the flood dropped. Salem stayed name-only (no channel/provider/jurisdiction) for hours after council-platform had everything.

Changes

  • Topic-filtered SAC subscriptions — the SAC is never watched raw. transfer patterns pinned to known channel addresses (deposit + settlement) and fee patterns pinned to registered PP keys (bundles), one filter per getEvents call. Patterns verified against the real mainnet events (transfer = 4 topics, fee = 2).
  • Gap-free cursor-paged drains — shared poll position advances only across RPC-scanned positions (response cursor or scannedTo + 1), never latestLedger jumps. Handles both measured RPC behaviors: bounded scans that return a below-head cursor with a partial page, and -32001 processing limit rejections (windowed endLedger requests, halving on rejection). Store dedup is now id-based beyond the 20-entry ring so designed re-read overlap can't double-count.
  • Periodic topology re-sync (60s) — backstop for DB-only registrations (channels, jurisdictions, labels) that emit no chain event. provider_added piggyback refresh kept for fast linkage.
  • Adoption TTL 120s → 1h, with drain-triggered refreshes throttled to one per 30s (immediate for brand-new unknowns). The old TTL permanently blacklisted the demo councils because their platform registration landed minutes after deploy.
  • README brought to reality: subprotocol moonlight.network.v2, v2 snapshot shape, channel_bundle kind, new sync architecture (hourly re-sync text was stale).

Verification

  • deno task check / lint / fmt:check / test green (58 tests; new coverage for the query builder, windowed/cursor drain semantics, processing-limit fallbacks, and store dedup).
  • Booted this branch locally against live mainnet (read-only): the 24h cold-start backfill ingested the full run — both council_formed, provider_added (Orion), the 18:59:32Z deposit (20.05 XLM, 00c81d08…), 19:00:21Z bundle (f69effe0…), 19:01:06Z settlement (775ddecc…), plus the later 20:48–21:07Z cycles — 11 events, eventsLast24h: 11, activePPs: 1, and Salem's topology complete (channel CCLT…, Provider Orion, AU). Cold start walks the full 24h in ~7 windowed requests per filter and completes in ~40s; the 5s forward tick resumes at the head cursor in one request per filter.

Deploying this to moonlight-mainnet-network-dashboard (tagged main push) and restarting is what makes the mainnet dashboard pick the run up — the run must be within the 24h window at record time (or re-run the demo actions), since the no-persistence design is unchanged.

…ins; restore periodic topology re-sync.

The mainnet XLM SAC emits hundreds of events per ledger, so the unfiltered
contractId subscription returned pages that never spanned a full ledger while
the poll cursor jumped to latestLedger, dropping every Moonlight event; scans
are now one filter per call, endLedger-windowed with processing-limit halving,
and consumption advances only across RPC-scanned positions.
@AquiGorka
AquiGorka merged commit 17288ab into main Jul 31, 2026
5 checks passed
@AquiGorka
AquiGorka deleted the fix/mainnet-watcher-sac-flood branch July 31, 2026 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant