Skip to content

Fix mesh reliable echo and stale membership#2041

Open
tlongwell-block wants to merge 1 commit into
mainfrom
perci/fix-mesh-reliability
Open

Fix mesh reliable echo and stale membership#2041
tlongwell-block wants to merge 1 commit into
mainfrom
perci/fix-mesh-reliability

Conversation

@tlongwell-block

Copy link
Copy Markdown
Collaborator

Summary

Fix both inter-relay mesh failures observed on bb-public:

  • restore reliable demo echo delivery after the QUIC stream is established
  • evict stale gossip members and prevent third-party gossip from reporting nonexistent transport connections

Root cause

Reliable stream timeout

The inbound reliable-stream intake loop awaited the session payload before returning to accept and dispatch later frames. The payload therefore could not reach the session receiver that was waiting to echo it, leaving the caller to time out after a successful stream open.

This change dispatches each accepted session concurrently, allowing the intake loop to continue receiving frames while preserving per-session processing.

Zombie gossip membership

Membership was insert-only. Expired Redis ready records could not remove peers already learned through gossip, and surviving relays continually re-advertised those stale records. Gossip application also marked peers Connected even when the record was learned third-hand rather than through an actual transport connection.

This change:

  • evicts records with heartbeats older than 5 minutes during reconciliation
  • removes corresponding transport entries
  • rejects already-stale gossip records so evicted peers cannot be resurrected
  • leaves connection state under the authority of the real transport lifecycle

Live peers remain self-healing through periodic Redis ready-record publication.

Validation

  • cargo test -p buzz-relay-mesh — 34 passed
  • both mesh demo tests passed, including forwarded echo
  • cargo clippy passed for affected packages
  • full repository pre-push suite passed (Rust, desktop, Tauri, and mobile checks/tests)

Live local relay proof

Ran two actual buzz-relay processes with mesh/demo enabled, separate HTTP/health/QUIC ports, and shared Redis:

  1. Relay A claimed a random session and returned owned.
  2. Relay B called the same session and returned forwarded with a byte-identical payload.
  3. Relay A's streams_received increased by 1.
  4. Relay B's streams_opened increased by 1.

Both relay processes and temporary Redis state were cleaned up after the smoke test.

Co-authored-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co>
@tlongwell-block
tlongwell-block requested a review from a team as a code owner July 17, 2026 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant