relay: gate push enqueue on live leases; batch matcher pipeline (T1b/T1a-repair/T2b)

[tlongwell-block]

Part of the write-amplification work (with #2125 T1a TTL-skip and #2126 T2a audit batching). Three coupled fixes to the push pipeline's per-message DB cost:

T1b — push gate (migration 0023)

Skip the push_match_queue enqueue for communities with no active, endpoint-enabled, unexpired push lease. Today every durable message in a lease-less community (most of them) pays the full matcher cost — enqueue + claim + lease scan + delete — to conclude "notify no one".

Lost-wake correctness: a naive EXISTS check could read "no lease" while a lease activation commits concurrently, silently dropping that user's first wake. Closed with a per-community advisory lock held to txn end:

• event inserts take it SHARED (concurrent with each other — no new serialization on the hot path),
• lease transitions that can turn eligibility on (accept_lease_event, replace_lease) take it EXCLUSIVE.

The conflict forces a total order: either the event's check sees the committed lease, or the activation strictly follows the event's commit — in which case no wake was owed. A backfill on activation re-enqueues recent events as product-recovery coverage (not part of the proof).

T1a repair — TTL trigger shared lock (migration 0024)

The 0022 trigger takes FOR UPDATE on the channel row before testing ttl_seconds, so every durable message in a permanent channel serializes on that tuple at commit time (deferred trigger), each holding the lock across its WAL flush. Observed live at 200 QPS: commit latency 0.07ms → ~15ms, non-CPU DB load ~9/vCPU with CPU under 45%. This is the current throughput ceiling for a hot channel.

Repair keeps 0022's stale-prefetch proof but moves synchronization to a per-channel advisory lock: event inserts take it SHARED (permanent-channel commits proceed concurrently, no tuple lock, no row write); a TTL transition (update_channel) takes the same key EXCLUSIVE before its UPDATE. Either ordering closes the stale-NULL hole. Ephemeral channels still pay their conditional row update — only they need it.

T2b — batched matcher pipeline

The relay push matcher claimed one job per transaction (claim + event load + lease scan + membership check + complete, per job) and polled every 250ms forever when idle. Now:

• Per-community batch claim — single SKIP LOCKED CTE, cap 64, one claim fence (claim_id) shared by the batch.
• Events, leases, and (channel, lease-author) membership pairs each load in one statement per batch; completes/retries are set-wise under the fence. Constant statement count regardless of batch size.
• Set-wise wake enqueue (round 2, 3173861ed): Max's red-team of cf018e1bd found every successful (event, lease) match still called enqueue_push_wake per pair — its own transaction + lease FOR UPDATE, keeping live-lease workload at O(events × matching leases) commits. Fixed: matcher evaluation is now pure (match_job, no DB access); all wake requests for a claimed batch flush through one push::enqueue_wakes transaction — a single ordered FOR UPDATE lease revalidation over the distinct (author, installation) set (ORDER BY pins lock order against replace_lease and sibling pods), one multi-row UNNEST insert under the existing (community, endpoint_hash, event_id) dedup key, and index-aligned per-request outcomes (Enqueued / Duplicate / InactiveLease). enqueue_wake is a batch-of-one wrapper — one code path. A failed flush sends only contributing jobs back for an idempotent rematch. With this, the constant-statements-per-batch claim holds through the whole pipeline.
• Poison reap moved off the claim path to a 30s sweep — its scan isn't served by the due partial index, so running it inside every claim made claims slowest exactly when a backlog needed them fastest.
• Idle backoff 250ms → 2s for matcher and delivery workers (an idle relay burned ~83 txns/10s just polling).
• The per-job claim API is deleted, not deprecated — no second code path to drift.

Tests

New PG tests: batch fence contract; single-community batch claim; T1a barrier (8 concurrent permanent-channel commits held at a barrier — zero channels tuple locks, all commits proceed, row untouched); T1b forced lost-wake ordering + activation backfill; exhausted-job sweep; setwise_enqueue_maps_outcomes_per_request (fresh insert, duplicate-of-committed, same-batch dedup collision, stale generation, unknown lease, second live lease in one batch). Migration inventory pins for 0023/0024, including a comment-stripped assert that no FOR UPDATE re-enters the executable trigger body.

Full buzz-db + buzz-relay suites green at 3173861ed on a 0001–0024 database (including all --ignored PG tests): buzz-db 202/202, buzz-relay 715/716. Lone red: api::mesh_demo::demo_join_forwarded_arm_round_trips_echo under the full parallel -p buzz-relay run — reproduced identically on main (c1bca1b), passes in isolation on both; pre-existing, unrelated.

Independent verification (Max, at 3173861ed)

Fresh isolated PG17 + newly built release relay, new identity: focused patch tests green; two channels driven concurrently at ~300 QPS × 10 s over 60 WebSocket connections — 2,728/2,728 accepted, exact PG reconciliation, 0 duplicate event IDs; after drain: 0 match-queue rows, 0 wake-outbox rows, 0 eligible leases, 0 channel tuple locks; relay log free of errors/panics/deadlocks. Round-1 blocker (per-match enqueue) confirmed structurally closed; no remaining merge blocker.

Second independent E2E round (Max, same tip, separate DB/relay/identity): permanent + ephemeral (ttl=3600) + TTL-transitioned (permanent→TTL→permanent) channels; threads and tags reconciled exactly in raw event rows; 3 channels at 200 QPS × 8 s over 72 connections — 2,055/2,055 accepted, exact per-channel PG reconciliation, 0 duplicate IDs, all no-lease invariants held after drain, relay log clean. Across both rounds: 4,783/4,783 accepted messages present as durable PG rows — zero lost.

Benchmarks (at 3173861ed)

Wamp harness (same rig as #2125/#2126: isolated PG + mock push gateway, 180 s steps, idle-control-subtracted net commits/msg, quiesce-complete each step, wake outbox drained to delivered). All steps 0 rejected; q100 accepted 18,004/18,004 in both scenarios.

Scenario B — live push lease (the workload T2b exists for):

qps	baseline (main)	after	delta
5	~17.2–17.7	12.66	−4.6
10	~17.2–17.7	11.83	−5.5
30	~17.2–17.7	11.27	−6.1
50	~17.2–17.7	10.83	−6.5
100	~17.2–17.7	10.50	−6.9

The baseline was flat in qps (per-message cost never amortized). The after-curve improves with load — the batching signature: constant per-batch statements spread over bigger claimed batches. At 100 qps, scenario B (10.50) is cheaper than the old scenario A baseline (~11.0): the ~6.5 commits/msg the live lease used to add is nearly amortized away.

Scenario A — hot permanent channel, zero leases (T1b push-gate + T1a repair):

qps	baseline (main)	after	delta
5	~11.0	5.62	−5.4
10	~11.0	5.80	−5.2
30	~11.0	5.93	−5.1
50	~11.0	5.96	−5.0
100	~11.0	5.98	−5.0

Roughly half the per-message commit cost gone: with no live lease, the push-gate skips the per-message push transactions outright instead of batching them. Latency held healthy throughout (scenario A q100: p50 4.3 ms / p99 6.1 ms; scenario B q100: p50 4.0 ms / p99 7.4 ms).

Raw artifacts: /tmp/wamp-t2b-bench/{A,B}/ (per-step bench/summary/latency/statement captures).

Lock-domain note: buzz_push_gate: (0023), buzz_channel_ttl: (0024), and the audit lock (buzz_audit:, #2126) are distinct key domains; the deferred TTL lock is acquired at COMMIT, after any push-gate shared lock taken during insert, and no path takes two domains exclusively — no cross-domain deadlock ordering.