Skip to content

relay: batch audit-log appends per community (T2a write-amp)#2126

Open
tlongwell-block wants to merge 1 commit into
mainfrom
eva/wamp-t2a-audit-batch
Open

relay: batch audit-log appends per community (T2a write-amp)#2126
tlongwell-block wants to merge 1 commit into
mainfrom
eva/wamp-t2a-audit-batch

Conversation

@tlongwell-block

Copy link
Copy Markdown
Collaborator

What

Batch audit-log appends per community instead of one transaction + advisory-lock round trip per entry. T2a of the write-amplification workstream — the audit path is one commit per message today; batching amortizes it to ~1/N under load.

Design

  • AuditService::log_batch — single-community atomic multi-row append. Head re-read inside the txn under the same per-community advisory lock; entries chain in input order. log() delegates to a batch of one, so there is exactly one chain-append implementation. Mixed-community batches are rejected (MixedBatch) before any lock is taken.
  • Worker drain — up to BUZZ_AUDIT_BATCH_MAX entries (default 100) per BUZZ_AUDIT_BATCH_INTERVAL_MS window (default 50 ms), partitioned FIFO-per-community; each community's run appends atomically. Cross-community order carries no chain meaning.
  • Failure classification (AuditError::is_deterministic) — SQLSTATE 22/23 are entry-dependent; everything else is infrastructure, retried with backoff (3 attempts). An outage never marks healthy entries as poison.
  • Poison bisection — deterministic failures bisect the slice, so a poisoned batch of N drops exactly the 1 bad entry, never the other N−1. Cross-tenant isolation holds within one drained batch.
  • Shutdown drain reuses the same batched flush path.

Testing

  • 11/11 Postgres-gated audit tests green, incl. batch chaining, mixed-batch rejection, poisoned-batch atomicity.
  • Bisection property tests: 8-entry batch with mid-batch poison → exactly 7 committed, chain verifies; cross-tenant poison isolation.
  • Full cargo test -p buzz-relay: 687 pass / 1 fail — the 1 is the known mesh_demo failure that is also red on untouched main.
  • Before/after benchmark (same harness as relay: skip TTL deadline bump for known-permanent channels (T1a write-amp) #2125) queued next; numbers will be added here.

Part of the write-amp series; sibling PR: #2125 (T1a TTL skip, measured 11→10 commits/msg).

Amortize the per-message audit cost: the worker now drains up to
BUZZ_AUDIT_BATCH_MAX entries (default 100) per BUZZ_AUDIT_BATCH_INTERVAL_MS
window (default 50ms) instead of one txn + advisory lock round trip per
entry.

- AuditService::log_batch: single-community atomic multi-row append.
  Head re-read inside the txn under the same per-community advisory
  lock; entries chain in input order; log() delegates to a batch of one
  so there is exactly one chain-append implementation. Mixed-community
  batches are rejected (MixedBatch) before any lock is taken.
- Worker flush partitions FIFO-per-community, appends each community's
  run atomically. Cross-community order carries no chain meaning.
- Failure classification (AuditError::is_deterministic): SQLSTATE 22/23
  are entry-dependent; everything else is infrastructure and retried
  with backoff (3 attempts) — an outage never marks healthy entries as
  poison. Deterministic failures bisect the slice so a poisoned batch
  of N drops exactly 1 entry, never the other N-1.
- Shutdown drain reuses the same batched flush path.

Postgres-gated tests: batch chain linkage + verify, mixed-batch
rejection without writes, poisoned-batch atomicity + deterministic
classification, bisection drops exactly the poison with chain intact,
cross-tenant poison isolation in one drained batch.

Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
@tlongwell-block
tlongwell-block requested a review from a team as a code owner July 19, 2026 17:07
@tlongwell-block

Copy link
Copy Markdown
Collaborator Author

Not ready at d1e632a94; two merge blockers against the approved T2a contract.

  1. Infrastructure failure still drops up to the entire healthy batch. append_isolating_poison retries an infra error three times, then reaches the generic terminal arm and drops entries.len() (state.rs:1218-1241). With the default that is up to 100 healthy audit intents lost for a transient outage. The approved plan explicitly required infra retry with backpressure, and the original review rejected widening one dropped intent to 100. The bounded channel does not provide backpressure while the worker is sleeping and then discarding the batch; after ~600ms it resumes consumption. Preserve the batch and stop/drain/retry so producers actually backpressure, or otherwise prove an equivalent no-widening policy. Add a deterministic test with a fail-N-times fake/appender showing healthy entries are not discarded after the retry threshold.

  2. The runnable-from-tip evidence gate is absent. The approved plan requires T2a evidence for kill-9/restart chain continuity, multi-replica advisory-lock contention, and invariant equivalence, runnable from the pushed lane tip. This diff has good Postgres tests for batching, poison isolation, and tenant partitioning, but no kill/restart or concurrent-service proof/script. The PR body also omits the required disclosure that hard crash widens the accepted-but-memory-queued audit-loss window to one flush interval. Add those runnable gates and disclosure before approval.

Hardening while touching config: BUZZ_AUDIT_BATCH_MAX accepts any positive u64 cast to usize, but each row uses 9 bind parameters and Postgres has a finite bind limit; cap/validate the configured maximum rather than allowing an oversized allocation/query to enter the failure path.

Verified exact head d1e632a94f4f0f6695b814e6908abf2e17bef23c; diff is clean, cargo test -p buzz-audit passes (10/9 ignored), and I independently ran the Postgres poison-isolation relay test successfully. CI lint/unit/relay E2E are green so far; these are contract/correctness gaps, not compile failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant