Skip to content

Optimize backfill#854

Draft
cjonas9 wants to merge 13 commits into
backfill-testfrom
optimize-backfill
Draft

Optimize backfill#854
cjonas9 wants to merge 13 commits into
backfill-testfrom
optimize-backfill

Conversation

@cjonas9

@cjonas9 cjonas9 commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

What

This PR introduces several experimental changes to backfill for the sake of drastically reducing its runtime (currently, we see its runtime decrease from ~7 hours -> <3 hours). These changes flatten the curve of ledger insertion time vs. count of ledgers inserted, a curve that grew explosive over the last ~6 months as ledger complexity increased. These changes are:

  • Twin index rebuild: during backfill only, transactions is swapped for an indexless "bulk-load twin" (its DDL derived at runtime from the canonical schema). finalize copies rows back into the canonical table and rebuilds the deferred tx + events indexes, then a final frontfill catches up ledgers that closed during the finalize phase (since it can take about an hour)
  • Fixed-arity batching: tx/event writers accumulate rows and flush in fixed-size batches (e.g. 3,000 events), keeping the INSERT's bind-variable count constant so SQLite reuses the prepared statement instead of re-parsing variable-arity SQL
  • Decoding pipeline: ledger XDR is pre-decoded on separate goroutines into a channel feeding ingestRange, overlapping decode CPU work with DB writes instead of decoding inline
  • Backfill-tuned SQLite pragmas: backfill (and index-build) run on separate DB sessions with their own DSN pragmas (synchronous=OFF, large cache, etc.) instead of the serving pragmas

Of these changes, the only "truly touches hot-ingest path" change is fixed-arity batching, but functionally, this introduces no behavioral differences. the only thing it changes is when within the transaction the INSERTs execute, not what the transaction commits. Before, each ledger's events were INSERTed immediately during ingestLedgerCloseMeta in variable-sized batches (≤1000 rows). We just fix the batch size so rows accumulate in the in-memory pending slice and are written by flushPending() at Commit (plus any full 3000-row batches along the way). This still commits the same rows in the same order in the same write txs.

Why

Backfill has demonstrated to have an explosive tail latency as ledgers are ingested into the DB due to the indexing overhead incurred by ever-compounding tx, event, and other inserts. It was never intended to take 7 hours in the first place, and this is a regression that's snuck up on us as ledger composition has grown more complex. Every chunk of 960 ledgers that we backfill now takes a constant time of ~45 seconds to insert, as opposed to exploding from ~40 seconds to ~5-6+ minutes by the end of a backfill.

This is strictly necessary if we wish to maintain architectural work done for the performance evaluation epic: GHA imposes a 6 hour cap on job length, and backfill takes way too long. Because we intend on using the DB backfilled in the backfill test (see PR #830) for the RPC stood up for the stellar-rpc-blaster endpoint evaluation test, we must optimize backfill first -- these CI jobs are inherently serial.

Known limitations

N/A

@cjonas9 cjonas9 changed the base branch from main to backfill-test July 10, 2026 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant