feat(core): live/historic storage split with per-epoch history buckets - #12144
Closed
muXxer wants to merge 12 commits into
Closed
feat(core): live/historic storage split with per-epoch history buckets#12144muXxer wants to merge 12 commits into
muXxer wants to merge 12 commits into
Conversation
muXxer
force-pushed
the
feat/historic-object-store
branch
2 times, most recently
from
July 11, 2026 14:12
03e7b1a to
2514140
Compare
muXxer
marked this pull request as ready for review
July 11, 2026 14:12
muXxer
marked this pull request as draft
July 11, 2026 16:51
muXxer
added a commit
that referenced
this pull request
Jul 29, 2026
Following the per-epoch history buckets of #12144: the 13 history tables (transaction and event indexes) move from static column families into one shared column family per epoch, created at runtime when an epoch's first checkpoint is indexed. Within a bucket the tables are TaggedDBMaps, separated by a tag byte prefixed to every key, so one family per epoch instead of thirteen keeps the column-family and SST-file counts an order of magnitude lower across a ~100-epoch horizon, and makes a bucket's existence atomic. Transactions are numbered by network order and epochs partition that order contiguously, so each bucket is a disjoint, epoch-ordered segment of every table: queries chain per-bucket scans in epoch order (cursors move into the scan bounds to compose across buckets), digest lookups probe buckets newest first through their bloom filters, and the crash-recovery replay check only consults the checkpoint's own epoch bucket. Pruning becomes one constant-time column-family drop per expired epoch, replacing the compaction-filter machinery wholesale: the per-table filter configurations, their metrics, the pruner watermark, and the time-to-sequence cut are deleted. num_epochs_to_retain_for_indexes now means exactly that. The newest N bucket epochs are kept. On-disk column-family names are the ground truth for which buckets exist; opening rediscovers them and passes them at open with tuned options sharing one block cache.
muXxer
added a commit
that referenced
this pull request
Aug 7, 2026
Following the per-epoch history buckets of #12144: the 13 history tables (transaction and event indexes) move from static column families into one shared column family per epoch, created at runtime when an epoch's first checkpoint is indexed. Within a bucket the tables are TaggedDBMaps, separated by a tag byte prefixed to every key, so one family per epoch instead of thirteen keeps the column-family and SST-file counts an order of magnitude lower across a ~100-epoch horizon, and makes a bucket's existence atomic. Transactions are numbered by network order and epochs partition that order contiguously, so each bucket is a disjoint, epoch-ordered segment of every table: queries chain per-bucket scans in epoch order (cursors move into the scan bounds to compose across buckets), digest lookups probe buckets newest first through their bloom filters, and the crash-recovery replay check only consults the checkpoint's own epoch bucket. Pruning becomes one constant-time column-family drop per expired epoch, replacing the compaction-filter machinery wholesale: the per-table filter configurations, their metrics, the pruner watermark, and the time-to-sequence cut are deleted. num_epochs_to_retain_for_indexes now means exactly that. The newest N bucket epochs are kept. On-disk column-family names are the ground truth for which buckets exist; opening rediscovers them and passes them at open with tuned options sharing one block cache.
muXxer
added a commit
that referenced
this pull request
Aug 7, 2026
Following the per-epoch history buckets of #12144: the 13 history tables (transaction and event indexes) move from static column families into one shared column family per epoch, created at runtime when an epoch's first checkpoint is indexed. Within a bucket the tables are TaggedDBMaps, separated by a tag byte prefixed to every key, so one family per epoch instead of thirteen keeps the column-family and SST-file counts an order of magnitude lower across a ~100-epoch horizon, and makes a bucket's existence atomic. Transactions are numbered by network order and epochs partition that order contiguously, so each bucket is a disjoint, epoch-ordered segment of every table: queries chain per-bucket scans in epoch order (cursors move into the scan bounds to compose across buckets), digest lookups probe buckets newest first through their bloom filters, and the crash-recovery replay check only consults the checkpoint's own epoch bucket. Pruning becomes one constant-time column-family drop per expired epoch, replacing the compaction-filter machinery wholesale: the per-table filter configurations, their metrics, the pruner watermark, and the time-to-sequence cut are deleted. num_epochs_to_retain_for_indexes now means exactly that. The newest N bucket epochs are kept. On-disk column-family names are the ground truth for which buckets exist; opening rediscovers them and passes them at open with tuned options sharing one block cache.
muXxer
added a commit
that referenced
this pull request
Aug 7, 2026
Following the per-epoch history buckets of #12144: the 13 history tables (transaction and event indexes) move from static column families into one shared column family per epoch, created at runtime when an epoch's first checkpoint is indexed. Within a bucket the tables are TaggedDBMaps, separated by a tag byte prefixed to every key, so one family per epoch instead of thirteen keeps the column-family and SST-file counts an order of magnitude lower across a ~100-epoch horizon, and makes a bucket's existence atomic. Transactions are numbered by network order and epochs partition that order contiguously, so each bucket is a disjoint, epoch-ordered segment of every table: queries chain per-bucket scans in epoch order (cursors move into the scan bounds to compose across buckets), digest lookups probe buckets newest first through their bloom filters, and the crash-recovery replay check only consults the checkpoint's own epoch bucket. Pruning becomes one constant-time column-family drop per expired epoch, replacing the compaction-filter machinery wholesale: the per-table filter configurations, their metrics, the pruner watermark, and the time-to-sequence cut are deleted. num_epochs_to_retain_for_indexes now means exactly that. The newest N bucket epochs are kept. On-disk column-family names are the ground truth for which buckets exist; opening rediscovers them and passes them at open with tuned options sharing one block cache.
muXxer
added a commit
that referenced
this pull request
Aug 10, 2026
Following the per-epoch history buckets of #12144: the 13 history tables (transaction and event indexes) move from static column families into one shared column family per epoch, created at runtime when an epoch's first checkpoint is indexed. Within a bucket the tables are TaggedDBMaps, separated by a tag byte prefixed to every key, so one family per epoch instead of thirteen keeps the column-family and SST-file counts an order of magnitude lower across a ~100-epoch horizon, and makes a bucket's existence atomic. Transactions are numbered by network order and epochs partition that order contiguously, so each bucket is a disjoint, epoch-ordered segment of every table: queries chain per-bucket scans in epoch order (cursors move into the scan bounds to compose across buckets), digest lookups probe buckets newest first through their bloom filters, and the crash-recovery replay check only consults the checkpoint's own epoch bucket. Pruning becomes one constant-time column-family drop per expired epoch, replacing the compaction-filter machinery wholesale: the per-table filter configurations, their metrics, the pruner watermark, and the time-to-sequence cut are deleted. num_epochs_to_retain_for_indexes now means exactly that. The newest N bucket epochs are kept. On-disk column-family names are the ground truth for which buckets exist; opening rediscovers them and passes them at open with tuned options sharing one block cache.
muXxer
added a commit
that referenced
this pull request
Aug 10, 2026
Following the per-epoch history buckets of #12144: the 13 history tables (transaction and event indexes) move from static column families into one shared column family per epoch, created at runtime when an epoch's first checkpoint is indexed. Within a bucket the tables are TaggedDBMaps, separated by a tag byte prefixed to every key, so one family per epoch instead of thirteen keeps the column-family and SST-file counts an order of magnitude lower across a ~100-epoch horizon, and makes a bucket's existence atomic. Transactions are numbered by network order and epochs partition that order contiguously, so each bucket is a disjoint, epoch-ordered segment of every table: queries chain per-bucket scans in epoch order (cursors move into the scan bounds to compose across buckets), digest lookups probe buckets newest first through their bloom filters, and the crash-recovery replay check only consults the checkpoint's own epoch bucket. Pruning becomes one constant-time column-family drop per expired epoch, replacing the compaction-filter machinery wholesale: the per-table filter configurations, their metrics, the pruner watermark, and the time-to-sequence cut are deleted. num_epochs_to_retain_for_indexes now means exactly that. The newest N bucket epochs are kept. On-disk column-family names are the ground truth for which buckets exist; opening rediscovers them and passes them at open with tuned options sharing one block cache.
muXxer
added a commit
that referenced
this pull request
Aug 11, 2026
Following the per-epoch history buckets of #12144: the 13 history tables (transaction and event indexes) move from static column families into one shared column family per epoch, created at runtime when an epoch's first checkpoint is indexed. Within a bucket the tables are TaggedDBMaps, separated by a tag byte prefixed to every key, so one family per epoch instead of thirteen keeps the column-family and SST-file counts an order of magnitude lower across a ~100-epoch horizon, and makes a bucket's existence atomic. Transactions are numbered by network order and epochs partition that order contiguously, so each bucket is a disjoint, epoch-ordered segment of every table: queries chain per-bucket scans in epoch order (cursors move into the scan bounds to compose across buckets), digest lookups probe buckets newest first through their bloom filters, and the crash-recovery replay check only consults the checkpoint's own epoch bucket. Pruning becomes one constant-time column-family drop per expired epoch, replacing the compaction-filter machinery wholesale: the per-table filter configurations, their metrics, the pruner watermark, and the time-to-sequence cut are deleted. num_epochs_to_retain_for_indexes now means exactly that. The newest N bucket epochs are kept. On-disk column-family names are the ground truth for which buckets exist; opening rediscovers them and passes them at open with tuned options sharing one block cache.
muXxer
force-pushed
the
feat/historic-object-store
branch
from
August 17, 2026 15:55
2514140 to
5284b44
Compare
Execution-driven pruning (#12186) blocked the checkpoint executor while pruning had fallen more than an hour of chain time behind. On upgrade, many real nodes start with a larger backlog than that (downtime, retention changes, catch-up sync, an old ticker-based pruner that fell behind), and since the pruner only publishes progress after a full drain, such nodes would stall execution for the whole first drain instead of being throttled. Pruning is now best-effort background work that never gates execution; a backlog grows the database temporarily and is surfaced instead of prevented: - `pruning_chain_time_lag_ms` gauge: chain time between the executed watermark and the target of the pruner's last completed drain, plus a rate-limited warning above the old one-hour threshold. - `last_pruned_checkpoint_timestamp_ms` / `last_pruned_effects_checkpoint_timestamp_ms` gauges, published per pruned batch so dashboards show progress during long drains. The nudge channel and the drain loop are unchanged.
…pruning Every node now relocates superseded object versions and pruned checkpoint data into historic epoch buckets — there is no enable flag, no fullnode-only gating, and no alternative pruning implementation: - Config collapses to a single `historic-epochs-to-retain` knob (default 2, roughly today's default disk profile; RPC operators raise it). `num-epochs-to-retain`, `enable-compaction-filter` and the `historic-store` section are gone; unknown keys in existing config files are ignored. - The compaction-filter pruning mode (`ObjectsCompactionFilter`, `AuthorityPrunerTables`, the `pruner` database) and the range-delete branch are deleted; relocation is the only implementation. - The objects walker drains to the executed watermark with no retention window: relocation is not deletion, so nothing needs protecting. For data written by this version it finds nothing (commit-time relocation already moved it) — its work is pre-existing databases and capture misses. - `HistoricStore` is constructed unconditionally (node, tools, test builder), `Option<Arc<HistoricStore>>` becomes `Arc<HistoricStore>` throughout, and superseded pre-images are always captured in transaction outputs. The whole existing test suite now runs with relocation active. - DB-checkpoint upload compacts only (nothing left to prune in a snapshot of a continuously pruned source).
Nodes upgrading to commit-time relocation start with superseded versions already in the live objects table. A crash-resumable migration drains them once and remembers completion: - A singleton `historic_migration` row in the perpetual store tracks progress (sweeping -> sweep complete -> complete). Databases created at genesis by this version start out complete; only upgraded databases migrate. - The checkpoint walker (the relocation backstop) is the migration core: it drains backlog from the objects watermark and now reports how many superseded versions it actually found. The migration is marked complete once a drain reaches the executed watermark having found nothing, with the legacy sweep finished. - The legacy sweep handles rows the walker can never reach — databases whose checkpoint data below the objects watermark was already pruned before the upgrade. It iterates the live table in bounded slices (cursor persisted atomically with each slice's moves), relocating non-heads into the current epoch's bucket and recording tombstone heads in its expiry list. It keeps slicing even without execution progress and retires permanently once done. - Relocation into a bucket already past the retention horizon deletes outright instead of copying (equivalent to relocating and immediately dropping the bucket), so catching up through deep backlog does not write data just to drop it. - After completion, any superseded version the walker still finds is a commit-time capture miss: `historic_capture_miss_total` is incremented and `debug_fatal!` fires. One quiet release in the wild proves capture exhaustive, after which the walker, the sweep, and the marker can be deleted. `historic_migration_state` exposes progress to operators.
Commit-time capture built `superseded` from `input_objects` only, so mutations of runtime-loaded objects (dynamic fields) were never captured — the walker backstop silently relocated every one of them, which would have kept the capture-miss alarm firing forever. Found by that alarm in the e2e suite: the randomness state update mutates a child of the randomness object and panicked `test_validator_tx_finalizer_fastpath_tx`. `commit_transaction` now completes the capture for any superseded version missing from the carried pre-images by reading it back through the object cache — the transaction just read those objects, so the lookups are memory-hot, and a version already relocated is absent from the live view and needs no move. The walker also logs each version it still finds, to make future alarm firings diagnosable.
…ead API With relocation at commit time, superseded versions leave the live objects table the moment their checkpoint commits, so the JSON-RPC response assembly for freshly executed transactions (balance and object changes read input pre-images by exact version) failed deterministically with "could not find the referenced object". `read_object_at_version` now falls back to the historic buckets after a live miss — a read-API path only, unreachable from consensus and execution. This also makes `iota_tryGetPastObject` serve versions within the historic retention window.
…xpectations The from-local-history epoch_info rebuild assembles old epoch-boundary checkpoints, whose transaction, effects, events, and output objects have usually been relocated into the historic buckets by the time a rebuild runs (objects at commit time, the rest by the checkpoint pruner). Every read in the assembly now falls back to the buckets after a live miss, so `Missing` again only means the data aged past historic retention. `object_pruning_test` asserted delete-mode semantics; under relocation the live table keeps each lineage's tombstone head (including a stale `Wrapped` tombstone below a resurrected lineage, removed only by its bucket's expiry) while everything below moved to the buckets.
… fallback
A catching-up devnet node crashed assembling `CheckpointData` for
checkpoint 0: the clock object's genesis version had already been
superseded by consensus prologues of later checkpoints and relocated
out of the live table before the lagging checkpoint-data stage read it
as a genesis output.
Sweep of every remaining exact-version read that serves responses:
- `load_checkpoint_data` output objects (the crash): store read now
falls back to the historic buckets when the buffered outputs are
gone (replay after restart, or a stage lagging behind later commits).
- `AuthorityState::get_transaction_{input,output}_objects` (fullnode
execute-transaction responses, validator gRPC responses): a
transaction's own commit relocates the versions it superseded, so
responses assembled after the commit read through the buckets.
- The local transaction-KV serving reads (`get_object`,
`multi_get_objects`).
Consensus and execution paths keep no fallback: a miss there stays a
loud bug. The remaining exact-version readers are pre-commit index
post-processing (inputs still live by construction), validator
object-info requests (past versions unavailable there today too), and
the best-effort forensic dump (tolerates missing rows).
muXxer
force-pushed
the
feat/historic-object-store
branch
from
August 17, 2026 15:55
5284b44 to
34fc5b8
Compare
github-actions
Bot
added
the
documentation
Contributor
Author
|
super-seeded by #12695, which uses the new |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of change
Splits node storage into a hot live set and per-epoch historic buckets, making pruning a constant-time bucket drop instead of per-key deletes plus compaction churn, and letting RPC fullnodes serve long history (target: ~100 epochs) without the hot tables paying for it.
When the
historic-storepruning config is enabled (fullnode-only, off by default), the pruner relocates instead of deletes:objectstable into per-epoch column families of a new always-openhistoryRocksDB, bucketed by supersession epoch. The live table becomes heads-only (includingDeleted/Wrappedtombstone heads, which every latest-version read depends on); tombstone heads are recorded in a per-bucket expiry list and point-deleted from the live table only when the bucket expires.Expiring an epoch of history is a
drop_cfper family — RocksDB unlinks the column-family SSTs outright, with no tombstones and no compaction pass. Crash consistency follows one contract everywhere: write history → flush (durability barrier) → delete source rows + advance watermark in one atomic batch; replay from the watermark is idempotent.Reads: consensus and execution paths never touch history — a live-table miss there remains a loud failure. Old data is served through fallbacks that trigger only after a live miss: gRPC exact-version object lookups (
GrpcReadStore), and gRPC + state-sync reads of old transactions/effects/events/checkpoints (RocksDbStore), with the advertisedlowest_available_checkpointextending back to the earliest relocated checkpoint. DB checkpoints include the history DB (snapshotted last — source-first ordering closes the torn-window between the two stores), so restored nodes keep serving their history; formal snapshots intentionally remain live-set-only.Supporting changes: typed-store gains runtime
Database::create_cf(withcf_nameskept in sync forflush_all); the compaction-filter pruner is mutually exclusive with the historic store (enforced before the perpetual DB opens); nodes that previously ran with objects pruning disabled get a one-time watermark fast-forward past already-pruned checkpoint data.Layout rationale: one DB with per-epoch column families (rather than one DB per epoch) shares a single block cache and file-descriptor budget, and RocksDB's native per-SST bloom filters answer the epoch-routing problem — exact-key lookups probe buckets newest-to-oldest, where a miss in a sealed, compacted bucket costs ~µs from in-memory filters.
Follow-ups tracked in
plans/on this branch: e2e simtests with the feature enabled and the benchmark matrix (which gate enabling this on production fullnodes), plus a behavior-preservingPrunerContextrefactor of the pruner's parameter threading.Links to any relevant issues
How the change has been tested
Details:
cargo ci-clippy(workspace, all targets,-D warnings),cargo +nightly fmt, andIOTA_SKIP_SIMTESTS=1 cargo nextest run -p typed-store -p iota-core --lib(751 passed) run locally. ~25 new tests cover: heads-only invariant with tombstones as live heads, supersession-epoch bucketing, crash-replay idempotence for both object and checkpoint relocation, retention expiry with unwrap-resurrected lineages, live-object-set invariance under relocation and expiry, legacy V1-row migration at read, bucket rediscovery/seal state across restarts, availability-watermark tracking, DB-checkpoint copies serving all families, and the source-first snapshot-ordering torn-window test. Not yet run: e2e simtests with the feature enabled and performance benchmarks (seeplans/historic-store-testing.md); the feature is off by default, so existing e2e coverage exercises the unchanged path.Release Notes
historic-storepruning mode for RPC fullnodes: the pruner relocates superseded object versions and checkpoint-keyed history into per-epoch buckets instead of deleting them, keeping the hot tables small while serving configurable epochs of history; expired epochs are dropped in constant time. Off by default; validators are unaffected (the setting is ignored with a warning). Incompatible withenable-compaction-filter. DB checkpoints of nodes with the feature enabled include the history DB.GetObjectsand transaction/effects/events/checkpoint lookups keep working for relocated (pruned) data, and the advertised lowest available checkpoint extends back to the earliest retained history instead of the pruning watermark.