feat(iota-core): live/historic storage split with per-epoch history buckets - #12695
Draft
muXxer wants to merge 14 commits into
Draft
feat(iota-core): live/historic storage split with per-epoch history buckets#12695muXxer wants to merge 14 commits into
muXxer wants to merge 14 commits into
Conversation
…me_objects from disk
Revert the FromIterator impl added to ObjectSet for a test's sake; it already had get/insert/default, which is what the brief said to use. Also add a test that pins input_objects winning over read_objects when a key is present in both.
Open the per-epoch historic object buckets alongside the perpetual store, and move the versions a checkpoint's transactions superseded out of the live objects table and into the epoch's bucket in the batch that commits those transactions, so a crash can leave a version in one table or the other, never in neither. The new epoch's bucket is created during reconfiguration rather than on the commit path: EpochBuckets::ensure creates a column family on the calling thread, which would otherwise land on the first checkpoint commit of every epoch. The DBMapUtils open dropped options for column families that are not fields of the struct, so the buckets would have been reopened with rocksdb defaults; it now opens them with the options the caller passes.
…toric buckets
Superseded versions leave the live objects table the moment their checkpoint
commits, so every exact-version read that assembles a response has to reach
the historic buckets after a live miss:
- read_object_at_version, behind iota_tryGetPastObject and the JSON-RPC
balance- and object-change assembly, which read input pre-images by exact
version;
- get_transaction_{input,output}_objects, behind fullnode
execute-transaction responses and validator gRPC responses;
- load_checkpoint's output objects, which a stage lagging behind later
commits reads after those commits relocated them - the genesis clock output
of an early checkpoint on a catching-up node, for instance;
- the local transaction key-value store's serving reads.
Consensus and execution keep no fallback: they read current versions, which
never leave the live table, so a miss there stays a bug.
…the buckets GrpcReadStore delegated try_get_object_by_key to RocksDbStore, which reads the execution cache alone, so every gRPC response that reads a past version by exact key failed once relocation moved it: a transaction's input pre-images, a checkpoint's transaction objects, an explicitly requested past version. It now reads through the historic buckets after a live miss. RocksDbStore keeps no fallback, because state sync holds it too and a miss there is a bug rather than a relocated version. The new unit test pins the readers the checkpoint-data path cannot reach: its pre-images are genesis objects, written straight to the store, so no cached copy can answer in the buckets' place.
…options Each bucket's column-family options were built from a fresh default_db_options(), which installs a block-based table factory carrying a newly allocated 128 MiB LRU cache. Options were built once per bucket, so a node's block-cache ceiling grew by that much for every bucket on disk, and buckets found at open shared nothing with buckets created afterwards. The options are now built once from the perpetual database's own DBOptions and cloned per column family, so every bucket shares the block cache of the tables it lives beside and inherits the write-throughput tuning and the AuthorityPerpetualTablesOptions override. The retention-floor column family joins the same list instead of being created with options of its own. The options a database was opened with cannot be read back from a second cache, so the buckets are opened where the options exist: AuthorityPerpetualTables::open_with_historic_objects returns both, and the AuthorityStore constructors take the buckets instead of opening them.
…issed A modified version whose pre-image is in neither the transaction's inputs nor the objects it read was only counted. The count cannot lose data — the same set drives the bucket insert and the live delete, so the version stays where it was — but relocation silently stops for that shape of object, and nothing in the test gate notices. debug_fatal! crashes under msim and in debug builds, so the e2e simtest suite now fails on a miss, and release builds keep the counter and the system_invariant_violations metric. The check sits at the call site rather than in build_superseded_counting, which a unit test calls with a deliberately absent pre-image.
…t comments hist_obj_earliest_retained shared the bucket prefix hist_obj_e, so telling a bucket from a non-bucket rested on "arliest_retained" failing to parse as an epoch. hist_obj_retention cannot be read as a bucket whatever follows it. ExecutedTransaction, the alias for what execute_transaction returns, is renamed to TransactionExecutionResult: the name already belongs to iota_types::full_checkpoint_content::ExecutedTransaction, which means a transaction with its effects, events and unchanged loaded runtime objects. The comment on the superseded deletes justified their placement with an earlier transaction's inserts, which precede the whole call and settle nothing about order within it. It now gives the reasons that hold: a transaction never supersedes a version it writes, and build_db_batch appends transactions in checkpoint order. The relocation unit test claimed to prove crash atomicity, which it cannot - it observes the post-state. Atomicity comes from both halves joining one DBBatch, which refuses a map from another database. Also: epoch_buckets no longer describes itself as an RPC-index-only module now that the historic objects use it; the explanation of how pre-images are sourced moves from the #[cfg(test)] helper onto the production function; the "observe them in neither table" double negative is replaced by the claim it was meant to make; and the fallback doc now says why a miss on get_object_received_at_version is the defined answer rather than a bug.
db-tool dump asserted that a table name of the validator store is a declared perpetual table, so it panicked on every historic bucket - hist_obj_e5 on a real node database, and now also hist_obj_retention, which the perpetual open creates on every database. The assert is now an error, and the historic column families get a reader of their own: they are not fields of AuthorityPerpetualTables, so the dump derived from that struct cannot reach them, and only historic_objects.rs knows the bucket prefix and the tag its rows carry. The comments on the bucket options claimed every column family of the database shares one block cache. It is the base options' cache that the buckets share, together with the column families that take those options unchanged; objects, live_owned_object_markers, transactions and effects install caches of their own.
12 tasks
muXxer
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of change
Stops deleting superseded object versions and relocates them instead, so an exact-version read of a recent version keeps working.
Today a superseded version is deleted once it ages past
num-epochs-to-retain, and any exact-version read of it fails. This PR moves it: at checkpoint commit, the versions a transaction superseded leave the liveobjectstable for the current epoch's history bucket — a column family of the same perpetual database — in the sameWriteBatchas the commit. Afterwardsobjectsholds current versions and tombstone heads only, and no reader can see a version in neither table.The buckets are the
EpochBucketsmachinery the RPC index store already uses, pointed at the perpetual database. One column family per epoch, holding its tables by tag, so pruning an epoch is one column-family drop.This PR only ever writes buckets — nothing here drops one. That split is deliberate: dropping a bucket has a live-table side effect (its tombstone heads must be point-deleted from
objectsfirst, or they are stranded silently and permanently), and folding that in would make the first and least familiar review carry the hardest part.The consequence is that a node running this build alone grows its object footprint without bound, so it is the first of three stacked PRs that must be deployed together. No network should run a build that relocates without pruning. The two follow-ups:
unchanged_loaded_runtime_objects.prunerdatabase; and removing theobjects ≤ ledgerTODOinauthority_store_pruner.rs, whose dependency this split is what finally breaks.Where the superseded set comes from
From
effects.modified_at_versions(), with each pre-image taken from the transaction's input objects and, failing that, from what execution actually read. The second source is what makes dynamic-field mutations work: a child object mutated at runtime is never an input, and building the set from inputs alone silently drops it.Because a missed pre-image suppresses both the bucket insert and the live delete, a miss cannot lose data — the version simply stays live. That also makes it invisible, so a miss now raises
debug_fatal!: it crashes undermsimand debug builds and incrementssystem_invariant_violationsin release. The e2e suite is therefore what asserts that no miss occurs.Read paths
Every exact-version read that serves a response falls back to the buckets after a live miss: the JSON-RPC
read_object_at_version(which is what makesiota_tryGetPastObjectserve versions inside the window), the epoch-info rebuild,load_checkpoint_data's output objects,get_transaction_input_objects/get_transaction_output_objects, the balance- and object-change response assembly, and the gRPC read store.Deliberately not everywhere. The state-sync store keeps no fallback, because there a live miss is a bug and must stay one. Consensus and execution read current versions only.
get_object_received_at_versionstays unguarded because a miss there is the defined answer for an already-consumedReceivingargument, not a failure.Known gap, closed by the follow-up that adds pruning
try_find_object_lt_or_eq_version— "the newest version at or below V" — gets no fallback here, so two JSON-RPC reads lose reach the moment this lands:iota_tryGetObjectBeforeVersion, and the withdrawn-stake branch ofiotax_getStakes/getStakesByIds. This affects every fullnode, not only ones that disable pruning: object pruning is a chain-time window, so a node atnum-epochs-to-retain: 1answers these from roughly an epoch of live superseded versions today, and relocation removes them at commit.It is left to that follow-up because the obvious repair is wrong in a way that is worse than the gap: falling back whenever the live scan returns
Noneresurrects deleted objects, since the live scan collapses "a tombstone covers this range" with "nothing is in this range" into the sameNone. Telling those apart needs the per-bucket record of tombstone heads that the pruning change introduces.Not in scope
No migration — versions already superseded on an upgraded node keep draining through today's pruner, which stays in place, so historic reads serve post-upgrade data only. No configuration change:
num-epochs-to-retainis not read here, because nothing is dropped.Note for reviewers
The first two commits are cherry-picked from #12642 for its
TrackingBackingStore, which is where the runtime-read pre-images come from. They are not part of this change and will be dropped once #12642 lands ondevelop.Stacked on #12693.
Links to any relevant issues
How the change has been tested
cargo clippy -p iota-core -p iota-types -p iota-json-rpc -p iota-node -p iota-tool --all-targets --all-features -- -D warningsclean;cargo +nightly fmtclean;cargo check --workspace --all-targetsclean;IOTA_SKIP_SIMTESTS=1 cargo nextest run -p iota-core --lib848 passed / 2 skipped;-p iota-tool3 passed;cargo simtest -p iota-e2e-tests293 passed / 22 skipped, run with the capture-missdebug_fatal!live so it also asserts no pre-image was missed anywhere in the suite.New tests: a superseded version is readable from its bucket after the commit that superseded it; the reopen path finds buckets written by a previous process; a dynamic-field mutation has its child pre-image captured; response assembly for a freshly executed transaction returns balance and object changes;
db-tool dumpreads a bucket and the retention floor.object_pruning_testis adapted rather than deleted: it now asserts both halves of relocation — absent from the live table and present in the bucket — so it fails both if relocation degrades back to deletion and if it never happens.Release Notes
iota-tool db-tool dumpgains thehist_obj_*families.iota_tryGetPastObjectand the other exact-version reads serve superseded versions from the history buckets, so they keep working for versions that were previously deleted once they aged past the object retention.Breaking Changes Rollout
Affected Crates:
Required User Actions: