feat(core,node,grpc-server): retention per data purpose - #12693
Draft
muXxer wants to merge 11 commits into
Draft
Conversation
- The index module doc described the digest history as gRPC's source for a transaction's checkpoint; it holds the sequence number and gRPC reads the ledger. - "Only the APIs read this" sat next to a pointer at the same-named epoch table, which is also read by the transaction orchestrator; move the pointer to the end so it cannot be read as a claim about that table. - Fetching the effects has never depended on the checkpoint or timestamp being requested.
This was referenced Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of change
Prunes each retention-relevant table with the knob that matches what the data is for, and moves the one table that was in the wrong place.
A finality answer could expire before the transaction it describes. The digest → checkpoint mapping lived in the RPC index history, so it was governed by
num-epochs-to-retain-for-indexes. A node with a shorter index window than checkpoint window would still hold a transaction in the ledger, still return it by digest, and yet answer "no checkpoint" for it — the index bucket carrying that answer having been dropped first.The answer now comes from
executed_transactions_to_checkpointin the perpetual store, which prunes in the same batch astransactionsandeffects. What follows from that:(sequence, checkpoint)becomes digest →sequence. It keeps the sequence number because JSON-RPC queries read it to place a cursor, and checkpoint ingest reads it to tell a replayed checkpoint from a new one.get_transaction_infoleavesRpcIndexesStoreand theGrpcIndexestrait;GrpcStateReadergainsget_transaction_checkpoint.GetTransactionno longer needs an index at all to answerinclude_checkpoint/include_timestamp— it reads the ledger through the checkpoint cache.The rest is the model written down. Every retention-relevant table in
AuthorityPerpetualTablesnow carries a doc comment naming the knob that owns it and the reason —objectsto its own knob because non-latest versions serve only historical API reads and peers never need them to sync;transactions,effectsandexecuted_effectsto the ledger because they are theExecutionDatapeers sync;events_2to the ledger despite being API-only, because the event indexes are rebuilt from it and have no other source.Two ordering constraints are documented where they bite rather than left to be rediscovered:
iotax_queryTransactionBlocks()return digests whose transactions are gone.TODOpointing at the historic-table split that removes the dependency.The operator documentation gains the counterpart: an index window shorter than the checkpoint window is fine and costs only query reach, since every retained transaction stays fetchable by digest and still reports its confirming checkpoint.
Stacked on #12652.
Links to any relevant issues
How the change has been tested
New e2e test
transaction_checkpoint_survives_a_shorter_index_windowdrives the case the change exists for: with the index window shorter than the checkpoint window, a transaction whose index bucket has been dropped is still fetched by digest over gRPC and still reports its confirming checkpoint and timestamp.The index-store unit tests lose the assertions that read the checkpoint half of the digest row, and
iota-tool'sdigests_search_decodes_what_the_index_wrotecovers the narrowed row through the inspection tooling.Release Notes
num-epochs-to-retain-for-indexesmay now be set shorter thannum-epochs-to-retain-for-checkpoints; the only effect is that query-style endpoints stop reaching further back, while every transaction the checkpoint window retains stays fetchable by digest and still reports the checkpoint that confirmed it.GetTransactionanswersinclude_checkpointandinclude_timestampfrom the ledger instead of the RPC index, so it reports the confirming checkpoint for any transaction the node still holds, whatever the index retention, and no longer fails when the index is unavailable.