Skip to content

feat(core,node,grpc-server): retention per data purpose - #12693

Draft
muXxer wants to merge 11 commits into
feat/cleanup-indexes-config-valuesfrom
feat/retention-per-data-purpose
Draft

feat(core,node,grpc-server): retention per data purpose#12693
muXxer wants to merge 11 commits into
feat/cleanup-indexes-config-valuesfrom
feat/retention-per-data-purpose

Conversation

@muXxer

@muXxer muXxer commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Description of change

Prunes each retention-relevant table with the knob that matches what the data is for, and moves the one table that was in the wrong place.

A finality answer could expire before the transaction it describes. The digest → checkpoint mapping lived in the RPC index history, so it was governed by num-epochs-to-retain-for-indexes. A node with a shorter index window than checkpoint window would still hold a transaction in the ledger, still return it by digest, and yet answer "no checkpoint" for it — the index bucket carrying that answer having been dropped first.

The answer now comes from executed_transactions_to_checkpoint in the perpetual store, which prunes in the same batch as transactions and effects. What follows from that:

  • The history row loses its checkpoint half: digest → (sequence, checkpoint) becomes digest → sequence. It keeps the sequence number because JSON-RPC queries read it to place a cursor, and checkpoint ingest reads it to tell a replayed checkpoint from a new one.
  • get_transaction_info leaves RpcIndexesStore and the GrpcIndexes trait; GrpcStateReader gains get_transaction_checkpoint.
  • gRPC GetTransaction no longer needs an index at all to answer include_checkpoint / include_timestamp — it reads the ledger through the checkpoint cache.
  • The row is tagged 1, the tag its predecessor used in the JSON-RPC-only store, rather than a fresh 13. The unified store is new in feat(iota-core,iota-snapshot,iota-tool): unified rpc indexes store #12645 and unshipped, so no bucket on disk carries the old meaning of that tag, and the two rows are the same type.

The rest is the model written down. Every retention-relevant table in AuthorityPerpetualTables now carries a doc comment naming the knob that owns it and the reason — objects to its own knob because non-latest versions serve only historical API reads and peers never need them to sync; transactions, effects and executed_effects to the ledger because they are the ExecutionData peers sync; events_2 to the ledger despite being API-only, because the event indexes are rebuilt from it and have no other source.

Two ordering constraints are documented where they bite rather than left to be rediscovered:

  • indexes ≤ ledger — an index window longer than the checkpoint window lets iotax_queryTransactionBlocks() return digests whose transactions are gone.
  • objects ≤ ledger — both pruners share one walker that reads checkpoint contents and effects, so a shorter checkpoint window makes the objects pruner fail on a checkpoint whose rows were already deleted. Carries a TODO pointing at the historic-table split that removes the dependency.

The operator documentation gains the counterpart: an index window shorter than the checkpoint window is fine and costs only query reach, since every retained transaction stays fetchable by digest and still reports its confirming checkpoint.

Stacked on #12652.

Links to any relevant issues

How the change has been tested

  • Basic tests (linting, compilation, formatting, unit/integration tests)
  • Patch-specific tests (correctness, functionality coverage)
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked that new and existing unit tests pass locally with my changes

New e2e test transaction_checkpoint_survives_a_shorter_index_window drives the case the change exists for: with the index window shorter than the checkpoint window, a transaction whose index bucket has been dropped is still fetched by digest over gRPC and still reports its confirming checkpoint and timestamp.

The index-store unit tests lose the assertions that read the checkpoint half of the digest row, and iota-tool's digests_search_decodes_what_the_index_wrote covers the narrowed row through the inspection tooling.

Release Notes

  • Protocol:
  • Nodes (Validators and Full nodes): num-epochs-to-retain-for-indexes may now be set shorter than num-epochs-to-retain-for-checkpoints; the only effect is that query-style endpoints stop reaching further back, while every transaction the checkpoint window retains stays fetchable by digest and still reports the checkpoint that confirmed it.
  • Indexer:
  • JSON-RPC:
  • GraphQL:
  • CLI:
  • Rust SDK:
  • gRPC: GetTransaction answers include_checkpoint and include_timestamp from the ledger instead of the RPC index, so it reports the confirming checkpoint for any transaction the node still holds, whatever the index retention, and no longer fails when the index is unavailable.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 18, 2026
@iota-ci iota-ci added core-protocol node Issues related to the Core Node team labels Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core-protocol documentation Improvements or additions to documentation node Issues related to the Core Node team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants