feat(iota-core): retention per data purpose - #12690
Closed
muXxer wants to merge 64 commits into
Closed
Conversation
…nfo at query time
Following the per-epoch history buckets of #12144: the 13 history tables (transaction and event indexes) move from static column families into one shared column family per epoch, created at runtime when an epoch's first checkpoint is indexed. Within a bucket the tables are TaggedDBMaps, separated by a tag byte prefixed to every key, so one family per epoch instead of thirteen keeps the column-family and SST-file counts an order of magnitude lower across a ~100-epoch horizon, and makes a bucket's existence atomic. Transactions are numbered by network order and epochs partition that order contiguously, so each bucket is a disjoint, epoch-ordered segment of every table: queries chain per-bucket scans in epoch order (cursors move into the scan bounds to compose across buckets), digest lookups probe buckets newest first through their bloom filters, and the crash-recovery replay check only consults the checkpoint's own epoch bucket. Pruning becomes one constant-time column-family drop per expired epoch, replacing the compaction-filter machinery wholesale: the per-table filter configurations, their metrics, the pruner watermark, and the time-to-sequence cut are deleted. num_epochs_to_retain_for_indexes now means exactly that. The newest N bucket epochs are kept. On-disk column-family names are the ground truth for which buckets exist; opening rediscovers them and passes them at open with tuned options sharing one block cache.
…s and pass live objects by reference
… index only after its data is durable
…index row and resolve values deterministically
…tabases and cancel rebuilds on shutdown
…om a formal snapshot safely
… let a dropped bucket stay dropped
Adds crates/iota-core/src/rpc_indexes/grpc_api.rs: get_transaction_info,
dynamic_field_iter, get_coin_info, package_versions_iter on
RpcIndexesStore, plus impl iota_node_storage::GrpcIndexes, ported from
grpc_indexes.rs onto the unified schema. Every read errors with
StorageError::custom("the gRPC index group is not enabled") when the
store does not serve IndexGroup::Grpc.
Reuses the owner index reads Task 2 already added instead of
duplicating owner_iter/owner_bounds: the gRPC trait's OwnedObjectCursor
converts to an OwnerIndexKey at the call site (owner is supplied
separately), so no adaptation of the existing method was needed.
Extends unit_tests/rpc_indexes_tests.rs with the gRPC digest-lookup,
reopen, and pruning tests, owner/coin/package read tests, a gating
test, and the cross-API digest test.
… store Adds the write side of the unified RPC index store: one checkpoint staging and commit path, one live-object indexer, and one formal-snapshot restore, each filling the tables of whichever API groups the store maintains. - `index_checkpoint` stages one batch per checkpoint and `commit_update_for_checkpoint` writes it, skipping transactions already covered by the epoch's digest rows. - The owner and dynamic-field deltas are shared; the JSON-RPC group adds the history rows and the balance cache deltas, the gRPC group the coin metadata and package versions. - With the JSON-RPC coin table gone, a coin's balance before the checkpoint comes from the transaction's own input coins instead of a table read. - `live_scan` fills the live-state tables from a rebuild's object scan or a restore's object partitions, replacing the rebuild's no-op indexer.
…e group gating - Cover the merge of a coin type's metadata objects: one checkpoint creating all three, and a later checkpoint contributing one of them. Both cases fail against the behaviour the merged ingest replaced. - Cover the converse group gating: a JSON-RPC-only store fills the history tables and leaves the gRPC group's coin and package tables empty, in the checkpoint ingest and in the live scan. - Drop `record_written`'s unreachable prior-balance lookup: the deletion path claims the key first for every address-owned input, so the first write of a pair always means the owner did not hold the coin.
…ied rpc_indexes store Wires every caller onto `RpcIndexesStore` and deletes `jsonrpc_index.rs` and `grpc_indexes.rs`: - `AuthorityState` keeps one `rpc_indexes_store` field; the JSON-RPC reads go through it when the node maintains the `JsonRpc` group, the gRPC state reader when it maintains `Grpc`. - The checkpoint executor stages and commits one index update; the pruner makes one `prune()` call, so the digest history now follows the index retention instead of the checkpoint retention. - The node builds the store from `enable_index_processing` / `enable_grpc_api`, and removes the `indexes`, `jsonrpc_indexes` and `grpc_indexes` directories of earlier releases at startup. - `getCoins` reports the coin's own `T` again, as the wire format expects, instead of the `Coin<T>` the owner index carries. - `verify_indexes` recomputes the shared owner index; the JSON-RPC coin table it also checked is gone.
…dexes store - iota-tool download-formal-snapshot: replace --skip-grpc-indexes / --skip-jsonrpc-indexes with one --skip-rpc-indexes. A restore cannot know the node's future config, so it now always builds both index groups unless index building is skipped entirely. - typed-store: delete bulk_ingestion_options_split_between now that a restore ingests into only one index store; bulk_ingestion_options() no longer divides the memtable budget between stores. - iota-core: RpcIndexesRestorer and the rebuild path open with plain bulk_ingestion_options(); drop the now-unused concurrent_stores parameter from open_for_bulk_ingestion. - iota-tool db-tool index-search: point the pre-unification table names (owner_index, dynamic_field_index, txs_seq) at their new names instead of falling through to a generic error. - iota-tool db-tool prune-checkpoints: document why it does not also prune the RPC index history (that retention now follows its own knob, independent of checkpoint retention).
…pshot restore docs - iota-tool db-tool prune_checkpoints: the doc comment explaining why it doesn't also prune the RPC index history misstated the blocker. RpcIndexesStore::new_without_init needs only a path and IndexGroups, not a running node's worth of setup; the real blocker is that it hardcodes epochs_to_retain to None, so prune() would always no-op through it. - docs/content/operator/common/snapshots.mdx: replace the removed --skip-grpc-indexes references with --skip-rpc-indexes and its build-both-groups-unconditionally semantics.
`getOwnedObjects` now returns coins largest-balance first, so the wallet helpers that take "the first owned coin" no longer return the same coin on every call. Two tests depended on the old order: - `call_shared_object_contract` let the wallet pick gas for each of the two `assert_value` calls, and the second one asserts it depends on the first through the shared gas object. Both calls now pay with the same coin. - `execute_add_validator_transactions` took the first coin covering the joining stake, which is now the biggest one, leaving only barely-sufficient coins for the next join to spend on gas. It now picks both coins by balance instead: the smallest one that covers the stake, and the largest one to pay with.
`meta.groups` was written only when the store was seeded or rebuilt, so it could permanently claim more groups than the store actually maintains: a formal-snapshot restore stamps both groups, a node running JSON-RPC only then stops maintaining the gRPC tables, and enabling the gRPC API later passed the subset check and served an index frozen at the restore. Every open now writes the group set it maintains, once it is established that no rebuild is needed and before the store can index anything.
…s type `get_owner_objects` scanned every row the owner holds and read each one from the object store before applying the filter, so a filtered page cost one object read per scanned row. A filter that pins the object type now narrows the index scan the way `get_move_objects` and the coin reads already do; the filter is still applied per row, so the page holds the same objects. Also documents that `getCoins` paging is only stable while the owner's coins keep their balances, since the order is by balance and the cursor's position is rebuilt from the coin's live balance.
The operator pruning page still tied index pruning to `enable-index-processing`, but a node running only the gRPC API maintains the same store and grows it per epoch unless `num-epochs-to-retain-for-indexes` is set. Rewords the page for the one `rpc_indexes` store, and names the gRPC digest lookups in the node's unset-retention warning so a gRPC-only operator recognises what grows.
…ining docs Review of task 1 found leftover references to the retired enable-index-processing flag that the initial repo-wide grep missed (--include glob failed silently under zsh): - the iota-swarm-config snapshot test, which broke cargo nextest - CLI help and a code comment in iota-tool - a sample node config in iota-proxy's README - sample validator/SSFN configs under setups/validator/
…ex-processing A config file is loaded with unknown keys ignored, so a node upgraded with `enable-index-processing` left in place would drop that key and take the `enable-jsonrpc-api` default of `true` — mounting a JSON-RPC listener and blocking startup on a full index rebuild on exactly the nodes the old flag was set to turn indexing off. `NodeConfig` now captures the renamed key and `check_renamed_keys` refuses it; `IotaNode::start_async` calls that first, ahead of any expensive work. The field is never serialized, so it cannot reach a config the node writes. Also state the `enable-jsonrpc-api` default the right way round, name the config keys in the kebab-case operators actually grep for, and reject a retention of `0` before announcing the index store.
- `EpochBuckets::prune` records that `0` keeps the newest bucket exactly as `1` does, so the caller's clamp to at least 1 is visibly redundant rather than silently so. - `prune_checkpoints` no longer argues against a constructor with an `epochs_to_retain` override; that constructor now exists. - The retention block in the pruning guide follows the file's alignment.
- The index module doc described the digest history as gRPC's source for a transaction's checkpoint; it holds the sequence number and gRPC reads the ledger. - "Only the APIs read this" sat next to a pointer at the same-named epoch table, which is also read by the transaction orchestrator; move the pointer to the end so it cannot be read as a claim about that table. - Fetching the effects has never depended on the checkpoint or timestamp being requested.
github-actions
Bot
added
the
documentation
Contributor
Author
|
GitHub API was down / unreliable, so I accidentially created two PRs. This one is super-seeded by #12693 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.