feat(graphql): add the paginated transactionsByDigests query - #12684
Merged
Conversation
iota-ci
added
infrastructure
tomxey
force-pushed
the
sc-platform/graphql-tx-by-digests-paginated
branch
from
August 17, 2026 11:54
6127d7d to
a570323
Compare
tomxey
changed the title
kodemartin
reviewed
Aug 18, 2026
An error occurred while trying to automatically change base from
sc-platform/graphql-fallback-docs
to
sc-platform/graphql-fix-checkpoint-paginate-doc
August 19, 2026 06:36
Add CLI/env options to configure KV client in graphql, analogous to how it is configured in iota-indexer. Constructed client is attached to the Indexer Reader, to be used for actual fallback that will be introduced in subsequent PRs. fixes #11932 Describe the tests that you ran to verify your changes. Make sure to provide instructions for the maintainer as well as any relevant configurations. - [x] Basic tests (linting, compilation, formatting, unit/integration tests) - [ ] Patch-specific tests (correctness, functionality coverage) @iotaledger/infrastructure) - [ ] Synchronization of the indexer from genesis for a network including migration objects. - [ ] Restart of indexer synchronization locally without resetting the database. - [ ] Restart of indexer synchronization on a production-like database. - [ ] Deployment of services using Docker. - [ ] Verification of API backward compatibility. - [x] GraphQL: new optional CLI flags `--fallback-kv-url`, `--fallback-kv-multi-fetch-batch-size`, `--fallback-kv-concurrent-fetches`, `--fallback-kv-cache-size` for the archival fallback. Setting `--fallback-kv-url` enables fallback to KV store for data pruned from postgres DB.
Makes the GraphQL `Query.checkpoint(id)` and `Query.checkpoints(...)` resolvers fall back to the historical fallback storage when reading checkpoints whose rows have been pruned from Postgres. - **`IndexerReader::get_stored_checkpoints_by_seqs_with_fallback`** — reads checkpoints by their sequence numbers from Postgres, falls back to historical fallback storage if needed. - **`IndexerReader::get_stored_checkpoints_by_digests_with_fallback`** — same as above but with "digest" key. ------- - **`DBReader::multi_get_checkpoints_by_seqs`** — internal helper; runs one Diesel query to load checkpoint rows for a list of sequence numbers. - **`DBReader::multi_get_checkpoints_by_digests`** — internal helper; runs one Diesel query to load checkpoint rows for a list of digests. ---------- - **`HistoricalFallbackReader::checkpoints_by_digests`** — reads a batch of checkpoints from the historical fallback storage by their digests (analogous to already existing `checkpoints` function) fixes #11934 Describe the tests that you ran to verify your changes. Make sure to provide instructions for the maintainer as well as any relevant configurations. - [x] Basic tests (linting, compilation, formatting, unit/integration tests) - [x] Patch-specific tests (correctness, functionality coverage) **Setup:** 10 checkpoints across 4 epochs (epoch 0: cps 0–2, epoch 1: cps 3–4, epoch 2: cps 5–6, epoch 3: cps 7–9). With `--epochs-to-keep 1` and `--wait-for-checkpoint-pruned 6`, cps 0–6 are pruned and cps 7–9 remain. No historical fallback configured. | # | Input | Expected | |---|---|---| | **A** | `checkpoints { ... }` (defaults) | nodes `[7, 8, 9]`, `hasPreviousPage = false`, `hasNextPage = false` — the reachable range is `[7, 9]` and fits in one page | | **B** | `checkpoints(first: 2) { ... }` | nodes `[7, 8]`, `hasPreviousPage = false`, `hasNextPage = true` — at the lower bound, more above | | **C** | `checkpoints(first: 2, after: cursor{c: 9, s: 2})` | error `DATA_PRUNED` — cursor's seq is below the pruning watermark | | **C2** | `checkpoints(first: 2, after: cursor{c: 9, s: 42})` | error `BAD_USER_INPUT` — cursor's seq is above its own `checkpoint_viewed_at` (malformed) | | **D** | `checkpoints(first: 2, after: cursor{c: 9, s: 7})` | nodes `[8, 9]`, `hasPreviousPage = true`, `hasNextPage = false` — cursor at the lower boundary is in-range | | **E** | `checkpoint(id: { sequenceNumber: 0 })` | `null` — pruned, no fallback to consult | | **F** | `checkpoint(id: { sequenceNumber: 7 })` | `{ sequenceNumber: 7 }` — unpruned, resolves | | **G** | `epoch(id: 0) { epochId, checkpoints { ... } }` | `epochId: 0`; `checkpoints` errors `DATA_PRUNED` — epoch 0's range is entirely below the watermark | | **H** | `epoch(id: 3) { epochId, checkpoints { ... } }` | `epochId: 3`; nodes `[7, 8, 9]`, both flags `false` — unpruned epoch paginates normally | @iotaledger/infrastructure) - [ ] Synchronization of the indexer from genesis for a network including migration objects. - [ ] Restart of indexer synchronization locally without resetting the database. - [ ] Restart of indexer synchronization on a production-like database. - [ ] Deployment of services using Docker. - [ ] Verification of API backward compatibility.
Adds historical fallback support for GraphQL transaction-by-digest lookups. The four covered paths: - `Query.transactionBlock(digest)` - `Query.transactionBlocksByDigests(digests)` - `Event.transactionBlock` - `MoveObject.previousTransactionBlock` All four go through `IndexerReader::multi_get_transactions_with_fallback`, the same method JSON-RPC already uses for transactions by digest. `multi_get_transactions` and `multi_get_transactions_with_fallback` now return `Vec<TransactionRead>` - a new enum that tags each row as `Checkpointed` or `Optimistic`, so the GraphQL loader can build the right `TransactionBlockInner` variant. `Event.transactionBlock` switched from a sequence-number lookup to a digest lookup. A seq-based lookup would not work for events extracted from fallback as we would not be able to do the seq_num->digest conversion for already pruned transactions. fixes #11935 New e2e test: `crates/iota-graphql-e2e-tests/tests/prune/transaction_block_pruning.move` covers the four affected paths. - [x] Basic tests (linting, compilation, formatting, unit/integration tests) - [x] Patch-specific tests (correctness, functionality coverage) @iotaledger/infrastructure) - [ ] Synchronization of the indexer from genesis for a network including migration objects. - [ ] Restart of indexer synchronization locally without resetting the database. - [ ] Restart of indexer synchronization on a production-like database. - [ ] Deployment of services using Docker. - [ ] Verification of API backward compatibility.
# Description of change
Adds historical fallback for GraphQL queries that list the transactions
of a single checkpoint. Covered paths:
- `Query.transactionBlocks(filter: { atCheckpoint: ... })`
- `Checkpoint.transactionBlocks`
Queries with only the `atCheckpoint` filter are served by the new
`IndexerReader::query_stored_transactions_by_checkpoint_seq_with_fallback`
method. A pruned checkpoint is served from the historical storage when
configured, and returns a `DATA_PRUNED` error otherwise (before this
change, the page came back empty). Queries with other filters keep using
the SQL path.
The fallback reader's `checkpoint_transactions` now takes the cursor
either by transaction digest (used by JSON-RPC) or by
`tx_sequence_number` (used by GraphQL, see `CheckpointTxCursor`). The
page is fetched with one extra row on each side, so
`hasPreviousPage`/`hasNextPage` can be filled.
## Links to any relevant issues
fixes #11936
## How the change has been tested
- [x] Basic tests (linting, compilation, formatting, unit/integration
tests)
- [x] Patch-specific tests (correctness, functionality coverage)
### Infrastructure QA (only required for crates that are maintained by
@iotaledger/infrastructure)
- [ ] Synchronization of the indexer from genesis for a network
including migration objects.
- [ ] Restart of indexer synchronization locally without resetting the
database.
- [ ] Restart of indexer synchronization on a production-like database.
- [ ] Deployment of services using Docker.
- [ ] Verification of API backward compatibility.
…phql (#12214) # Description of change Add possibility to read txs affected by address in graphql. This is to match the same ability in JSON RPC, and in the followup issue to enable serving such queries from the KV fallback. ## Links to any relevant issues fixes #12213 ## How the change has been tested - [x] Basic tests (linting, compilation, formatting, unit/integration tests) - [x] Patch-specific tests (correctness, functionality coverage) ### Infrastructure QA (only required for crates that are maintained by @iotaledger/infrastructure) - [ ] Synchronization of the indexer from genesis for a network including migration objects. - [ ] Restart of indexer synchronization locally without resetting the database. - [ ] Restart of indexer synchronization on a production-like database. - [ ] Deployment of services using Docker. - [ ] Verification of API backward compatibility. ### Release Notes - [x] GraphQL: Added the `affectedAddress` filter on `transactionBlocks` and the `AFFECTED` relation on `Address.transactionBlocks`. They list transactions where the address is the sender or a recipient, providing the same functionality as `FromOrToAddress` filter in the JSON-RPC.
Make GraphQL fall back to the archival store when reading events of a
single transaction.
Covered paths:
- `Query.events(filter: { transactionDigest })` — when this is the only
filter, the query is served by
`IndexerReader::query_stored_events_by_tx_digest_with_fallback`, which
reads from Postgres and falls back to the archival store when the
transaction has been pruned. Uses `Page::paginate_results`.
- `TransactionBlockEffects.events` — worked already, no change needed
Event filters (sender, emitting module, event type) disable the
fallback.
To have common code between JSON-RPC and Graphql the PR refactors some
of DBReader and HistoricalFallbackReader functions to accept only
`event_seq` cursor/range, instead of JSON-RPC or Graphql specific
cursors. (JSON-RPC use (tx_digest, event_seq) cursor, Graphql uses
(tx_seq, event_seq) ranges).
The tx part of the cursor can be stripped away after proper validation,
since we are fetching events from single transaction anyway.
Behavior change: `Query.events(filter: { transactionDigest })` on a
pruned (or unknown) transaction now errors with `DATA_PRUNED` when no
fallback is configured, instead of returning an empty connection. This
matches JSON-RPC.
fixes #11938
- New e2e test
`crates/iota-graphql-e2e-tests/tests/prune/events_by_tx_digest.move`:
`DATA_PRUNED` error on a pruned transaction, events of an unpruned
transaction, cursor window, cursor from a different transaction, digest
combined with sender.
- [x] Basic tests (linting, compilation, formatting, unit/integration
tests)
- [x] Patch-specific tests (correctness, functionality coverage)
@iotaledger/infrastructure)
- [ ] Synchronization of the indexer from genesis for a network
including migration objects.
- [ ] Restart of indexer synchronization locally without resetting the
database.
- [ ] Restart of indexer synchronization on a production-like database.
- [ ] Deployment of services using Docker.
- [ ] Verification of API backward compatibility.
- [x] GraphQL: `Query.events(filter: { transactionDigest })` now
supports fallback for pruned transactions.
# Description of change Make GraphQL fall back to the archival KV store when reading an object at a version whose row has been pruned from Postgres. Covered paths: - `Query.object(address, version)` - `Query.owner(address, rootVersion)` → `MoveObject.dynamicField` / `dynamicObjectField` — the field object's ID is derived from the parent and the field name, and looked up at the latest version `<= rootVersion` Notes: - The fallback is wired into the two object DataLoaders (batched KV multi-gets) through a new `IndexerReader::multi_get_fallback_objects`. - Both loaders (`HistoricalKey` — an exact version, `ParentVersionKey` — the largest version `<= parent_version`) resolve the requested version through `objects_version` and join it with the row content from `checkpointed_objects` / `objects_backward_history`: - an active row is served from Postgres, - a wrapped or deleted version resolves as non-existent, without consulting the fallback, - a version whose row content is outside the retention window of `objects_backward_history` is fetched from the fallback at the exact resolved version - a key with no `objects_version` entry resolves as non-existent when the table covers the whole chain history, and errors with `DATA_PRUNED` when it is incomplete - `ParentVersionKey` loader does not support fallback for versions not in `objects_version` because KV doesnt store wrapped-or-deleted objects and therefore `before_version` queries would not return correct version for this kind of query - Look-ups that cannot be served return a `DATA_PRUNED` error per key (instead of failing the whole batch) ## Links to any relevant issues fixes #11933 ## How the change has been tested - New e2e test `crates/iota-graphql-e2e-tests/tests/prune/objects_at_version.move`: pruned versions and a pruned dynamic field error with `DATA_PRUNED` (no fallback configured); current and never-existing versions resolve as before. - Extended `consistency/dynamic_fields/dof_add_reclaim_transfer.move`: fetching the deleted field wrapper at its exact deletion version resolves as non-existent. - [x] Basic tests (linting, compilation, formatting, unit/integration tests) - [x] Patch-specific tests (correctness, functionality coverage) ### Infrastructure QA (only required for crates that are maintained by @iotaledger/infrastructure) - [ ] Synchronization of the indexer from genesis for a network including migration objects. - [ ] Restart of indexer synchronization locally without resetting the database. - [ ] Restart of indexer synchronization on a production-like database. - [ ] Deployment of services using Docker. - [ ] Verification of API backward compatibility. ### Release Notes - [x] GraphQL: `Query.object(address, version)` and dynamic field look-ups at a `rootVersion` resolve pruned object versions from the archival store when the historical fallback is configured; look-ups that cannot be served return a `DATA_PRUNED` error instead of resolving as non-existent.
…12553) # Description of change Enable GraphQL fallback to the archival KV store when listing transactions that affect a given address. Covered paths: - `Query.transactionBlocks(filter: { affectedAddress })` - `Address.transactionBlocks` with the `AFFECTED` relation Notes: - The query is served by a new `IndexerReader::query_stored_transactions_by_affected_addresses_with_fallback`, extracted from the JSON-RPC `FromOrToAddress` fallback. - Converting a sequence number (used by graphql) back to a digest is not easily possible, that's why a small refactor of iota-indexer was done to use tx sequence numbers in the core functions. - The check rejecting cursors below the pruning watermark now looks at the first transaction of the page instead of the cursor row (which is never returned), this also fixes JSON-RPC handling of cursors right at the watermark. ## Links to any relevant issues fixes #11937 ## How the change has been tested - New e2e test `crates/iota-graphql-e2e-tests/tests/prune/transaction_blocks_by_affected_address.move` (no fallback configured): paginating forward from the pruned earliest history and from cursors in the pruned range errors with `DATA_PRUNED`; paginating backward serves the unpruned range; a cursor window starting exactly at the watermark resolves; the `AFFECTED` relation paginates the same way. - The KV-served paths (fast path, whole page from KV, descending top-up across the watermark, cursor cache) cannot be covered by the e2e tests and will be manually tested in followup issues. - [x] Basic tests (linting, compilation, formatting, unit/integration tests) - [x] Patch-specific tests (correctness, functionality coverage) ### Infrastructure QA (only required for crates that are maintained by @iotaledger/infrastructure) - [ ] Synchronization of the indexer from genesis for a network including migration objects. - [ ] Restart of indexer synchronization locally without resetting the database. - [ ] Restart of indexer synchronization on a production-like database. - [ ] Deployment of services using Docker. - [ ] Verification of API backward compatibility. ### Release Notes - [x] GraphQL: `Query.transactionBlocks(filter: { affectedAddress })` and `Address.transactionBlocks` with the `AFFECTED` relation resolve pruned history from the archival store when the historical fallback is configured, and report `DATA_PRUNED` otherwise.
…12571) # Description of change Add Prometheus metrics for the requests the historical fallback client sends to the archival KV store: - `historical_fallback_requests` — requests to the KV store, labeled by `resource` (`tx`, `fx`, `cs`, `cc`, `ob`, `txa`, ...). - `historical_fallback_request_errors` — requests that failed, with the same label. Counted per HTTP request, cache hits are not counted. The counters are exposed for both indexer JSON-RPC and GraphQL, next to the existing `historical_fallback_cache_*` metrics. ## Links to any relevant issues fixes #11940 ## How the change has been tested Manual smoke test against the testnet archival KV store and a pruned testnet indexer database: - Queries for pruned checkpoints and transactions by affected address increment `historical_fallback_requests` for each resource type involved. - Repeating the same query is served from the cache and does not increment the request count. - With the KV store unreachable, the failing query increments `historical_fallback_request_errors`. - [x] Basic tests (linting, compilation, formatting, unit/integration tests) - [x] Patch-specific tests (correctness, functionality coverage) ### Infrastructure QA (only required for crates that are maintained by @iotaledger/infrastructure) - [ ] Synchronization of the indexer from genesis for a network including migration objects. - [ ] Restart of indexer synchronization locally without resetting the database. - [ ] Restart of indexer synchronization on a production-like database. - [ ] Deployment of services using Docker. - [ ] Verification of API backward compatibility.
…ion IDs (#12616) # Description of change `Query.transactionBlocks(filter: { transactionIds })` now uses the fallback KV store. It gets the requested transactions with `multi_get_transactions_with_fallback`, keeps the ones that are visible at the viewed checkpoint and inside the cursor range, and returns them ordered by `tx_sequence_number`. This makes it work the same way as the parts that already use the fallback: - JSON-RPC `iota_multiGetTransactionBlocks` and `iota_getTransactionBlock` - the GraphQL `transactionBlock(digest)` query When `transactionIds` is used together with other filters, it still reads only from Postgres, same as the other fallback cases. ## Links to any relevant issues fixes #12589 ## How the change has been tested - New e2e test `crates/iota-graphql-e2e-tests/tests/prune/transaction_blocks_by_ids.move`, with no fallback set up: when a pruned digest and an unpruned digest are asked for together, only the unpruned one comes back, and paging (forward and backward) works. - [x] Basic tests (linting, compilation, formatting, unit/integration tests) - [x] Patch-specific tests (correctness, functionality coverage) ### Infrastructure QA (only required for crates that are maintained by @iotaledger/infrastructure) - [ ] Synchronization of the indexer from genesis for a network including migration objects. - [ ] Restart of indexer synchronization locally without resetting the database. - [ ] Restart of indexer synchronization on a production-like database. - [ ] Deployment of services using Docker. - [ ] Verification of API backward compatibility. ### Release Notes - [x] GraphQL: `Query.transactionBlocks(filter: { transactionIds })` can now read pruned transactions from the fallback KV store when it is configured.
…12617) # Description of change The doc comment on `Checkpoint::paginate` said that a cursor or epoch in the pruned range always returns an error. This is not true anymore: when the fallback KV store is configured, they are read from it. Updated the comment to say so. Doc comment only, no behavior change. ## Links to any relevant issues fixes #12590
# Description of change
Documents the archival (historic) fallback for `iota-graphql-rpc`:
- **`crates/iota-graphql-rpc/README.md`** — a standalone "Historic
fallback (REST KV store)" section: which queries fall back, how to
enable it (`--fallback-kv-url` and the tuning flags), and the list of
covered queries.
- **`docs/content/operator/extended-data-services/graphql.mdx`**
(operator doc) — a "Pruned data" section mirroring the JSON-RPC operator
doc, the four `--fallback-kv-*` flags added to the settings table, and
the stale dependency note ("fallback service cannot be configured for
GraphQL") replaced.
Both note the one limit: resolving a dynamic field or dynamic object
field at a parent object's version needs that version in
`objects_version`, so on snapshot-restored indexers versions below the
restore point return `DATA_PRUNED`.
Docs only, no code change.
## Links to any relevant issues
fixes #11941
`transactionBlocksByDigests` now returns a cursor-paginated connection, ordered by transaction digest and including optimistic transactions. Missing digests are absent from the result instead of returned as null.
`transactionBlocksByDigests` is restored unchanged and deprecated, so the change is non-breaking. The paginated connection is now the new `transactionsByDigests` query.
`transactionsByDigests` now returns a page with one entry per digest, in the order of the `digests` argument, null when the transaction was not found. Pagination is forward-only: `cursor` resumes after an entry and `limit` caps the page size.
tomxey
changed the base branch from
sc-platform/graphql-fallback-docs
to
infra/feat/graphql-historical-fallback
August 19, 2026 07:34
sergiupopescu199
approved these changes
Aug 19, 2026
tomxey
force-pushed
the
sc-platform/graphql-tx-by-digests-paginated
branch
from
August 19, 2026 08:04
5bb7eb4 to
4c72565
Compare
Base automatically changed from
infra/feat/graphql-historical-fallback
to
develop
August 19, 2026 11:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of change
Adds a new
transactionsByDigestsquery that returns transactions in pages. Each page holds one entry per digest, in the order of thedigestsargument, null when the transaction was not found — the same shape astransactionBlocksByDigests, just chunked. Like the old query, it includes transactions that are not checkpointed yet.Pagination is forward-only:
limitcaps the page size andcursorresumes after an entry of a previous page. The result is a plain page type (nodes,hasNextPage,endCursor), not a connection. The cursor also pins the checkpoint the first page was viewed at, so later pages keep the same view.The existing
transactionBlocksByDigestsquery is unchanged and is now deprecated, to be removed in v1.38. This keeps the change non-breaking.Links to any relevant issues
fixes #9156
Removal of the deprecated
transactionBlocksByDigestsis tracked in #12688.How the change has been tested
New graphql-e2e test
crates/iota-graphql-e2e-tests/tests/transactions/by_digests_paginated.move: input order, a null entry for a digest that is not found,limitpages, cursor resume, a full page of nulls, and an empty digest list.New e2e test
test_transactions_by_digests: the query returns transactions right after execution (before they are checkpointed), in input order, with null for a missing digest.Basic tests (linting, compilation, formatting, unit/integration tests)
Patch-specific tests (correctness, functionality coverage)
Infrastructure QA (only required for crates that are maintained by @iotaledger/infrastructure)
Release Notes
transactionsByDigestsquery that returns transactions in pages, one entry per requested digest in input order (null when not found), with forward-onlycursor/limitpagination. It includes both old transactions from fallback and fresh transactions just executed via GraphQL.transactionBlocksByDigestsis deprecated and will be removed in v1.38.