Proposal: Canton Index#319
Conversation
Proposal migrated to github Signed-off-by: Web3Slimchance <Web3Slimchance@gmail.com>
Web3Slimchance
requested review from
hythloda,
isegall-da,
stas-sbi,
tkatrichenko and
waynecollier-da
|
|
||
| ### Abstract | ||
|
|
||
| We propose Canton Index, an open-source indexing and query engine for Canton Network. It sits alongside a validator that runs PQS (the Participant Query Store), reads ledger events from PQS using developer-defined filter specifications, transforms those events into application-ready documents, and serves the results through a document-query API. Phase 2 will add Scan as a second source for network-level data, cross-validator views, and historical periods that local PQS may have pruned. |
There was a problem hiding this comment.
Minor comment that Scan only solves for historical CC data PQS would need to be set to not prune data to solve history for private transactions.
There was a problem hiding this comment.
Thank you, you are right. We deliberately kept the abstract and Phase 2 description at a high level to keep the proposal tight, but we could have been clearer: "Scan can fill in network-level and historical CC data, but cannot recover pruned private transactions." Happy to tighten the wording in the proposal if useful.
| └─────────────────────────────────────────────────────────┘ | ||
| ``` | ||
|
|
||
| **PQS Reader:** This component polls the validator's Participant Query Store using its documented SQL API: `creates(name, from, to)`, `exercises(name, from, to)`, `archives(name, from, to)`, and `active(name, at_offset)`. The PQS Reader tracks its own progress cursor by ledger offset to know what has already been processed. PQS itself handles the upstream Ledger API subscription, reconnection, backpressure, and pruning, so Canton Index inherits those guarantees rather than reimplementing them. PQS is assumed to be available on validators (the Canton Foundation's separately-funded "PQS as common good" work brings PQS into the open-source surface for all operators). |
There was a problem hiding this comment.
Architecture comment. PQS uses streaming interface from participant ledger API, so this architecture goes from streaming to DB to poll based mechanism
There was a problem hiding this comment.
This is a deliberate tradeoff for Phase 1. PQS already gives us a stable SQL interface with offset semantics, reconnection, and durability handled upstream. A direct Ledger API streaming path at chain head is a reasonable future optimization, and we'd address it in a later phase once we have real latency data from validators.
| - Tracks source offset per document for consistency | ||
| - Supports pruning by age or offset to keep tables lean | ||
|
|
||
| **Query API:** HTTP document-query API over materialized documents: |
There was a problem hiding this comment.
How are you thinking about the authentication and authorization model for access to the data?
There was a problem hiding this comment.
This is out of scope for Phase 1. Most enterprise validator operators have custom auth workflows. We would prefer understanding the concrete needs before we lock down the authentication and authorization model. For Phase 1 the Query API is assumed to run inside the operator's trust boundary, fronted by their existing auth. We'll revisit if a concrete need emerges from our design partners.
|
@Web3Slimchance, your proposal is missing a Special Interest Group (SIG) label. Adding the right SIG label ensures the relevant domain experts can find and review your proposal, Check more about SIGs. Please add one of the following labels to this PR:
Not sure which one fits? Pick the closest match to your proposal's domain. You can add a label from the right sidebar under "Labels". |
1 similar comment
|
@Web3Slimchance, your proposal is missing a Special Interest Group (SIG) label. Adding the right SIG label ensures the relevant domain experts can find and review your proposal, Check more about SIGs. Please add one of the following labels to this PR:
Not sure which one fits? Pick the closest match to your proposal's domain. You can add a label from the right sidebar under "Labels". |
|
I do not have write access to this repo. Can one of the maintainers please add the |
4 participants
Development Fund Proposal Submission
Proposal file:
/proposals/canton-index.mdSummary
Canton Index reduces the time from "I have a validator" to "I can query my data" from weeks to minutes. It gives every Canton builder a shared, open-source primitive for accessing ledger data. Instead of every team hand-rolling JSONB SQL pipelines, they can focus on actual application logic.
Canton Index sits alongside a validator running PQS, applies developer-defined filters to ledger events, materializes them into application-ready documents, and exposes them through a document-query HTTP API.
This proposal is scoped to stateless projections. The proposal outlines a roadmap with follow-on work:
Checklist
/proposals/Notes for Reviewers