Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/data-infrastructure/indexers.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,17 @@ Because indexers listen to the *stream of data* from the blockchain and the data
Another example that highlights the need for a "wide query" is when you use a seed phrase to recover one or more accounts. Since a seed phrase essentially represents a signing key pair, the recovery is for all accounts that share the associated public key. Therefore, when a seed phrase is used to recover an account via [NEAR Wallet](https://wallet.near.org), the query requires that all accounts with a matching public key are found and recovered. Utilizing [Near Lake Framework](https://github.com/near/near-lake-framework-rs) can be used to store this data in a permanent database and this allows [NEAR Wallet](https://wallet.near.org) to perform such "wide queries". This is impossible to achieve using JSON-RPC only.

---

## Indexers in the NEAR ecosystem

There are [multiple indexing options](../tools/data-services.md) available in the NEAR ecosystem. If you are ready to host your own indexer, we recommend using the [Near Lake Framework](./lake-framework/near-lake) as it is simple, reliable, and available in multiple languages (JavaScript, Rust, Python).

If speed is critically important for your indexing needs, consider using the [Near Indexer](./near-indexer.md). However, please note that maintaining it can be more complex and costly, as it essentially operates as an independent node in the network.

If you prefer not to host your own solution, you can utilize [partner services](../tools/data-services.md).

---

## Summary

We hope this article gives you an understanding of the Indexer concept. Also, we hope now you can easily decide whether you need an indexer for your application.
Expand Down
40 changes: 27 additions & 13 deletions docs/data-infrastructure/tutorials/near-indexer.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,17 +128,31 @@ After we finish initializing the indexer, and configuring it, we can start it by
```
</TabItem>
</Tabs>

<details>
<summary>How it works</summary>

- The command initializes indexer configuration: home directory, sync mode, streaming mode, finality, etc.
- Creates a Tokio runtime on a dedicated thread
- Creates an instance of the Indexer using the provided configuration, starts it, and streams blocks to our handler. Within the handler (`listen_blocks` method), there is an infinite loop that parses block data for each new block received.

<Github fname="main.rs" language="rust"
url="https://github.com/near/nearcore/blob/master/tools/indexer/example/src/main.rs"
start="271" end="289" />

</details>

#### Run into an Error?
- If your indexer cannot find `boot nodes`, check the [boot nodes](#boot-nodes) section
---

## Parsing the Block Data

From the block data, we can access the transactions, their receipts and actions. See the code below for an example of how to parse the block data:
Within `listen_blocks` method, we can parse the block data as we get it from the stream. From the block data, we can access the transactions, their receipts and actions. See the code below for an example of how to parse the block data:

<Github fname="main.rs" language="rust"
url="https://github.com/near/nearcore/blob/master/tools/indexer/example/src/main.rs"
start="13" end="243" />
start="10" end="254" />

---

Expand Down Expand Up @@ -183,8 +197,8 @@ You can choose Indexer Framework sync mode by setting what to stream:
- BlockHeight(u64) - Specific block height to start syncing from.

<Github fname="main.rs" language="rust"
url="https://github.com/near-examples/near-indexer/blob/main/src/main.rs"
start="34" end="34" />
url="https://github.com/near/nearcore/blob/master/tools/indexer/example/src/main.rs"
start="274" end="274" />

<hr class="subsection" />

Expand All @@ -195,8 +209,8 @@ You can choose Indexer Framework streaming mode by setting what to stream:
- WaitForFullSync - Don't stream until the node is fully synced

<Github fname="main.rs" language="rust"
url="https://github.com/near-examples/near-indexer/blob/main/src/main.rs"
start="35" end="35" />
url="https://github.com/near/nearcore/blob/master/tools/indexer/example/src/main.rs"
start="275" end="275" />

<hr class="subsection" />

Expand All @@ -208,8 +222,8 @@ You can choose finality level at which blocks are streamed:
- Final - `final`, the block is final and irreversible.

<Github fname="main.rs" language="rust"
url="https://github.com/near-examples/near-indexer/blob/main/src/main.rs"
start="36" end="36" />
url="https://github.com/near/nearcore/blob/master/tools/indexer/example/src/main.rs"
start="276" end="276" />

<hr class="subsection" />

Expand Down Expand Up @@ -285,11 +299,11 @@ You can also use NEAR Indexer Framework as a dependency in your own Rust project

```toml
[dependencies]
near-indexer = { git = "https://github.com/near/nearcore", tag = "2.8.0" }
near-indexer-primitives = { git = "https://github.com/near/nearcore", tag = "2.8.0" }
near-config-utils = { git = "https://github.com/near/nearcore", tag = "2.8.0" }
near-o11y = { git = "https://github.com/near/nearcore", tag = "2.8.0" }
near-primitives = { git = "https://github.com/near/nearcore", tag = "2.8.0" }
near-indexer = { git = "https://github.com/near/nearcore", tag = "2.9.1" }
near-indexer-primitives = { git = "https://github.com/near/nearcore", tag = "2.9.1" }
near-config-utils = { git = "https://github.com/near/nearcore", tag = "2.9.1" }
near-o11y = { git = "https://github.com/near/nearcore", tag = "2.9.1" }
near-primitives = { git = "https://github.com/near/nearcore", tag = "2.9.1" }
```

<MovingForwardSupportSection />