Block explorer data

[yanivtal]

Block explorer data deserves first class support in The Graph. Not every node needs to index block explorer data but those that want to should be able to efficiently. Block explorer data includes indexing:

• Blocks
• Transactions
• Accounts
• Receipts

This should be possible without having to define a subgraph specifically for this data. Rather this should be specified as a CLI argument when starting up the node.

Questions:

• Should we automatically index internal transactions?
• What options should we enable for sharding? Should we skip this for a first pass?
• How do we namespace these entity types? Should there still be GraphQL types that live in a subgraph?

[leoyvens]

@tarrencev expressed the wish to call contracts from the graphql client, so that he could get the latest state of the contract without having to index it. This can be done Etherscan, seems reasonable to put it on the wishlist for our block explorer functionality.

[Jannis]

Rationale / Use Cases

I think Yaniv's original description covers this. We need this for users to be able to query blockchain-intrinsic data from Ethereum (blocks, transactions, accounts, transaction receipts). We also want subgraphs to be able to reference this kind of information from their subgraphs.

TODO: add use cases.

Requirements

• Supports at least:
  • blocks
  • transactions
  • transaction receipts
  • logs
  • accounts (with balances).
• Comes in the form of a built-in subgraph, served at /subgraphs/ethereum and /subgraphs/ethereum/graphql (GraphiQL), similar to the /subgraphs subgraph. Supports the same GraphQL API as we use for all subgraphs.
• Can be enabled with a --ethereum-subgraph command line flag for Graph Node. Disabled by default.
• All indexed Ethereum data can be referenced from other subgraphs.
• This importing approach drives subgraph index and replaces the block ingestor.

Proposed User Experience

Querying block explorer data

After started with --ethereum-subgraph or ETHEREUM_SUBGRAPH=true, the Graph Node indexes the entire chain from the genesis block to the latest block, and then follows the chain as new blocks are being added.

Users can access the data by going to http://localhost:8000/subgraphs/ethereum/graphql (GraphiQL) or by using http://localhost:8000/subgraphs/ethereum (or the WS alternative) in their apps. They can send queries like the following to this endpoint to query Ethereum data:

{
  blocks(where: { number_gte: 0, number_lt: 1000 }, orderBy: number) {
    hash
    transactions(orderBy: gas, orderDirection: desc) {
      hash
      receipt { ... }
    }
  }
  accounts(where: { address: "0x..." }) {
    balance
  }
}

Referencing block explorer data

From a user's perspective, whether data comes from one subgraph or another should not matter. Assuming a field owner: Account! refers to the Account entity from the built-in Ethereum subgraph, a query like

{
  domains {
    owner {
      balance
    }
  }
}

should just work™. This includes being able to introspect the Account type and any related types in the GraphQL playground.

From a subgraph developer's perspective, the main novelty is subgraph composition. Given a subgraph name or deployment ID, types from the subgraph with that name or deployment ID can be imported and referenced in the GraphQL schema using a new @import directive:

@import(
  from: {
    name: 'ethereum' # or id: 'Qm...'
  }
  as: 'Ethereum' # required prefix
)

type User @entity {
  account: Ethereum__Account!
}

Open Questions

• Should subgraph composition allow subgraph mappings to access instances of the imported entity types in the store?
  • Decision: no.
• An alternative to the @import directive would be an import comment syntax a la https://github.com/prisma/graphql-import
  • Decision: the @import directive is good.
• Referencing and querying entities across the subgraph boundary could lead to querying both subgraphs at different times/blocks. What do we do here?
  • Decision: To be solved later when we support time-travel queries.

Proposed Implementation

Graph Node

• Add a GraphQL schema for the built-in ethereum subgraph.
• Add an --ethereum-subgraph CLI flag to graph-node.
• Add an EthereumIngestor component to datasource/ethereum.
  • This component can be used as an alternative to the existing BlockIngestor, allowing our ingestion of Ethereum block explorer data to drive subgraph indexing in the same way.
  • The ingestor detects reorgs and marks blocks that are no longer part of the chain as uncled.
• When the --ethereum-subgraph CLI flag is provided, use EthereumIngestor instead of BlockIngestor.
• Because the EthereumIngestor will be writing blocks constantly when catching up, change ChainHeadUpdateListener to poll for the latest block in repeated intervals as long as the latest block we have is far enough behind the head of the chain.
• Add a new EthereumSubgraphResolver to resolve queries for the built-in ethereum subgraph.
• Update the GraphQL query execution to allow querying across different subgraphs, switching resolvers where necessary.
• Update the GraphQL servers to resolve the schema for the built-in ethereum subgraph.
• Add /subgraphs/ethereum routes to the GraphQL servers.
• Add @import validation to Graph CLI.

Open Questions

• When composing subgraphs, how do we make introspection work? We'd have to merge schemas basically but that may result in type conflicts.
  • Decision: use the prefix provided in the @import directive and include the whole schema using the prefix.
• Should we try piggybacking on entities for the initial pass? Or should we go straight to custom database tables? (My recommendation: try entities initially.)
  • Decision: custom tables.
• Should we go straight to custom ID fields (e.g. hash and number for blocks) or stick to id for the initial pass? (My recommendation: stick to id initially.)
  • Decision: id for the initial pass.

Proposed Documentation Updates

• Document the --ethereum-subgraph CLI flag in the graph-node README.
• Document the @import feature on https://thegraph.com/docs.
• Document the /subgraphs/ethereum endpoint and the GraphQL schema on https://thegraph.com/docs.

Proposed Tests / Acceptance Criteria

• Test the basic ingestion of all types of data using a mocked EthereumAdapter.
• Test querying the Ethereum subgraph endpoint against a store with test data.
• Test validating and building subgraphs with @import in Graph CLI.

Tasks

• Database preparation
  • Add a GraphQL schema for the built-in ethereum subgraph.
  • Define schemas for Ethereum tables.
  • Add a database migration to create the Ethereum tables.
  • Add a basic EthereumIngestor component to datasource/ethereum (no reorgs).
• Data ingestion
  • Add an --ethereum-subgraph CLI flag to graph-node.
  • Add a basic EthereumIngestor that doesn't handle reorgs yet and doesn't replace the BlockIngestor yet.
  • Allow EthereumIngestor and BlockIngestor to be used interchangeably. This may require changes to BlockStream, EthereumAdapter, BlockIngestor and ChainStore. Use EthereumIngestor instead of BlockIngestor when --ethereum-subgraph is provided.
  • Handle chain reorgs in EthereumIngestor by marking discarded blocks as uncled.
• Querying
  • Add an EthereumResolver to resolve block explorer queries.
  • Make GraphQL query execution capable of executing queries across different subgraphs, switching resolvers where necessary.
  • Load and merge schemas according to @import directives.
  • Update the GraphQL servers to resolve the schema for the built-in ethereum subgraph.
  • Add /subgraphs/ethereum routes to the GraphQL servers.
• Validation and testing
  • Add @import validation to Graph CLI, including tests.
  • Add tests for basic ingestion of all Ethereum data types using a mocked EthereumAdapter.
  • Add tests for querying block explorer data against a store with test data.
• Documentation
  • Document the --ethereum-subgraph CLI flag in the graph-node README.
  • Document the @import feature on https://thegraph.com/docs.
  • Document the /subgraphs/ethereum endpoint and the GraphQL schema on https://thegraph.com/docs.

[Zerim]

Nice job putting this together!

My input on some of the open questions:

• Should subgraph composition allow subgraph mappings to access instances of the imported entity types in the store?
  My vote would be no. We should assume that in the long term relationships across subgraph entities will be plentiful, and requiring an Indexer to have all the subgraphs indexing (or available) in order to run the mappings for a single subgraph would be untenable. For the block explorer use case specifically, I believe we already expose much of the data in the mappings anyways.
• An alternative to the @import directive would be an import comment syntax a la https://github.com/prisma/graphql-import
  I like the directive syntax better than the comment syntax option. I presume that I could supply: from: { id: Qmsdf58... } if I wanted to reference a subgraph by ID rather than by name?
• Referencing and querying entities across the subgraph boundary could lead to querying both subgraphs at different times/blocks. What do we do here?
  I think by default all subgraphs should be queried as of the same block. Currently we only support querying as of the "latest" block but we should also support querying as of a block supplied by the user in the query. Eventually, it should be possible to specify different blocks to query as of when traversing entity relationships. Not sure what the right stop gap is for right now, but I don't think it's desirable to have state queried across different blocks w/o this intention being expressed by the user.
• When composing subgraphs, how do we make introspection work? We'd have to merge schemas basically but that may result in type conflicts.
  Could we make the as field in the @import directive supply a namespace for the entire imported subgraph schema, rather than just alias a specific type?
• Should we go straight to custom ID fields (e.g. hash and number for blocks) or stick to id for the initial pass? (My recommendation: stick to id initially.)
  I'm also a fan of making id the block hash initially, and make number a normal attribute (which could return multiple blocks if we have uncles/ forked blocks in our DB).

We could make double underscore reserved in type names, so that we always have it available to prefix imported types with a namespace.

i.e.,

@import( 
  from: { name: 'ethereum' }
  as: 'Eth', # Optional renaming
)

type User @entity {
  account: Eth__Account!
}

Other feedback/questions:

1. I'm a little bit hesitant to bake Ethereum any deeper into graph-node, with a special CLI flag, endpoint, etc. But I suppose we have a bigger refactor in store, anyways when we switch to the multi-blockchain architecture.
2. Do we have an API in mind for traversing reverse relationships at query time?

{
  domains {
    owner {
      __from( name: 'ens' ) {
        ownedDomains {
          names
        }
      }
    }
  }
}

3. What should the behavior be if a user runs graph-node locally and deploys a subgraph locally which references block explorer types? Should we automatically run the block explorer subgraph as well, or simply fail gracefully at query time? We will have to answer this question in the general case when we support subgraph composition as a first-class feature.