[idea] Consider namespaced nodes

[Wondertan]

Celestia's blocks are sharded with namespaces in a way that you can access only a specific part of a block over the network from the full node. Similarly, some nodes can only store specific parts of the block and not all the parts for reasons like disk usage optimization. Future rollup sequencers/aggregators might be interesting in this as they won't care about other rollups besides their own. However, this requires our network to have a discovery mechanism for such nodes, so that other client nodes can discover them - "namespaced node" and access a specific part they need. This can be implemented using DHT service discovery and by implementing NamespaceAvailability which will sync only required namespaced shares. Also, there might be functionality for a full node to become a "namespaced node" by pruning "all besides needed" or "set of specific" namespaces.

[musalbas]

I'm not sure if it makes sense to implement per-namespace discovery on the main network. This is more suitable for rollup subnets, where clients can request the rollup data from nodes directly. Implementing per-block discovery is much simpler, and still allows nodes to query the blocks by namespace, once they find a peer with the block height.

In general, on the main network, we want to decouple the data layer from the application layer. This means that all data should be treated as being equal to each other. Adding preferential treatment for different application namespace would not be in line with this.

[Wondertan]

@musalbas, Imagine the two following architectures where there are Celestia and Rollup nodes are attached to each other and maintain a trusted connection over Celestia Node API/RPC for accessing data and other stuff like PayForMsg:

• (1) The Celestia Node(Light) does not store namespaced data for the Rollup Node and only proxies it from the DA network to Rollups own data/block storage. Essentially, this is what you are thinking of.

• (2) The Celestia Node(Light) stores the namespaced data for the Rollup Node. Rollup Node accesses the data from Celestia Node ad-hoc and deserializes shares into its own block format. Rollup Node doesn't store the block's data on it own.

Now, let's compare:

• (1) Forces Rollup to have its own data storage and data syncing

  • Celestia provides an efficient storage engine for shares at least for its own needs(Full nodes store block data as shares). Rollups can reuse it instead of reinventing the wheel, but if they really want to - nothing would stop them.
  • The easiest form of block sync for rollups is syncing by namespaces from DA network. “roll(ti)mint” currently doesn't have its own block sync and on the last "pilgrimage", we discussed possibly implementing the namespace approach first, as API to work with namespaces is already available. Furthermore, the approach has to be implemented for one to sync when no rollup peers to sync from are found, as DA is always available. The following question would be: why would Rollup need to implement its own data/block sync in general if it can sync from DA?

• (1) does not serve/contribute back relevant data to the network. It is a huge loss not to serve back data which Rollup nodes will store anyway for their own use. A loss not just for syncing(explained above), but also for DASing.

  • DAS without discovery. I expect namespaced nodes to be tightly connected what can reduce the choke on Full nodes which will only serve shares in current conditions. Moreover, there is no incentive to run a full node and we will see our own deployed full nodes being overloaded at some point. Rollup nodes store data for the own usage and thus need no incentivization to host it, sofor can facilitate the whole network
  • DAS with discovery. While doing DASing a Celestia Node traverses the NMT tree, and it knows min and max namespaces, which it can use to discover peers serving those. Our MVP experiments showed us that DHT re/announcing is the most time consuming while discovering is not! In the case of namespaces re/announcing is cheap so this solves the problem we had and thus almost completely solves the partial node problem in a more elegant way. That is, DASing and a discovery over namespaces(as a fallback) is possible.
  • Imagine a Celestia Node DASing the network from the end to beginning. While doing so, it discovers namespaced(partial) nodes of deployed rollups(if directly connected fullnode are overloaded) and requests shares from them. This works when the default pruning strategy removes all other namespaced data besides needed which in most cases should eliminate the need to prune past blocks by height. If own rollup data becomes to big we can granularize partiality even more, but this would far away, so I suggest starting the journey of partial nodes with namespaced nodes.
  • It is simpler to implement discovery for a namespace rather than for independent block.
    • Simply because former is implemented and the latter is not. We use IPFS stack and discovery for namespace is solved there, we just need to use it, like we do for all DASing stuff. Note that the current success of DASing would not be possible without beutifull IPFS's stack that was stress tested for years. I am sure that it will allow us DAS even TiB blocks without issues in its current state.
    • Interfaces allow us to implement this. When Bitswap can't find the share immediately, it knocks to discovery with a CID. The CID has a namespace that we can take and ask DHT for a set of peers that can serve it to us. Once connected Bitswap “want”s them and we are good.

• (2) Fits exceptionally well into the segregation of the Data and Application model. The Celestia Node would not just cares about data on theoretical/academic notion of modularity, but practically cares about data for Rollups on a physical and system level. Rollups here doing a minimal amount of work moving us closer to the world where a single CLI command launches your chain.

In general, on the main network, we want to decouple the data layer from the application layer.

[liamsi]

Our MVP experiments showed us that DHT re/announcing is the most time consuming while discovering is not! In the case of namespaces re/announcing is cheap so this solves the problem we had and thus almost completely solves the partial node problem in a more elegant way. That is, DASing and a discovery over namespaces(as a fallback) is possible.

The assumption here is that you keep the namespace for each height? What happens if you prune old heights though? Retry? Or will the key be a mix of height and namespace?

This works when the default pruning strategy removes all other namespaced data besides needed which in most cases should eliminate the need to prune past blocks by height.

I disagree with that. Not having pruning on the Cosmos Hub was a major pain for node operators. You could easily argue that the block data in the early days of the cosmos hub is comparable to some light weight rollup (back then it only supported gov proposals, staking Txs, and transfers).

[Wondertan]

The assumption here is that you keep the namespace for each height? Or will the key be a mix of height and namespace?

DASing in its current state does not know anything about heights. To make it so we would probably need to throw away the Bitswap and write something ourselves, as it hash based. So the assumption is that there is no heights and only namespaces encoded into cid.

What happens if you prune old heights though? Retry?

We rely on the assumption that there will be at least one node serving a height to the network. Therefore, the will be at least one node serving a namespaced sample of a block for the height. By discovering peers under the namespace, we are zooming into a subset of peers of the whole network, where at least one of them should provide us with the sample. The subset can be small enough and we might instantly connect to the provider or we might need to take some more time to find it if the subnet is bigger.

So to answer your question, if someone prunes by the height it would still be possible to find another one who did not prune.

I disagree with that. Not having pruning on the Cosmos Hub was a major pain for node operators. You could easily argue that the block data in the early days of the cosmos hub is comparable to some light weight rollup (back then it only supported gov proposals, staking Txs, and transfers).

Ok, also, you can still remove past heights. My point is more about rollup nodes wanted to be full nodes for the whole network. They will want this to be able to generate state/msg inclusion fraud proofs for the whole network. Namespacing allows them to do so by only caring there own namespace and mainnet namespace once the whole network becomes super expensive(imagine your example multiplied by thousands) by pruning other uninterested namespaces. Then If your own chain becomes super expensive you can still prune past blocks and be discovered for your namespace and serve back what you have.

So the idea here is not completely remove pruning be the height, but to make pruning and discovery two dimensional(namespace:height).

The following is the order of those being applicable to our network and thus the implementation:

1. Height pruning
2. Namespace pruning
3. Namespace discovery
4. Height discovery

The reasons why I put height discovery to the last place are:

• The most complicated to implement. Really, that's a new protocol, and all levels of our networking stack for data/shares know nothing about heights.
• We can live without it for a long time. Trying each peer in a namespace subnet will find you the one storing sample. In worst case that can take some time, especially if the subnet is hundreds of peers.
• It's an optimization. Think of it as one more zooming from a namespace subent into a namespace:height subnet. So by having this additional zooming we decrease the time it takes to find the peer who can serve as sample.

The namespace discovery on the other hand is much simpler to implement, as the whole stack is aware about namespaces deeply already and for it we don't need to implement our own protocol, so it can be a good middle term solution that is possible to deliver even before mainnet.