IPC state provider

[mralj]

📝 Summary

rbuilder can now work with any node via IPC.
This means that any node can provide the state for rbuilder, revm is still used as the EVM.
Node requirements are that it exposes the following RPC calls:

Calls not available in the ETH JSON RPC spec:

1. rbuilder_calculateStateRoot for state root calculation
2. rbuilder_getCodeByHash given Bytecode hash, returns Bytecode

Calls optimised for rbuilder, but have a counterpart in the ETH JSON RPC spec:

3. rbuilder_getBlockHash gets the block hash given block number (similar to eth_getBlockByNumber, but just returns block hash)
4. rbuilder_getAccount gets account info (similar to eth_getProof, but w/o proof stuff)

To use rbuilder with node via IPC, the config.toml must have the following config (example):

[ipc_provider]
ipc_path = "/root/execution-data/nethermind.ipc"
mempool_server_url = "ws://localhost:8546"
request_timeout_ms = 75

Implementation details

IPC

This implementation was initially intended to introduce a remote state provider. By remote, I mean that the idea was that state could be provided via HTTP/WS/IPC. Unfortunately, due to implementation issues/constraints, I've decided only to implement state provisioning via IPC.
I don't think this has any practical downside, especially since the state provider must be fast. There is a non-trivial number of calls to read state (~300/s), meaning it would be unrealistic to use this over the network and have near disk read latency.

Code-wise, issues above and constraints stem mainly from the fact that the traits to read state are sync. Initially, I relied on tokio and alloy.rs to fetch this remote state, but this implementation had many issues.
Firstly, each call to fetch any data (e.g. fetching account or Bytecode) had to be wrapped in a function call like this:

    /// Runs fututre in sync context
    // StateProvider(Factory) traits require sync context, but calls to remote provider are async
    // What's more, rbuilder is executed in async context, so we have a situation
    // async -> sync -> async
    // This helper function allows execution in such environment
    fn run<F, R>(&self, f: F) -> R
    where
        F: Future<Output = R>,
    {
        tokio::task::block_in_place(|| self.runtime_handle.block_on(f))
    }

This adds additional overhead on Tokio runtime and doesn't play well with some parts of the codebase, specifically mutex locking. Not to go too deep into explaining issues around this in the PR description, but we would end up in scenarios where the whole Tokio runtime I/O would be blocked or we would dead-lock parking-lot mutexes (in some scenarios).

The solutions (monitoring thread + async mutex) seemed hacky and suboptimal.
This is why, in the end, I reached for sync (but concurrent) IPC solution, which I implemented from scratch here. It's called REIPC (coming from request/response IPC).

This solution was tested using Nethermind node, and while Nethermind will do some improvements on IPC to reduce latency and increase throughput, here are the initial request&response latencies:

Dashmap & QuickCache in IPC provider

We need caches because otherwise the number of IPC calls would be pretty high (in thousands per sec).
I also reached for concurrent caches because both StateProviderFactory and StateProvider need to be Send + Sync.

QuickCache is used so that I don't have to implement concurrent-cache invalidation by hand :)
Here is some info on QuickCache vs Moka (other popular caching crate), TL;DR; for our simple case QuickCache seems a better fit.

On enum StateProviderFactories

The reason cli.rs passes StateProviderFactories enum to config.new_builder is because I wanted the state provider for rbuilder to be chosen via config at the runtime.
This is why, AFAIK, static dispatch is not an option. So I was left with the choice of refactoring to dynamic dispatch (akin to StateProviderBox) OR the enum solution.
I chose the enum solution for following reasons:
1. It seemed to me that code diff would be smaller (smaller change to implement)
2. AFIAK it's faster. Enum matching will compile to JMP vs CALL (in case of dynamic dispatch), which compiler will be able to optimize better (especially in this scenario), and given branch predicitoning I guess that it'll be almost free (+ no need for vtable/pointer loading).]

UPDATE: this was properly handled in this commit thanks to suggestions from @ZanCorDX

On MempoolSource enum

The reason I chose to use WebSockets for streaming transactions when rbuilder uses IPC state provider is because, currently, REIPC doesn't support ETH subscriptions/streams.

✅ I have completed the following steps:

• Run make lint
• Run make test
• Added tests (if applicable)

[SozinM]

I'm in favor of using concrete type here. This will help in case we will do any refactoring and accidentally change parameters. This way we would be able to find problem instantly.

[mralj]

Sorry, I completely missed the comment 🤦‍♂️
I disagree; the type here is inferred from the function arguments, i.e., (num, false). ' If we mess it up, rust will yell at us.
That being said, I don't feel strongly about this if @ZanCorDX agrees with you, I'll refactor :)

[SozinM]

hm, will it yell if we change params to (num, false, false) for example? If yes - all good
But i assume it will send them to rpc and get error because there is extra false param

[mralj]

Correct, it won't :)
So the idea is that if we have written this out in 2 places (in the type and in the args) there is smaller chance of us making mistake in both places when refactoring?

ie. If we change one and not the other compiler will give us error, and we'll stop to think about i (hopefully not just copying args to type, because then compiler will tell you everything is ok and you'll still get rpc error xD)

But all in all I agree this reduces the chance of developer error, I'll fix it :)

[mralj]

@SozinM done :)
Sorry it took me so long!

fbe5087

[dvush]

Great! Thanks, I'll review it this week. (cc @ZanCorDX don't merge before pls)

[dvush]

Awesome, pretty clear and isolated change. I've been running it locally and it works fine.

The only thing that I notices is root hash is much slower on this version. I can give more data tomorrow when I run the default rbuilder. But if feels like its 2-3x slower.

I think our root hash implementation (eth-sparse-mpt) is quite good for the block building purposes and it will be much better than generic root hash when we finish all the possible improvements and it can be used for integrations with other clients through API. The only thing that it needs is ability to fetch proofs using path. Instead of default eth_getProof that asks for address we need to fetch proof by hash(address) and hash(slot) for the given hash(address)/address

[dvush]

Can you please clarify for me the difference between DashMap and Cache here?

[mralj]

code_cache can live as long as rbuilder lives, i.e., it doesn't have to be invalidated because it will stay the same no matter the current state.

On the other hand, we need to clean state_provider_by_hash after we are done with the StateProvider for some block, because after slot is finished, this data becomes useless and just grows.
The reason I went with Cache here instead of Dashmap is because I didn't want to write concurrent cache invalidation myself :)

[mralj]

I think our root hash implementation (eth-sparse-mpt) is quite good for the block building purposes and it will be much better than generic root hash when we finish all the possible improvements and it can be used for integrations with other clients through API. The only thing that it needs is ability to fetch proofs using path. Instead of default eth_getProof that asks for address we need to fetch proof by hash(address) and hash(slot) for the given hash(address)/address.

Thanks for pointing this out, I'll discuss with my (NMC) team and see how we can leverage this, I'll also add some logging to our node to see how much milli seconds State Root Calculation actually takes and will get back to you on numbers :)