Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Call for input: Performance tricks for batch query multithreaded contract query #631

Open
banacer opened this issue May 2, 2022 · 0 comments

Comments

@banacer
Copy link

banacer commented May 2, 2022

Let's say I want to download all the uniswap v2 pairs with their reserves. One will do the following:
1- create the uniswap v2 contract object query, then contract.query AllPairsLength
2- foreach i in AllPairsLength contract.query allPairs(i) => this returns an address
2.1 - create a uniswapv2Pair contract using the address returned
2.2 - query the contract for functions token0 and token1 and getReserves
given there are 70k+ pairs, this can be done concurrently, one can simply create a create 10+ threads to concurrently fetch 1000 address per batch and for each of them create batch a batch to query 2.2

I have done something similar to what is described above in both golang (using github.com/lmittmann/w3) and this crate in rust but I have noticed that the rust version is at least 10x slower than golang. Both are running in the same machine query an erigon full node.
I have done the following approaches to speed things up:
1- create a single WebSocket and create a web3<batch> and cloned it in every tokio threac
2- created a single WebSocket to get the AllPairsLength but created new WebSocket object inside every thread to have a unique websocket connection. I create this in step 2- of the pseudo code above.

option 2 was even slower.
I am not sure why there is such a large performance difference between my rust implementation and golang implementation and I figured I may be using this crate in the wrong way.
Can you provide guidance as to whether we should use the same websocket or create a new websocket for faster performance?
Any tricks to speed things up.

Appreciate your help and apologies for posting it here as I am not aware of another medium to share this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant