Skip to content
This repository was archived by the owner on Sep 8, 2020. It is now read-only.

Add multithreading to tools/dataProcessor.js #34

Open
nareddyt opened this issue Mar 23, 2018 · 0 comments
Open

Add multithreading to tools/dataProcessor.js #34

nareddyt opened this issue Mar 23, 2018 · 0 comments
Labels
enhancement New feature or request

Comments

@nareddyt
Copy link
Owner

Currently, the data processor processes deals and cashback rewards one at a time. This takes around 2 minutes to process all data.

It would be significantly faster if this was multithreaded so it can process multiple items at once. Since the majority of time is spent in the Googling hostnames phase (where we have to wait for network responses), it could definitely help.

One concern is that Google might rate limit or even mark us as a bot if we were to increase the speed. Someone will have to try it out and see.

@nareddyt nareddyt added the enhancement New feature or request label Mar 23, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant