New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce compute times: reduce costs #63

Open

radumas opened this issue Jul 2, 2019 · 1 comment

Labels

Milestone

0-Data Pipeline

Collaborator

radumas commented Jul 2, 2019

Because of #54 the script takes... longer. This also means it costs more.

Any way to reduce that compute time without getting blocked would be great!

radumas added the help wanted label

radumas added this to the 0-Data Pipeline milestone

Collaborator Author

radumas commented Aug 7, 2019

Forgot that we switched back to the single-thread scraper in order not to trigger flood protection on the API.

Seems like a way of limiting this is found here

radumas added a commit that referenced this issue


          #63 implemented connection limit in asyncio

8bf1623

radumas added a commit that referenced this issue


          #63 Bugfix with asyncio & raise_for_status

f350561

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment