Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup an aysnc io script once the scrapers are in place #9

Open
sakethramanujam opened this issue Nov 20, 2019 · 2 comments
Open

Setup an aysnc io script once the scrapers are in place #9

sakethramanujam opened this issue Nov 20, 2019 · 2 comments
Assignees
Labels
api issues and prs related to api

Comments

@sakethramanujam
Copy link
Member

Add an Asyncio scripts that runs all scrapers in intervals to periodically update the db with news articles.

Decide on a workflow mechanism to run async scripts.
Whether it should be a client that the scraper should call or one single script that runs all the scrapers at once is up to you.

@sakethramanujam sakethramanujam added the api issues and prs related to api label Nov 20, 2019
@mm-git01
Copy link

a client that calls the scrappers and waits or sleeps for certain info would be a good workflow.Some asyc threads could wait on certain scrapers to update while some others who have already got the data get scheduled for the next run as soon as the data is written.

@mm-git01
Copy link

@sakethramanujam have you found someone to write the scrapers yet ? this issue is being blocked by the scrapers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api issues and prs related to api
Projects
None yet
Development

No branches or pull requests

2 participants