New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

todos #1

Open

4 of 5 tasks

tomlue opened this issue Aug 7, 2024 · 0 comments

Assignees

Contributor

tomlue commented Aug 7, 2024 •

edited by bhlieberman

Loading

I added todos

dvc.yaml - todos for a status stage and for updating every stage to handle updates with persist: true
check_pdfs - need to check corrupt pdfs and redownload with ultrapremium
01_load_urls.sh - we shouldn't have a hardcoded date, we need to rewrite to handle updates.

extra todo:

output a brick/docs.parquet that associates urls with relative pdf filepaths
run, commit to dvc.lock, and push to s3

tomlue assigned bhlieberman

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment