Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cleanup jobs as Celery periodic tasks #2434

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open

Conversation

noliveleger
Copy link
Contributor

@noliveleger noliveleger commented Oct 2, 2019

3 periodic tasks have been added:

  • remove_s3_orphans: Remove every export in bucket which is not related to a record in a DB.
  • remove_import_tasks: Remove every import task older than --days days. (90 by default)
  • remove_asset_snaphots: Remove every asset snapshot older than --days days. (90 by default)

Closes kobotoolbox/kobo-docker#253
Related to kobotoolbox/kobocat#579

@jnm jnm force-pushed the cron-cleanup-jobs branch from 67372c9 to 893664e Compare October 22, 2019 20:00
Copy link
Member

@jnm jnm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, we're sitting at the same table. 👋

kobo/settings/base.py Outdated Show resolved Hide resolved
kpi/tasks.py Outdated Show resolved Hide resolved
kpi/management/commands/remove_s3_orphans.py Show resolved Hide resolved
kpi/utils/extended_s3boto_storage.py Outdated Show resolved Hide resolved
kpi/utils/lock.py Outdated Show resolved Hide resolved
kpi/utils/lock.py Show resolved Hide resolved
kpi/management/commands/remove_s3_orphans.py Show resolved Hide resolved
kpi/management/commands/remove_s3_orphans.py Outdated Show resolved Hide resolved
kpi/management/commands/remove_s3_orphans.py Outdated Show resolved Hide resolved
kpi/management/commands/remove_s3_orphans.py Show resolved Hide resolved
@noliveleger noliveleger requested a review from jnm November 12, 2019 14:14
@noliveleger noliveleger added the enhancement Ideas, improvements and features label Nov 19, 2019
@jnm jnm assigned dorey and unassigned jnm Mar 16, 2020
Copy link
Contributor

@dorey dorey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this and it works on a local database

@dorey dorey removed their assignment Jun 4, 2020
@noliveleger noliveleger reopened this Jul 2, 2020
@noliveleger noliveleger changed the base branch from two-databases to master July 2, 2020 14:42
@dorey dorey self-assigned this Jun 15, 2021
Copy link
Member

@jnm jnm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bufke to have a look

I'm generally 👍 on this, but let's verify that we aren't trashing anything precious in S3. If the cleanup only affects exports, I'm fine with it staying in this PR, but if it touches irreplaceable things like asset media files, then I'd rather have it split into a separate PR with greater scrutiny

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Back end enhancement Ideas, improvements and features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add cron task for KPI and KoBoCAT cleanup jobs
3 participants