Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Calculate hash for new non-deferred datasets when finishing a job #19181

Merged
merged 5 commits into from
Nov 21, 2024
Merged

Calculate hash for new non-deferred datasets when finishing a job #19181

merged 5 commits into from
Nov 21, 2024

Conversation

nsoranzo
Copy link
Member

This is configurable with two new options:

  • calculate_dataset_hash: in which cases Galaxy should calculate a hash for a new dataset. Possible values: 'always', 'upload' (the default), 'never'.
  • hash_function. Possible values: 'md5', 'sha1', 'sha256', 'sha512'

Hashes are calculated via a Celery task, so currently only if the 'enable_celery_tasks' option is set to true.

Also:

  • Type annotation improvements
  • Small refactorings and fixes

How to test the changes?

(Select all options that apply)

  • I've included appropriate automated tests.
  • This is a refactoring of components with existing test coverage.
  • Instructions for manual testing are as follows:
    1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

  • I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

@nsoranzo
Type annotation fixes
abef2e8
@nsoranzo
Remove duplicated statement
6929fbb 
And fix a typo.
@nsoranzo
Replace redundant HashFunctionNames with HashFunctionNameEnum
fe4589d 
And fix a typo.
@nsoranzo
Remove redundant check
aceaa11
@nsoranzo
Calculate hash for new non-deferred datasets when finishing a job
1b54c0b 
This is configurable with two new options:
- `calculate_dataset_hash`: in which cases Galaxy should calculate
  a hash for a new dataset. Possible values: 'always', 'upload'
  (the default), 'never'.
- `hash_function`. Possible values: 'md5', 'sha1', 'sha256', 'sha512'

Hashes are calculated via a Celery task, so currently only if
the 'enable_celery_tasks' option is set to true.
@nsoranzo nsoranzo requested review from mvdbeek and a team November 21, 2024 16:03
@github-actions github-actions bot added this to the 25.0 milestone Nov 21, 2024
jmchilton
jmchilton approved these changes Nov 21, 2024
View reviewed changes
Copy link
Member

@jmchilton jmchilton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh the anxiety this causes but I think this is a good step forward.

@nsoranzo
Copy link
Member Author

Oh the anxiety this causes but I think this is a good step forward.

In case of issues, changing calculate_dataset_hash to never would be a quick fix.

@nsoranzo
Copy link
Member Author

Integration test failure unrelated.

@jmchilton jmchilton merged commit 4c46062 into galaxyproject:dev Nov 21, 2024
57 of 58 checks passed
@nsoranzo nsoranzo deleted the dataset_hashes_at_job_finish branch November 21, 2024 23:52
@nsoranzo nsoranzo mentioned this pull request Nov 22, 2024
Draft
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mvdbeek mvdbeek mvdbeek left review comments

@jmchilton jmchilton jmchilton approved these changes

Assignees
No one assigned
Labels
area/admin area/database Galaxy's database or data access layer area/datatypes area/documentation area/jobs area/testing/api area/testing/integration area/testing area/tool-framework area/util BioHackEU24 kind/enhancement
Projects
None yet
Milestone
25.0
Development

Successfully merging this pull request may close these issues.
3 participants
@nsoranzo @jmchilton @mvdbeek