Add audbackend.checksum()

[hagenw]

Tackles part of audeering/audb#459

This copies the checksum handling for parquet files from audb.core.utils.md5() into the new function audbackend.checksum(). Inside audbackend, it replaces the usage of audeer.md5() by it, to make sure we are getting the correct checksum for local parquet files, which we use for example inside audbackend.backend.Base.get_file() to decide if the file exists already locally.

The checksum for parquet is given by its "hash" metadata entry, or if not available by audeer.md5().
As pyarrow is not a dependency of audbackend, and should also not be, I decided to also use audeer.md5() if pyarrow is not installed and we can hence not access the metadata of the parquet file (or is there a way to access it with pure Python functions?).

Summary by Sourcery

Add a new checksum function to the audbackend module, replacing the existing MD5 checksum method with support for deterministic checksums for parquet files using metadata. Update documentation and tests accordingly.

New Features:

• Introduce a new function audbackend.checksum() to calculate checksums for files, including support for deterministic checksums for parquet files using metadata.

Enhancements:

• Replace the usage of audeer.md5() with the new audbackend.checksum() function across the codebase for improved checksum handling.

Documentation:

• Update documentation to include the new audbackend.checksum() function and its usage.

Tests:

• Add new tests for the audbackend.checksum() function to ensure correct checksum calculation for different file types, including parquet files with and without metadata.

Summary by Sourcery

Add a new checksum function to the audbackend module, replacing the existing MD5 checksum method with support for deterministic checksums for parquet files using metadata. Update documentation and tests accordingly.

New Features:

• Introduce a new function audbackend.checksum() to calculate checksums for files, including support for deterministic checksums for parquet files using metadata.

Enhancements:

• Replace the usage of audeer.md5() with the new audbackend.checksum() function across the codebase for improved checksum handling.

Documentation:

• Update documentation to include the new audbackend.checksum() function and its usage.

Tests:

• Add new tests for the audbackend.checksum() function to ensure correct checksum calculation for different file types, including parquet files with and without metadata.

[sourcery-ai]

Reviewer's Guide by Sourcery

The PR introduces a new checksum() function in audbackend that handles file checksums with special treatment for parquet files. The implementation copies checksum handling from audb.core.utils.md5() and replaces all usage of audeer.md5() within the codebase. For parquet files, it first attempts to read the checksum from the file's metadata under the "hash" key, falling back to MD5 calculation if the metadata is unavailable or if pyarrow is not installed.

Class diagram for the new checksum function

classDiagram
    class audbackend {
        +String checksum(String file)
    }
    class audeer {
        +String md5(String file)
    }
    audbackend --|> audeer : uses
    note for audbackend "The checksum function handles parquet files with metadata and falls back to MD5 if necessary."

Loading

File-Level Changes

Change	Details	Files
Implement new checksum calculation function with parquet file support	Add new checksum() function that handles both regular and parquet files Add support for reading precalculated checksums from parquet metadata Implement fallback to MD5 calculation when metadata is unavailable or pyarrow is not installed Add comprehensive docstring with examples	audbackend/core/utils.py
Replace existing MD5 checksum calculations throughout the codebase	Update checksum comparisons in file operations Replace audeer.md5() calls with new utils.checksum() Update file existence checks to use new checksum function	audbackend/core/backend/base.py audbackend/core/backend/filesystem.py audbackend/core/interface/versioned.py tests/bad_file_system.py
Add comprehensive test coverage for new checksum functionality	Create test cases for regular files and parquet files Add tests for scenarios with and without pyarrow installed Test parquet files with and without metadata hash	tests/test_utils.py
Update project dependencies and documentation	Add audformat to test requirements Add intersphinx mapping for audformat documentation	tests/requirements.txt docs/conf.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.0%. Comparing base (e75620a) to head (791818b).
Report is 2 commits behind head on main.

Additional details and impacted files

Files with missing lines	Coverage Δ
audbackend/__init__.py	100.0% <100.0%> (ø)
audbackend/core/backend/base.py	100.0% <100.0%> (ø)
audbackend/core/backend/filesystem.py	100.0% <100.0%> (ø)
audbackend/core/interface/versioned.py	100.0% <ø> (ø)
audbackend/core/utils.py	100.0% <100.0%> (ø)

[sourcery-ai]

Hey @hagenw - I've reviewed your changes and they look great!

Here's what I looked at during the review

• 🟢 General issues: all looks good
• 🟢 Security: all looks good
• 🟡 Testing: 1 issue found
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[ChristianGeng]

All threads resolved. Approving.