Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: s3 data validation: cross checking tiltseries & frames files #229

Merged

Conversation

daniel-ji
Copy link
Contributor

@daniel-ji daniel-ji commented Aug 28, 2024

Merge after #223.

See the Tilts & RawTilts & Angles Cross Checks section of https://docs.google.com/document/d/1yMKM0DW9KRhlcYiBGPcR7oW0liGtUew6NAmBhMg5U3w/edit.

Documentation from the code:

    """
    A class to test tilt angle data. Only checking that the # of angles in these files are consistent with other files / 
    data and that they properly map to other files / data. Other data validation tests is done in respective classes.
    Spans .tlt, .rawtlt, .mdoc, tiltseries_metadata.json, frames files.
    Ordering:
        - .tlt (<=) maps to .rawtlt (not necessarily 1:1)
        - .rawtlt (<=) maps to .mdoc (not necessarily 1:1)
        - .mdoc (==) one-to-one with frames files & tiltseries_metadata size["z"]
        - tiltseries metadata size["z"] == frames_count == # of frames files

    Extra test redundancy is added for the cases where files sometimes do not exist.
    Extra redundancy tests are done on the tiltseries metadata size["z"] since frames_count / # of frames files may be 0
        when no frames are present and that is a valid case. To ensure consistency, we rely on the check
        tiltseries metadata size["z"] == frames_count == # of frames files.
    """

…i/s3-data-validation-dataset-deposition-photos
…i/s3-data-validation-dataset-deposition-photos
… into daniel-ji/s3-data-validation-frames-gains
@daniel-ji daniel-ji force-pushed the daniel-ji/s3-data-validation-tiltseries branch from 9ad946a to 1deb4ab Compare August 28, 2024 03:00
@daniel-ji daniel-ji force-pushed the daniel-ji/s3-data-validation-angles-andframes-crosscheck branch from 17f776d to ca40efe Compare August 28, 2024 03:01
Comment on lines 32 to 38
@pytest.fixture(autouse=True)
def max_frames_count(self, tiltseries_metadata: Dict):
# TODO FIXME how to check if -0.0 should exist
return (
tiltseries_metadata["tilt_range"]["max"] - tiltseries_metadata["tilt_range"]["min"]
) / tiltseries_metadata["tilt_step"] + 1

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might not always hold true, as there is a possibility of frames that were generated not being used for the tiltseries.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, I'll remove all the max_frames_count checks in this PR and the future ones.

@daniel-ji daniel-ji changed the title s3 data validation: cross checking tiltseries & frames files feat: s3 data validation: cross checking tiltseries & frames files Sep 3, 2024
Copy link
Contributor

@manasaV3 manasaV3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comments documenting the tests are really helpful. 🔥

Base automatically changed from daniel-ji/s3-data-validation-tiltseries to main September 4, 2024 22:31
@daniel-ji daniel-ji merged commit 8244e9f into main Sep 4, 2024
4 checks passed
@daniel-ji daniel-ji deleted the daniel-ji/s3-data-validation-angles-andframes-crosscheck branch September 4, 2024 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants