feat: run / tiltseries s3 data validation (zarr, mrc, mdoc, and metadata checks) #223
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…i/s3-data-validation-dataset-deposition-photos
…i/s3-data-validation-dataset-deposition-photos
… into daniel-ji/s3-data-validation-frames-gains
…ji/s3-data-validation-tiltseries
daniel-ji
changed the base branch from
daniel-ji/s3-data-validation-frames-gains
to
main
daniel-ji
changed the base branch from
main
to
daniel-ji/s3-data-validation-frames-gains
daniel-ji
changed the title
daniel-ji
force-pushed
the
daniel-ji/s3-data-validation-frames-gains
branch
from
84da6a5
to
251214b
Compare
daniel-ji
force-pushed
the
daniel-ji/s3-data-validation-tiltseries
branch
from
9ad946a
to
1deb4ab
Compare
manasaV3
reviewed
Sep 2, 2024
manasaV3
requested changes
Sep 2, 2024
Comment on lines
58
to
59
| for ex in exclude: | ||
| tentatives = [tent for tent in tentatives if ex not in tent] |
There was a problem hiding this comment.
This check should be a part of the fixture that generates the run_names and not be done here. Which it already is.
There was a problem hiding this comment.
fixed in a later pr, see #250
|
|
||
| zarrays = header_data["zarrays"] | ||
| for i, zarray in zarrays.items(): | ||
| header = self.mrc_headers[mrc_file].header |
There was a problem hiding this comment.
nit: Could this header be outside the for loop?
Comment on lines
110
to
122
| ### BEGIN MDOC consistency tests ### | ||
| def test_tiltseries_pixel_spacing_mdoc(self, tiltseries_metadata: Dict, tiltseries_mdoc: pd.DataFrame): | ||
| """Check that the tiltseries pixel spacing matches the MDOC data.""" | ||
| assert len(set(tiltseries_mdoc["PixelSpacing"])) == 1 | ||
| assert tiltseries_metadata["pixel_spacing"] == tiltseries_mdoc["PixelSpacing"][0] | ||
|
|
||
| def test_tiltseries_image_size_mdoc(self, tiltseries_metadata: Dict, tiltseries_mdoc: pd.DataFrame): | ||
| """Check that the tiltseries image size matches the MDOC data.""" | ||
| assert len(set(tiltseries_mdoc["ImageSize"])) == 1 | ||
| assert tiltseries_metadata["size"]["x"] == tiltseries_mdoc["ImageSize"][0][0] | ||
| assert tiltseries_metadata["size"]["y"] == tiltseries_mdoc["ImageSize"][0][1] | ||
|
|
||
| ### END MDOC consistency tests ### |
There was a problem hiding this comment.
While the mdoc files are saved along with the tiltseries at the moment, they are associated to the frames. As, there is processing to go from frames to tiltseries, it is very unlikely for us to have the same Image size or pixel spacings. So, these two tests are more likely to fail and that is the expected behaviour.
Base automatically changed from
daniel-ji/s3-data-validation-frames-gains
to
main
September 3, 2024 00:28
daniel-ji
changed the title
manasaV3
approved these changes
Sep 4, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merge after #207.
See the tiltseries & runs sections of https://docs.google.com/document/d/1yMKM0DW9KRhlcYiBGPcR7oW0liGtUew6NAmBhMg5U3w/edit