feat: Update ingestion config validation for ctf and alignment entities #262

Bento007 · 2024-09-10T00:01:15Z

related to chanzuckerberg/cryoet-data-portal#997

Description

update existing ingest configs

added a migrate script to migrate existing ingest_configs to the new schema. This cause some reformatting of spaces and newlines of unrelated fields.
removed fields that were empty lists or empty mappings except for annotations

collection_metadata

move *.mdocs from rawtilts[].sources to collection_metadata[].sources[].source_multi_glob.list_globs

alignments

copytomogram[0].metadata.affine_transformation_matrix to alignment[].metadata.affine_transformation_matrix if it is not an identity matrix, and if alignment sources existed. affine_transformation_matrix was copied into each alignment[].source
add alignment[].metadata.format and set to appropriate value based on source formats.
- move *.tlt, *.xf, *.com, and *.aln from rawtilts[].sources to collection_metadata[].sources[].source_multi_glob.list_globs

tomograms

remove tomogram[].metadata.affine_transformation_matrix if it is an identity matrix.
add tomogram[].metadata.dates from deposition[0].metadata.dates if it existed, else the epoch date was used for all dates. Here is a list of the configs a default date was used
10011_draft.yaml
~~10431.yaml~~
~~10001.yaml~~
~~10002.yaml~~
~~10302.yaml~~
~~10427.yaml~~
~~10428.yam~~
~~10429.yaml~~
~~10430.yaml~~
~~10432.yaml~~
~~10434.yaml~~
~~10435.yaml~~
~~10437.yaml~~
~~10438.yaml~~
add tomogram[].metadata.is_visualization_default with default set to true

Template.yaml

add alignments
- alignment_type
- offset
- x_roation_offset
- tilt_offset
- affine_transformation_matrix
- is_canonical
- format
- volume_dimesion
add annotations[].sources.*.is_portal_standard
add frames[].metadata
- dose
- defocus
- astigmatism
- astigmatic_angle
add tomograms[].metadata
- is_visualization_default
- is_portal_standard
- cross_references
- dates

Ingestion Config Schema

define alignment metadata and sources and add it to ingest_config_models.yaml
define frame metadata and add it to the ingest_config_models.yaml
add the alignment Container for linkML
add extended validation for alignment sources.
add extended validation for affine_transformation_matrix in alignment and remove it for tomograms

Testing

Generated the schema using make build-ingestion-config
verified the schemas using make validate-configs and make validate-configs-with-network

Note

If you have better ideas for the descriptions in the schema I'd be happy to use them.

# Conflicts: # ingestion_tools/dataset_configs/template.yaml # schema/ingestion_config/v1.0.0/codegen/ingestion_config_models.py

- add enums for alignment types and format

…smith/997-ctf-alignment # Conflicts: # ingestion_tools/dataset_configs/template.yaml # schema/core/v1.1.0/metadata.yaml

# Conflicts: # ingestion_tools/dataset_configs/10003.yaml # ingestion_tools/dataset_configs/10006.yaml # ingestion_tools/dataset_configs/10436.yaml # ingestion_tools/dataset_configs/10439.yaml

jgadling · 2024-09-25T18:40:21Z

ingestion_tools/scripts/transform_ingestion_configs.py

@@ -35,6 +36,65 @@ def has_no_sources(data: list[dict[str, Any]] | dict[str, Any]) -> bool:
    return isinstance(data, dict) or not any(row.get("sources") for row in data)


+def rawtilts_to_alignments(data: dict) -> None:


Nit for this function in general -- in Python, I think the return early pattern is best practice. There are several points in this function where we do a check for a value and then nest more logic inside that only gets executed in that case, and yet there's no else condition - we could have just returned from this function (or added a continue/break to our loop) if the condition wasn't met. The code here would be shorter and less deeply nested (thus: easier to read) if it were refactored in this way

manasaV3

🚀

manasaV3 · 2024-09-25T18:24:54Z

schema/requirements.txt

@@ -1,3 +1,3 @@
-linkml
+linkml==1.8.2 # upgrade blocked by https://github.com/chanzuckerberg/cryoet-data-portal-backend/issues/274


Thanks for capturing this issue. :)

manasaV3 · 2024-09-25T18:26:17Z

schema/ingestion_config/v1.0.0/ingestion_config_models.yaml

+      - dataset
+      - deposition
+      - run
+      - tomogram


As tomogram cannot be the parent of alignment, this should be updated?

manasaV3 · 2024-09-25T18:38:10Z

schema/ingestion_config/migrations/issue_997.py

+                config['collection_metadata'] = [{"sources": [{"source_multi_glob": {"list_globs": []}}]}]
+            config["collection_metadata"][0]["sources"][0]["source_multi_glob"]["list_globs"].extend(list_globs)
+
+def rawtilts_to_alignments(data: dict) -> None:


I believe this has been moved to ingestion_tools/scripts/transform_ingestion_configs.py. Do we still want this here?

manasaV3 · 2024-09-25T18:47:41Z

schema/ingestion_config/migrations/issue_997.py

+import yaml
+
+
+def rawtilts_to_collection_metadata(config: dict) -> None:


I believe this is the only method that hasn't been ported over to the script.

…nfigs.py

jgadling

Approved once these things are completed:

Add code from ingest_997.py to transform_ingestion_configs.py
Make sure tomograms can't be a parent of alignments in the ingestion configs

- Add a docstring and description for the upgrade command - Update transform_ingestion_configs to be extended to support upgrading an ingest config from version 0.0.0 and beyond. - move schema migrations code to ingest_tools/schema_migration - update gjensen_config.py to run with reorganization. - related to #262

# Conflicts: # ingestion_tools/dataset_configs/10002.yaml # ingestion_tools/dataset_configs/10004.yaml # ingestion_tools/dataset_configs/10008.yaml # ingestion_tools/dataset_configs/10009.yaml # ingestion_tools/dataset_configs/10426.yaml # ingestion_tools/dataset_configs/10436.yaml # ingestion_tools/dataset_configs/10437.yaml # ingestion_tools/dataset_configs/10438.yaml # ingestion_tools/dataset_configs/10439.yaml # ingestion_tools/dataset_configs/deposition_10301.yaml # ingestion_tools/dataset_configs/deposition_10303.yaml # ingestion_tools/dataset_configs/deposition_10304.yaml # ingestion_tools/dataset_configs/deposition_10308.yaml # ingestion_tools/scripts/schema_migration/transform_ingestion_configs.py

Bento007 · 2024-09-26T22:38:19Z

follow up PR #292

Bento007 and others added 30 commits September 4, 2024 17:59

add collection_metadata to template and update configs

f284234

add collection metadata to ingestion config

bb63951

add collection metadata to ingest_config_schema

99adb45

address comments

a7b83cc

address comments

b4319d5

add extended validation check for collection metadata (#259)

9aaf3f4

Merge branch 'refs/heads/main' into tsmith/997-ctf

fa05f4f

# Conflicts: # ingestion_tools/dataset_configs/template.yaml # schema/ingestion_config/v1.0.0/codegen/ingestion_config_models.py

update schema

b4c611b

remove rawtilts from 10010.yaml

87f7732

lint

7421f5d

Merge branch 'refs/heads/main' into tsmith/997-ctf-alignment

bf3a2cb

update ingest schema

35b05ee

update ingest config template

991b9d7

add AlignmentImporter

06e1ea0

lint

e9b190d

correct offset format

8473693

update existing ingestion_configs

df622bb

Add alignment to ingest_config validation

49e9290

- add format, volume_dimensions to ingest config schema

ac69f5e

- add enums for alignment types and format

Merge remote-tracking branch 'origin/tsmith/997-ctf-alignment' into t…

94f2c02

…smith/997-ctf-alignment # Conflicts: # ingestion_tools/dataset_configs/template.yaml # schema/core/v1.1.0/metadata.yaml

add alignment metadata

b92c420

fix format documentation

1e8fe98

Update config for alignment

afe9d0c

Update config for tomogrames and frames

f89290d

update template and schema

72208ba

update all configs

a48b4a4

Merge branch 'refs/heads/main' into tsmith/997-ctf-alignment

796d647

add docstring to migrate script

97ab3c9

run linter

3ec3c58

removed extra changes and run schema gen

517df5c

Bento007 added 11 commits September 20, 2024 17:38

add identity matrix back in

2e58482

remove voxel spacing from parent filter of alignment

465893c

lock the verison of linkml and regenerate files

54c811f

add xtlt and copy to transform_ingestion_configs.py

90734f1

lint

a3394fb

iron out tilt formats

e2072d8

iron out tilt formats

2096a74

update schema

4b627a2

Merge branch 'refs/heads/main' into tsmith/997-ctf-alignment

05c4769

# Conflicts: # ingestion_tools/dataset_configs/10003.yaml # ingestion_tools/dataset_configs/10006.yaml # ingestion_tools/dataset_configs/10436.yaml # ingestion_tools/dataset_configs/10439.yaml

merge main

61c75e9

update transform_ingestion_configs.py

ffafd79

Bento007 requested a review from manasaV3 September 25, 2024 16:36

jgadling reviewed Sep 25, 2024

View reviewed changes

manasaV3 reviewed Sep 25, 2024

View reviewed changes

flatten code in ingest_997.py add dosctring to transform_ingestion_co…

326f227

…nfigs.py

Bento007 mentioned this pull request Sep 26, 2024

feat: transform_ingestion_configs.py support multiple versions #290

Merged

jgadling approved these changes Sep 26, 2024

View reviewed changes

remove tomogram from alignment in ingest-config

4048d79

Bento007 added 2 commits September 26, 2024 14:27

move migrate script to ingest-tools

dded1d1

Bento007 requested a review from manasaV3 September 26, 2024 21:46

Bento007 added 3 commits September 26, 2024 14:48

lint

918ed8e

update configs

6912e06

liknt

15f0309

manasaV3 approved these changes Sep 27, 2024

View reviewed changes

Bento007 merged commit 6c52df8 into main Sep 27, 2024
6 of 7 checks passed

Bento007 deleted the tsmith/997-ctf-alignment branch September 27, 2024 16:28

Bento007 restored the tsmith/997-ctf-alignment branch September 27, 2024 17:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Update ingestion config validation for ctf and alignment entities #262

feat: Update ingestion config validation for ctf and alignment entities #262

Bento007 commented Sep 10, 2024 •

edited

Loading

jgadling Sep 25, 2024 •

edited

Loading

manasaV3 left a comment •

edited

Loading

manasaV3 Sep 25, 2024

manasaV3 Sep 25, 2024

manasaV3 Sep 25, 2024

manasaV3 Sep 25, 2024

jgadling left a comment •

edited

Loading

Bento007 commented Sep 26, 2024

		@@ -35,6 +36,65 @@ def has_no_sources(data: list[dict[str, Any]] \| dict[str, Any]) -> bool:
		return isinstance(data, dict) or not any(row.get("sources") for row in data)


		def rawtilts_to_alignments(data: dict) -> None:

		@@ -1,3 +1,3 @@
		linkml
		linkml==1.8.2 # upgrade blocked by https://github.com/chanzuckerberg/cryoet-data-portal-backend/issues/274

		import yaml


		def rawtilts_to_collection_metadata(config: dict) -> None:

feat: Update ingestion config validation for ctf and alignment entities #262

feat: Update ingestion config validation for ctf and alignment entities #262

Conversation

Bento007 commented Sep 10, 2024 • edited Loading

Description

update existing ingest configs

collection_metadata

alignments

tomograms

Template.yaml

Ingestion Config Schema

Testing

Note

jgadling Sep 25, 2024 • edited Loading

Choose a reason for hiding this comment

manasaV3 left a comment • edited Loading

Choose a reason for hiding this comment

manasaV3 Sep 25, 2024

Choose a reason for hiding this comment

manasaV3 Sep 25, 2024

Choose a reason for hiding this comment

manasaV3 Sep 25, 2024

Choose a reason for hiding this comment

manasaV3 Sep 25, 2024

Choose a reason for hiding this comment

jgadling left a comment • edited Loading

Choose a reason for hiding this comment

Bento007 commented Sep 26, 2024

Bento007 commented Sep 10, 2024 •

edited

Loading

jgadling Sep 25, 2024 •

edited

Loading

manasaV3 left a comment •

edited

Loading

jgadling left a comment •

edited

Loading