Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Update ingestion config validation for ctf and alignment entities #262

Merged
merged 57 commits into from
Sep 27, 2024
Merged

feat: Update ingestion config validation for ctf and alignment entities #262

Show file tree
Hide file tree
Changes from 39 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
f284234
add collection_metadata to template and update configs
Bento007 Sep 5, 2024
bb63951
add collection metadata to ingestion config
Bento007 Sep 5, 2024
99adb45
add collection metadata to ingest_config_schema
Bento007 Sep 5, 2024
a7b83cc
address comments
Bento007 Sep 5, 2024
b4319d5
address comments
Bento007 Sep 6, 2024
9aaf3f4
add extended validation check for collection metadata (#259)
daniel-ji Sep 6, 2024
fa05f4f
Merge branch 'refs/heads/main' into tsmith/997-ctf
Bento007 Sep 6, 2024
b4c611b
update schema
Bento007 Sep 6, 2024
87f7732
remove rawtilts from 10010.yaml
Bento007 Sep 6, 2024
7421f5d
lint
Bento007 Sep 7, 2024
bf3a2cb
Merge branch 'refs/heads/main' into tsmith/997-ctf-alignment
Bento007 Sep 9, 2024
35b05ee
update ingest schema
Bento007 Sep 9, 2024
991b9d7
update ingest config template
Bento007 Sep 9, 2024
06e1ea0
add AlignmentImporter
Bento007 Sep 9, 2024
e9b190d
lint
Bento007 Sep 9, 2024
8473693
correct offset format
Bento007 Sep 9, 2024
df622bb
update existing ingestion_configs
Bento007 Sep 9, 2024
49e9290
Add alignment to ingest_config validation
Bento007 Sep 11, 2024
ac69f5e
- add format, volume_dimensions to ingest config schema
Bento007 Sep 12, 2024
94f2c02
Merge remote-tracking branch 'origin/tsmith/997-ctf-alignment' into t…
Bento007 Sep 12, 2024
b92c420
add alignment metadata
Bento007 Sep 12, 2024
1e8fe98
fix format documentation
Bento007 Sep 12, 2024
afe9d0c
Update config for alignment
Bento007 Sep 13, 2024
f89290d
Update config for tomogrames and frames
Bento007 Sep 13, 2024
72208ba
update template and schema
Bento007 Sep 13, 2024
a48b4a4
update all configs
Bento007 Sep 13, 2024
796d647
Merge branch 'refs/heads/main' into tsmith/997-ctf-alignment
Bento007 Sep 13, 2024
97ab3c9
add docstring to migrate script
Bento007 Sep 13, 2024
3ec3c58
run linter
Bento007 Sep 13, 2024
517df5c
removed extra changes and run schema gen
Bento007 Sep 13, 2024
2a2ca56
lint
Bento007 Sep 13, 2024
af75836
move mdoc_name to collection_metadata
Bento007 Sep 13, 2024
125fce7
extract txt and csv to from rawtilts to alignment
Bento007 Sep 14, 2024
f91a1e7
add correct dates
Bento007 Sep 16, 2024
a5bfe46
added is_portal_standard
Bento007 Sep 16, 2024
32a1523
undo v2.0 changes
Bento007 Sep 16, 2024
f978bca
Apply suggestions from code review
Bento007 Sep 16, 2024
6244d11
add affinity matrix back to tomograms
Bento007 Sep 16, 2024
b8a8514
add affinity matrix back to tomograms
Bento007 Sep 16, 2024
2e58482
add identity matrix back in
Bento007 Sep 21, 2024
465893c
remove voxel spacing from parent filter of alignment
Bento007 Sep 21, 2024
54c811f
lock the verison of linkml and regenerate files
Bento007 Sep 21, 2024
90734f1
add xtlt and copy to transform_ingestion_configs.py
Bento007 Sep 23, 2024
a3394fb
lint
Bento007 Sep 23, 2024
e2072d8
iron out tilt formats
Bento007 Sep 24, 2024
2096a74
iron out tilt formats
Bento007 Sep 24, 2024
4b627a2
update schema
Bento007 Sep 24, 2024
05c4769
Merge branch 'refs/heads/main' into tsmith/997-ctf-alignment
Bento007 Sep 24, 2024
61c75e9
merge main
Bento007 Sep 24, 2024
ffafd79
update transform_ingestion_configs.py
Bento007 Sep 24, 2024
326f227
flatten code in ingest_997.py add dosctring to transform_ingestion_co…
Bento007 Sep 25, 2024
4048d79
remove tomogram from alignment in ingest-config
Bento007 Sep 26, 2024
10fac28
Merge branch 'refs/heads/main' into tsmith/997-ctf-alignment
Bento007 Sep 26, 2024
dded1d1
move migrate script to ingest-tools
Bento007 Sep 26, 2024
918ed8e
lint
Bento007 Sep 26, 2024
6912e06
update configs
Bento007 Sep 26, 2024
15f0309
liknt
Bento007 Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
42 changes: 24 additions & 18 deletions ingestion_tools/dataset_configs/10000.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
alignments:
- metadata:
format: IMOD
sources:
- source_multi_glob:
list_globs:
- metadata/{run_name}.xf
annotations:
- metadata:
annotation_method: Cumulative template-matching trained 2D CNN predictions + visual
Expand Down Expand Up @@ -51,6 +58,7 @@ annotations:
columns: xyz
file_format: csv
glob_string: particle_lists/{run_name}_fas.csv
is_portal_standard: false
is_visualization_default: false
- metadata:
annotation_method: Cumulative template-matching trained 2D CNN predictions + visual
Expand All @@ -72,6 +80,7 @@ annotations:
columns: xyz
file_format: csv
glob_string: particle_lists/{run_name}_cyto_ribosomes.csv
is_portal_standard: false
is_visualization_default: true
- metadata:
annotation_method: spectrum equalization filter + 2D CNN prediction + manual correction
Expand All @@ -91,6 +100,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 1
- metadata:
Expand All @@ -111,6 +121,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false
is_visualization_default: true
mask_label: 2
- metadata:
Expand All @@ -131,6 +142,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 3
- metadata:
Expand All @@ -151,6 +163,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false
is_visualization_default: true
mask_label: 4
- metadata:
Expand All @@ -171,6 +184,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false
is_visualization_default: true
mask_label: 5
- metadata:
Expand All @@ -191,6 +205,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false
is_visualization_default: true
mask_label: 6
- metadata:
Expand All @@ -211,6 +226,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 7
- metadata:
Expand All @@ -231,6 +247,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 8
- metadata:
Expand All @@ -251,6 +268,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 9
- metadata:
Expand All @@ -271,6 +289,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false
is_visualization_default: true
mask_label: 10
- metadata:
Expand All @@ -291,6 +310,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 11
- metadata:
Expand All @@ -311,6 +331,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/180426_{mapped_segmask_name}.labels.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 13
- metadata:
Expand All @@ -331,6 +352,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_membranes.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 1
collection_metadata:
Expand Down Expand Up @@ -443,7 +465,6 @@ rawtilts:
- source_multi_glob:
list_globs:
- metadata/{run_name}.rawtlt
- metadata/{run_name}.xf
runs:
- sources:
- source_glob:
Expand Down Expand Up @@ -485,26 +506,11 @@ tiltseries:
match_regex: (.*)\.st
tomograms:
- metadata:
affine_transformation_matrix:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we are retaining this for now, to ensure backward compatibility.

Copy link
Contributor Author

@Bento007 Bento007 Sep 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added them back in

- - 1
- 0
- 0
- 0
- - 0
- 1
- 0
- 0
- - 0
- 0
- 1
- 0
- - 0
- 0
- 0
- 1
authors: *id003
ctf_corrected: false
dates: *id002
fiducial_alignment_status: NON_FIDUCIAL
is_visualization_default: true
offset:
x: 0
y: 0
Expand Down
45 changes: 27 additions & 18 deletions ingestion_tools/dataset_configs/10001.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
alignments:
- metadata:
format: IMOD
sources:
- source_multi_glob:
list_globs:
- metadata/{run_name}.xf
annotations:
- metadata:
annotation_method: Cumulative template-matching trained 2D CNN predictions + visual
Expand Down Expand Up @@ -51,6 +58,7 @@ annotations:
columns: xyz
file_format: csv
glob_string: particle_lists/{run_name}_fas.csv
is_portal_standard: false
is_visualization_default: false
- metadata:
annotation_method: Cumulative template-matching trained 2D CNN predictions + visual
Expand All @@ -72,6 +80,7 @@ annotations:
columns: xyz
file_format: csv
glob_string: particle_lists/{run_name}_cyto_ribosomes.csv
is_portal_standard: false
is_visualization_default: true
- metadata:
annotation_method: manual segmentation
Expand All @@ -91,6 +100,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 1
- metadata:
Expand All @@ -111,6 +121,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false
is_visualization_default: true
mask_label: 2
- metadata:
Expand All @@ -131,6 +142,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 3
- metadata:
Expand All @@ -151,6 +163,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false
is_visualization_default: true
mask_label: 4
- metadata:
Expand All @@ -171,6 +184,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false
is_visualization_default: true
mask_label: 5
- metadata:
Expand All @@ -191,6 +205,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false
is_visualization_default: true
mask_label: 6
- metadata:
Expand All @@ -211,6 +226,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 7
- metadata:
Expand All @@ -231,6 +247,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 8
- metadata:
Expand All @@ -251,6 +268,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 9
- metadata:
Expand All @@ -271,6 +289,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false
is_visualization_default: true
mask_label: 10
- metadata:
Expand All @@ -291,6 +310,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 11
- metadata:
Expand All @@ -311,6 +331,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_organelles.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 13
- metadata:
Expand All @@ -331,6 +352,7 @@ annotations:
- SemanticSegmentationMask:
file_format: mrc
glob_string: labels/{run_name}_membranes.mrc
is_portal_standard: false
is_visualization_default: false
mask_label: 1
collection_metadata:
Expand Down Expand Up @@ -427,7 +449,6 @@ rawtilts:
- source_multi_glob:
list_globs:
- metadata/{run_name}.rawtlt
- metadata/{run_name}.xf
runs:
- sources:
- source_glob:
Expand Down Expand Up @@ -469,26 +490,14 @@ tiltseries:
match_regex: (.*)\.st
tomograms:
- metadata:
affine_transformation_matrix:
- - 1
- 0
- 0
- 0
- - 0
- 1
- 0
- 0
- - 0
- 0
- 1
- 0
- - 0
- 0
- 0
- 1
authors: *id003
ctf_corrected: false
dates:
deposition_date: '2023-04-01'
last_modified_date: '2023-06-01'
release_date: '2023-06-01'
fiducial_alignment_status: NON_FIDUCIAL
is_visualization_default: true
offset:
x: 0
y: 0
Expand Down
Loading
Loading