Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dp sf pap #547

Merged
merged 52 commits into from
Jan 12, 2024
Merged
Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
a724d2d
use both _1 and _2 segmentations
lucidtronix Sep 5, 2023
7d0fa9c
use both _1 and _2 segmentations
lucidtronix Sep 5, 2023
21d95bd
TEMP: add dice metrics, copied from neuron
daniellepace Sep 14, 2023
f5d90ff
ENH: Use dice loss and metrics
daniellepace Sep 14, 2023
814f68f
ENH: Remove kidney label and merge body/background labels
daniellepace Sep 14, 2023
aa36994
FIX: Fix bad number of channels
daniellepace Sep 14, 2023
6d40407
ENH: Use only one channel of the input image
daniellepace Sep 14, 2023
5005d62
WIP: hacking bottom of U-net
daniellepace Sep 21, 2023
d91c7f6
ENH: Add mean and std for normalization
daniellepace Sep 22, 2023
fa17ab8
Merge branch 'master' into dp_sf_pap
daniellepace Sep 26, 2023
3f1f27b
ENH: Add neurite and voxelmorph to docker
daniellepace Sep 26, 2023
e173f10
FIX: Use dice loss from neurite
daniellepace Sep 26, 2023
ab321bb
STYLE: Fix up WIP code on hacking bottom of U-net
daniellepace Sep 26, 2023
9e2d864
ENH: Add merged paps for segmentation tensormap
daniellepace Sep 29, 2023
78dd55d
WIP: Fix Unet concats
daniellepace Sep 29, 2023
33248c3
FIX: Fix soft dice metrics
daniellepace Oct 5, 2023
196e050
ENH: Add plot_dice to compare
daniellepace Oct 6, 2023
6051b8d
ENH: Add median computation for papillary segmentation project
daniellepace Oct 13, 2023
598f4c6
FIX: Fix double plot on one graph
daniellepace Oct 13, 2023
3e8e2d7
ENH: Allow generator to have empty path, e.g., to test on all images
daniellepace Oct 13, 2023
f65a8f4
ENH: Prune list of structures for which we do stats
daniellepace Oct 13, 2023
d012a13
STYLE: rearranging
daniellepace Oct 13, 2023
fd7c0cf
WIP: Handle inference without ground truth labels
daniellepace Oct 13, 2023
989c680
ENH: Remove option for merged paps
daniellepace Oct 16, 2023
83e5dc6
FIX: Get all b2s images, instance_2s only
daniellepace Oct 17, 2023
75b0446
ENH: Add mri dates
daniellepace Oct 17, 2023
e54d4af
FIX: Fix normalization with correct padding
daniellepace Oct 20, 2023
ddb9d46
FIX: Fix soft dice metrics again
daniellepace Oct 20, 2023
84c5365
COMP: Add option for environment variable for jupyter notebooks
daniellepace Oct 20, 2023
01bb4f5
WIP: data augmentation
daniellepace Oct 20, 2023
ef92261
WIP: Better scatter plots for medians
daniellepace Oct 24, 2023
79e87a6
ENH: Report std too
daniellepace Oct 25, 2023
b87407c
ENH: Improve dice plots for a single model
daniellepace Oct 25, 2023
d2449a9
ENH: Log pearson correlation coefficients
daniellepace Oct 25, 2023
081b5da
STYLE: Adding TODOs to fix tensor_generators
daniellepace Nov 1, 2023
7b0643d
WIP: Add temporary code to save Dice scores
daniellepace Nov 1, 2023
e92f133
WIP: Add temporary code for plotting medians
daniellepace Nov 1, 2023
18eed01
STYLE: Clean up code for infer_medians
daniellepace Nov 1, 2023
9fa3897
STYLE: Clean up medians code
daniellepace Nov 3, 2023
0bf07bc
STYLE: Add command-line args for median computations
daniellepace Nov 3, 2023
ebcaa9c
ENH: Add percentiles and tsv for dice calculations
daniellepace Nov 3, 2023
94902da
STYLE: Add command-line args for data augmentation
daniellepace Nov 3, 2023
8d8f6b0
ENH: Improve log files for dice compare
daniellepace Nov 9, 2023
3967219
STYLE: Small edits requested in PR
daniellepace Nov 14, 2023
86a9803
STYLE: docstring and typehints for plot_dice
daniellepace Nov 14, 2023
364574a
STYLE: docstring for infer_statistics_from_segmented_regions
daniellepace Nov 14, 2023
0d2eb5f
STYLE: Docstring and typehints for augment_using_layers
daniellepace Nov 14, 2023
c64275e
FIX: Fix typo
daniellepace Dec 8, 2023
ba4e5fd
FIX: Fix parser for boolean arguments
daniellepace Dec 8, 2023
55a58b0
STYLE: Rename _unit_disk(r) to unit_disk(r)
daniellepace Jan 12, 2024
618d857
ENH: Remove --no_analyze_ground_truth option
daniellepace Jan 12, 2024
b15ef4d
Merge branch 'master' into dp_sf_pap
daniellepace Jan 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docker/vm_boot_images/config/tensorflow-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,7 @@ boto3
ml4ht==0.0.10
google-cloud-storage
umap-learn[plot]
neurite
voxelmorph
pystrum

20 changes: 19 additions & 1 deletion ml4h/arguments.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ def parse_args():
parser.add_argument('--dicoms', default='./dicoms/', help='Path to folder of dicoms.')
parser.add_argument('--sample_csv', default=None, help='Path to CSV with Sample IDs to restrict tensor paths')
parser.add_argument('--tsv_style', default='standard', choices=['standard', 'genetics'], help='Format choice for the TSV file produced in output by infer and explore modes.')
parser.add_argument('--app_csv', help='Path to file used to link sample IDs between UKBB applications 17488 and 7089')
parser.add_argument('--app_csv', help='Path to file used by the recipe')
parser.add_argument('--tensors', help='Path to folder containing tensors, or where tensors will be written.')
parser.add_argument('--output_folder', default='./recipes_output/', help='Path to output folder for recipes.py runs.')
parser.add_argument('--model_file', help='Path to a saved model architecture and weights (hd5).')
Expand Down Expand Up @@ -189,6 +189,9 @@ def parse_args():
parser.add_argument('--pool_z', default=1, type=int, help='Pooling size in the z-axis, if 1 no pooling will be performed.')
parser.add_argument('--padding', default='same', help='Valid or same border padding on the convolutional layers.')
parser.add_argument('--dense_blocks', nargs='*', default=[32, 32, 32], type=int, help='List of number of kernels in convolutional layers.')
parser.add_argument('--merge_dimension', default=3, type=int, help='Dimension of the merge layer.')
parser.add_argument('--merge_dense_blocks', nargs='*', default=[32], type=int, help='List of number of kernels in convolutional merge layer.')
parser.add_argument('--decoder_dense_blocks', nargs='*', default=[32, 32, 32], type=int, help='List of number of kernels in convolutional decoder layers.')
parser.add_argument('--encoder_blocks', nargs='*', default=['conv_encode'], help='List of encoding blocks.')
parser.add_argument('--merge_blocks', nargs='*', default=['concat'], help='List of merge blocks.')
parser.add_argument('--decoder_blocks', nargs='*', default=['conv_decode', 'dense_decode'], help='List of decoding blocks.')
Expand Down Expand Up @@ -269,6 +272,11 @@ def parse_args():
help='If true saves the model weights from the last training epoch, otherwise the model with best validation loss is saved.',
)

# 2D image data augmentation parameters
parser.add_argument('--rotation_factor', default=0., type=float, help='for data augmentation, a float represented as fraction of 2 Pi, e.g., rotation_factor = 0.014 results in an output rotated by a random amount in the range [-5 degrees, 5 degrees]')
parser.add_argument('--zoom_factor', default=0., type=float, help='for data augmentation, a float represented as fraction of value, e.g., zoom_factor = 0.05 results in an output zoomed in a random amount in the range [-5%, 5%]')
parser.add_argument('--translation_factor', default=0., type=float, help='for data augmentation, a float represented as a fraction of value, e.g., translation_factor = 0.05 results in an output shifted by a random amount in the range [-5%, 5%] in the x- and y- directions')

# Run specific and debugging arguments
parser.add_argument('--id', default='no_id', help='Identifier for this run, user-defined string to keep experiments organized.')
parser.add_argument('--random_seed', default=12878, type=int, help='Random seed to use throughout run. Always use np.random.')
Expand Down Expand Up @@ -376,6 +384,16 @@ def parse_args():
default='3M',
)

# Arguments for explorations/infer_stats_from_segmented_regions
parser.add_argument('--analyze_ground_truth', action='store_true', help='Filter by images with ground truth segmentations, for comparison')
parser.add_argument('--no_analyze_ground_truth', dest='analyze_ground_truth', action='store_false', help='Do not filter by images with ground truth segmentations, for comparison')
parser.set_defaults(analyze_ground_truth=True)
parser.add_argument('--structures_to_analyze', nargs='*', default=[], help='Structure names to include in the .tsv files and scatter plots')
parser.add_argument('--erosion_radius', default=1, type=int, help='Radius of the unit disk structuring element for erosion preprocessing')
parser.add_argument('--intensity_thresh', type=float, help='Threshold value for preprocessing')
parser.add_argument('--intensity_thresh_in_structures', nargs='*', default=[], help='Structure names whose pixels should be replaced if the images has intensity above the threshold')
parser.add_argument('--intensity_thresh_out_structure', help='Replacement structure name')

# TensorMap prefix for convenience
parser.add_argument('--tensormap_prefix', default="ml4h.tensormap", type=str, help="Module prefix path for TensorMaps. Defaults to \"ml4h.tensormap\"")

Expand Down
6 changes: 3 additions & 3 deletions ml4h/defines.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,9 +78,9 @@ def __str__(self):
'interventricular_septum': 5, 'interatrial_septum': 6, 'crista_terminalis': 7,
}
MRI_SAX_PAP_SEGMENTED_CHANNEL_MAP = {
'background': 0, 'body': 1, 'thoracic_cavity': 2, 'liver': 3, 'stomach': 4, 'spleen': 5, 'kidney': 6,
'interventricular_septum': 7, 'LV_free_wall': 8, 'anterolateral_pap': 9, 'posteromedial_pap': 10, 'LV_cavity': 11,
'RV_free_wall': 12, 'RV_cavity': 13,
'background': 0, 'thoracic_cavity': 1, 'liver': 2, 'stomach': 3, 'spleen': 4,
'interventricular_septum': 5, 'LV_free_wall': 6, 'anterolateral_pap': 7, 'posteromedial_pap': 8, 'LV_cavity': 9,
'RV_free_wall': 10, 'RV_cavity': 11,
}
MRI_SAX_SEGMENTED_CHANNEL_MAP = {
'background': 0, 'RV_free_wall': 1, 'interventricular_septum': 2, 'LV_free_wall': 3, 'LV_cavity': 4,
Expand Down
Loading