Merge the Updated Codebase #201

Linardos · 2024-10-13T20:51:12Z

No description provided.

Signed-off-by: kta-intel <[email protected]>

updates to enable latest openfl and gandlf

…ml blueprints

Update plan.yaml

…or classification

gitguardian · 2024-10-13T20:51:49Z

⚠️ GitGuardian has uncovered 50 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
-	-	RSA Private Key	`5ae77c9`	Task_1/cert/client/col_two.key	View secret
-	-	RSA Private Key	`3ba75b2`	Task_1/cert/ca/signing-ca/private/signing-ca.key	View secret
-	-	RSA Private Key	`3ba75b2`	Task_1/cert/ca/root-ca/private/root-ca.key	View secret
-	-	RSA Private Key	`295c046`	Task_1/cert/server/agg_in-ota-232347.ads.iu.edu.key	View secret
-	-	RSA Private Key	`6414e2a`	Task_1/cert/server/agg_in-ota-232347.ads.iu.edu.key	View secret
-	-	RSA Private Key	`fbef799`	Task_1/cert/ca/root-ca/private/root-ca.key	View secret
-	-	RSA Private Key	`5763b01`	Task_1/cert/ca/root-ca/private/root-ca.key	View secret
-	-	RSA Private Key	`295c046`	Task_1/cert/ca/signing-ca/private/signing-ca.key	View secret
-	-	RSA Private Key	`6414e2a`	Task_1/cert/ca/root-ca/private/root-ca.key	View secret
-	-	RSA Private Key	`5763b01`	Task_1/cert/ca/signing-ca/private/signing-ca.key	View secret
-	-	RSA Private Key	`a94d2b4`	Task_1/cert/ca/signing-ca/private/signing-ca.key	View secret
-	-	RSA Private Key	`5763b01`	Task_1/cert/client/col_one.key	View secret
-	-	RSA Private Key	`7f238e7`	Task_1/cert/ca/signing-ca/private/signing-ca.key	View secret
-	-	RSA Private Key	`a94d2b4`	Task_1/cert/server/agg_in-ota-232347.ads.iu.edu.key	View secret
-	-	RSA Private Key	`295c046`	Task_1/cert/ca/root-ca/private/root-ca.key	View secret
-	-	RSA Private Key	`a94d2b4`	Task_1/cert/client/col_one.key	View secret
-	-	RSA Private Key	`5763b01`	Task_1/cert/client/col_two.key	View secret
-	-	RSA Private Key	`5ae77c9`	Task_1/cert/ca/root-ca/private/root-ca.key	View secret
-	-	RSA Private Key	`a94d2b4`	Task_1/cert/client/col_two.key	View secret
-	-	RSA Private Key	`6414e2a`	Task_1/cert/client/col_two.key	View secret
-	-	RSA Private Key	`9d932a8`	Task_1/cert/ca/root-ca/private/root-ca.key	View secret
-	-	RSA Private Key	`6db76ad`	Task_1/cert/client/col_one.key	View secret
-	-	RSA Private Key	`7f238e7`	Task_1/cert/client/col_two.key	View secret
-	-	RSA Private Key	`9d932a8`	Task_1/cert/ca/signing-ca/private/signing-ca.key	View secret
-	-	RSA Private Key	`5ae77c9`	Task_1/cert/ca/signing-ca/private/signing-ca.key	View secret
-	-	RSA Private Key	`fbef799`	Task_1/cert/ca/signing-ca/private/signing-ca.key	View secret
-	-	RSA Private Key	`7f238e7`	Task_1/cert/client/col_one.key	View secret
-	-	RSA Private Key	`5763b01`	Task_1/cert/server/agg_in-ota-232347.ads.iu.edu.key	View secret
-	-	RSA Private Key	`295c046`	Task_1/cert/client/col_one.key	View secret
-	-	RSA Private Key	`6db76ad`	Task_1/cert/client/col_two.key	View secret
-	-	RSA Private Key	`9d932a8`	Task_1/cert/client/col_one.key	View secret
-	-	RSA Private Key	`3ba75b2`	Task_1/cert/client/col_two.key	View secret
-	-	RSA Private Key	`fbef799`	Task_1/cert/client/col_one.key	View secret
-	-	RSA Private Key	`6db76ad`	Task_1/cert/server/agg_in-ota-232347.ads.iu.edu.key	View secret
-	-	RSA Private Key	`9d932a8`	Task_1/cert/client/col_two.key	View secret
-	-	RSA Private Key	`fbef799`	Task_1/cert/server/agg_in-ota-232347.ads.iu.edu.key	View secret
-	-	RSA Private Key	`295c046`	Task_1/cert/client/col_two.key	View secret
-	-	RSA Private Key	`5ae77c9`	Task_1/cert/client/col_one.key	View secret
-	-	RSA Private Key	`3ba75b2`	Task_1/cert/client/col_one.key	View secret
-	-	RSA Private Key	`7f238e7`	Task_1/cert/server/agg_in-ota-232347.ads.iu.edu.key	View secret
-	-	RSA Private Key	`6db76ad`	Task_1/cert/ca/signing-ca/private/signing-ca.key	View secret
-	-	RSA Private Key	`a94d2b4`	Task_1/cert/ca/root-ca/private/root-ca.key	View secret
-	-	RSA Private Key	`6414e2a`	Task_1/cert/client/col_one.key	View secret
-	-	RSA Private Key	`5ae77c9`	Task_1/cert/server/agg_in-ota-232347.ads.iu.edu.key	View secret
-	-	RSA Private Key	`3ba75b2`	Task_1/cert/server/agg_in-ota-232347.ads.iu.edu.key	View secret
-	-	RSA Private Key	`9d932a8`	Task_1/cert/server/agg_in-ota-232347.ads.iu.edu.key	View secret
-	-	RSA Private Key	`fbef799`	Task_1/cert/client/col_two.key	View secret
-	-	RSA Private Key	`6db76ad`	Task_1/cert/ca/root-ca/private/root-ca.key	View secret
-	-	RSA Private Key	`7f238e7`	Task_1/cert/ca/root-ca/private/root-ca.key	View secret
-	-	RSA Private Key	`6414e2a`	Task_1/cert/ca/signing-ca/private/signing-ca.key	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

sarthakpati

@psfoley do you see any major changes preventing us from merging this ASAP?

Task_2/README.md

Task_1/fets_challenge.egg-info/PKG-INFO

psfoley · 2024-10-15T23:54:30Z

@psfoley do you see any major changes preventing us from merging this ASAP?

@sarthakpati Nope - I would expect this can be merged without issue.

sarthakpati · 2024-10-16T13:31:04Z

@psfoley do you see any major changes preventing us from merging this ASAP?

@sarthakpati Nope - I would expect this can be merged without issue.

Cool, thanks for the confirmation.

@Linardos: thank you once again for the herculean effort that you and @kta-intel have put together to modernize this codebase. Once you fix the comments that I have, I can start putting together some CI/CD stuff to make our lives much easier.

Linardos · 2024-11-06T20:23:40Z

@sarthakpati I addressed your comments. You may merge. Note that a version of generate_predictions for classification (i.e. inference for classification) still needs work to function though. I can do a separate PR for that when I have it

sarthakpati · 2024-11-07T18:10:14Z

There are some unresolved conversations, and I am unable to merge until they are resolved. Also, does this PR contain #198 as well?

Linardos · 2024-11-07T18:29:18Z

Seems #198 is an outdated part of the PR that I have since then removed because csv files are generated on-the-go. I resolved the conversations. I also made the code default to segmentation which is functional through both training and inference. Classification will still need some work in terms of inferring the metrics, but we can do that in a later PR.

.gitignore

Task_1/FeTS_Challenge.py

Task_1/README.md

sarthakpati · 2024-11-08T01:26:11Z

Task_1/setup.py

@@ -28,11 +28,11 @@
    ],
    include_package_data=True,
    install_requires=[
-        'openfl @ git+https://github.com/intel/openfl.git@f4b28d710e2be31cdfa7487fdb4e8cb3a1387a5f',
-        'GANDLF @ git+https://github.com/CBICA/GaNDLF.git@e4d0d4bfdf4076130817001a98dfb90189956278',
+        'openfl @ git+https://github.com/securefederatedai/openfl.git@kta-intel/fets-2024-patch-1',


This seems to be problematic:

Is there any plan to merge this with openfl's develop branch so that we can use that instead? This seems prone to breaking, right? Tagging @kta-intel for guidance.

I just opened a PR for it: securefederatedai/openfl#1141
Once it is merged, we can delete that branch and update the setup.py to use the develop branch directly

Couple notes:

I moved the conditional into loader_gandlf.py instead of runner_gandalf.py since it is logically a better spot to check if we are running in inference mode, rather than setting another. Simple move that tests fine on my end, but be mindful that it doesn't affect anything else on your end

The native API (i.e. import openfl.native as fx) will likely be deprecated at some point in this or an upcoming release, so it might be a good idea to pin to this specific openfl commit rather than develop as a whole

Point 2 is very important. Are there plans to preserve backward compatibility for a few versions?

It has been merged

How to cleanly handle Point 2 is still being discussed. Considering the importance of the FeTS challenge, let me open up the discussion and propose a way to preserve it at least for v1.7

propose a way to preserve it at least for v1.7

This would be a good solution. Ideal would be for this API to be supported (with a deprecation warning) until at least 1.9 (considering the conventions of semantic versioning).

sarthakpati · 2024-11-08T01:27:29Z

Task_1/setup.py

-        'openfl @ git+https://github.com/intel/openfl.git@f4b28d710e2be31cdfa7487fdb4e8cb3a1387a5f',
-        'GANDLF @ git+https://github.com/CBICA/GaNDLF.git@e4d0d4bfdf4076130817001a98dfb90189956278',
+        'openfl @ git+https://github.com/securefederatedai/openfl.git@kta-intel/fets-2024-patch-1',
+        'GANDLF @ git+https://github.com/CBICA/GaNDLF.git@0.1.0',


Suggested change

'GANDLF @ git+https://github.com/CBICA/[email protected]',

'GANDLF @ git+https://github.com/mlcommons/[email protected]',

Can we open an issue where we add a CI run on both GaNDLF and OpenFL master/develop branches respectively?

sarthakpati · 2024-11-08T01:28:59Z

Task_1/utils/inspect_pickled_model.py

+import pickle
+
+# Path to the pickle file
+pickle_file_path = '/home/locolinux2/.local/workspace/checkpoint/experiment_109/best_model.pkl'


This path (and anything similar) is going to be problematic. I will suggest using something that defined based on pathlib.Path.cwd() (or something similar) to ensure non-breakage.

sarthakpati · 2024-11-08T01:29:51Z

Task_1/FeTS_Challenge.py

+institution_split_csv_filename = '/home/locolinux2/datasets/MICCAI_FeTS2022_TrainingData/partitioning_2.csv' 
+institution_split_csv_filename = '/home/locolinux2/datasets/MICCAI_FeTS2022_TrainingData/sanity_partitioning.csv' # a small subset for sanity checks and debugging. Comment out to run the actual challenge partition.

 # change this to point to the parent directory of the data
-brats_training_data_parent_dir = '/raid/datasets/FeTS22/MICCAI_FeTS2022_TrainingData'
+brats_training_data_parent_dir = '/home/locolinux2/datasets/MICCAI_FeTS2022_TrainingData'
+brats_training_data_parent_dir = '/home/locolinux2/datasets/MICCAI_FeTS2022_Resized'


These paths (and anything similar) is going to be problematic. I will suggest using something that defined based on pathlib.Path.cwd() (or something similar) to ensure non-breakage.

sarthakpati · 2024-11-08T01:30:47Z

Task_2/README.md

- User inside singularity containers isn't root: This can lead to `PermissionError` when reading files from the file system like model checkpoints. Make sure that all files that need to be read from inside the container can be read by *all users*, either before copying them in the Dockerfile or adding chmod commands to the Dockerfile.
-
-Any other Errors ? Feel free to contact us: [forum](https://www.synapse.org/#!Synapse:syn28546456/discussion/default)
+Task 2 is still available on this link: https://github.com/FeTS-AI/Challenge/tree/2022/Task_2


Suggested change

Task 2 is still available on this link: https://github.com/FeTS-AI/Challenge/tree/2022/Task_2

Information related to Task 2 of the FeTS Challenge is available on this link: https://github.com/FeTS-AI/Challenge/tree/2022/Task_2

sarthakpati · 2024-11-08T01:32:10Z

Hey @Linardos, apologies for the late review (as you know, I have been swamped). Anyway, great work on all the changes. I am requesting a few minor points of consideration before we can merge this to be used by others. Thanks!

Co-authored-by: Sarthak Pati <[email protected]>

sarthakpati · 2024-11-13T15:30:41Z

@Linardos since the PR from @kta-intel has been merged (securefederatedai/openfl#1141), can you please update the setup appropriately?

Linardos and others added 16 commits September 3, 2024 12:56

made some updates. Stuck at OpenFL-relevant error.

6db76ad

kevin changes

ba5b35b

GaNDLF pinned to 0.0.17

295c046

GaNDLF pinned to 0.0.17

3ba75b2

.

fbef799

removed Task 2

5ae77c9

updates to enable latest openfl and gandlf

960fe35

Signed-off-by: kta-intel <[email protected]>

Merge pull request #2 from kta-intel/kta-intel/patch-1

4b62795

updates to enable latest openfl and gandlf

added classification plan, made separate plans folder for multiple ya…

7f238e7

…ml blueprints

Update plan.yaml

1818d33

Merge pull request #3 from Linardos/sarthakpati-patch-2

303ab5d

Update plan.yaml

simplified config

5763b01

added extract_csv for classification - i.e. generating of csv files f…

6414e2a

…or classification

added extract_csv for classification - i.e. generating of csv files f…

5b231b8

…or classification

functional classification

9d932a8

classification works

a94d2b4

sarthakpati reviewed Oct 15, 2024

View reviewed changes

Task_2/README.md Show resolved Hide resolved

sarthakpati reviewed Oct 15, 2024

View reviewed changes

Task_1/fets_challenge.egg-info/PKG-INFO Outdated Show resolved Hide resolved

Linardos added 2 commits October 31, 2024 23:22

generate_Preds for classification is functional

e0f0891

removed egg-info

368029f

default to segmentation

558b398

sarthakpati mentioned this pull request Nov 8, 2024

Add files via upload #198

Closed

sarthakpati requested changes Nov 8, 2024

View reviewed changes

Linardos and others added 3 commits November 8, 2024 16:14

Update Task_1/README.md

999a051

Co-authored-by: Sarthak Pati <[email protected]>

committed gitignore

9c8e985

Merge branch 'main' of github.com:Linardos/Challenge

8a03f37

kta-intel mentioned this pull request Nov 12, 2024

[FeTS][GaNDLF] skip loading train.csv during inference mode securefederatedai/openfl#1141

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge the Updated Codebase #201

Merge the Updated Codebase #201

Linardos commented Oct 13, 2024

gitguardian bot commented Oct 13, 2024 •

edited

Loading

sarthakpati left a comment

psfoley commented Oct 15, 2024

sarthakpati commented Oct 16, 2024

Linardos commented Nov 6, 2024

sarthakpati commented Nov 7, 2024

Linardos commented Nov 7, 2024

sarthakpati Nov 8, 2024

kta-intel Nov 12, 2024

sarthakpati Nov 12, 2024

kta-intel Nov 12, 2024 •

edited

Loading

sarthakpati Nov 13, 2024 •

edited

Loading

sarthakpati Nov 8, 2024

sarthakpati Nov 8, 2024

sarthakpati Nov 8, 2024

sarthakpati Nov 8, 2024

sarthakpati commented Nov 8, 2024

sarthakpati commented Nov 13, 2024

	'GANDLF @ git+https://github.com/CBICA/[email protected]',
	'GANDLF @ git+https://github.com/mlcommons/[email protected]',

	Task 2 is still available on this link: https://github.com/FeTS-AI/Challenge/tree/2022/Task_2
	Information related to Task 2 of the FeTS Challenge is available on this link: https://github.com/FeTS-AI/Challenge/tree/2022/Task_2

Merge the Updated Codebase #201

Are you sure you want to change the base?

Merge the Updated Codebase #201

Conversation

Linardos commented Oct 13, 2024

gitguardian bot commented Oct 13, 2024 • edited Loading

⚠️ GitGuardian has uncovered 50 secrets following the scan of your pull request.

sarthakpati left a comment

Choose a reason for hiding this comment

psfoley commented Oct 15, 2024

sarthakpati commented Oct 16, 2024

Linardos commented Nov 6, 2024

sarthakpati commented Nov 7, 2024

Linardos commented Nov 7, 2024

sarthakpati Nov 8, 2024

Choose a reason for hiding this comment

kta-intel Nov 12, 2024

Choose a reason for hiding this comment

sarthakpati Nov 12, 2024

Choose a reason for hiding this comment

kta-intel Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

sarthakpati Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

sarthakpati Nov 8, 2024

Choose a reason for hiding this comment

sarthakpati Nov 8, 2024

Choose a reason for hiding this comment

sarthakpati Nov 8, 2024

Choose a reason for hiding this comment

sarthakpati Nov 8, 2024

Choose a reason for hiding this comment

sarthakpati commented Nov 8, 2024

sarthakpati commented Nov 13, 2024

gitguardian bot commented Oct 13, 2024 •

edited

Loading

kta-intel Nov 12, 2024 •

edited

Loading

sarthakpati Nov 13, 2024 •

edited

Loading