Copernicus marine data retrieving tool #126
Conversation
Not totally fixed but we're getting there
Co-authored-by: Björn Grüning <[email protected]>
Co-authored-by: Björn Grüning <[email protected]>
|
Oh it went wrong really quickly .... It's an issue with the .shed.yml ? |
tools/ocean/copernicusmarine.xml
Outdated
| echo 'Error. Set your CMEMS credentials via: User -> Preferences -> Manage Information' && | ||
| #end if | ||
|
|
||
| eval '$input_text' --force-download --username $cmems_username --password $cmems_password |
There was a problem hiding this comment.
This is super dangerous, people can put it other commands that gets execute and potentially remove your harddrive I assume.
Why not doing a classical tool with copernicusmarine subset
There was a problem hiding this comment.
uhoh ...
Yes this is totally possible but less easy for user. I was trying to reproduce the same thing than Anne here https://usegalaxy.eu/root?tool_id=toolshed.g2.bx.psu.edu/repos/climate/cads/cads/0.1.0
But wihtout the file option because the CMEMS website does not offer the file option
There was a problem hiding this comment.
Would something like that
#set $command = $input_text.replace("copernicusmarine ", "", 1)
eval copernicusmarine $command --force-download --username $cmems_username --password $cmems_password
be better ?
Like that the user can't do anything except a copernicusmarine command no ?
There was a problem hiding this comment.
You are correct. Its also a bit scary in this tool as well. But at least its using literal_eval https://github.com/NordicESMhub/galaxy-tools/blob/de74da48cc04c0cb261d07c451d00598beb834d9/tools/cads/cads_retrieve.py#L37C20-L37C32 and and does some other stripping.
Do you think its feasible to make is less "give me random code and I will execute it"?
There was a problem hiding this comment.
What can this "$command" be. How does it usually look like?
There was a problem hiding this comment.
The $command would be something like that for instance
subset --dataset-id cmems_mod_glo_phy-cur_anfc_0.083deg_P1D-m --variable uo --variable vo --start-datetime 2024-09-15T00:00:00 --end-datetime 2024-09-15T00:00:00 --minimum-longitude -180 --maximum-longitude 179.91668701171875 --minimum-latitude -80 --maximum-latitude 90 --minimum-depth 0.49402499198913574 --maximum-depth 0.49402499198913574
| <data name="output_netcdf" label="Copernicus marine data" from_work_dir="./*.nc" format="netcdf"/> | ||
| </outputs> | ||
| <tests> | ||
| <test expect_num_outputs="1"> |
There was a problem hiding this comment.
We could test this tools with ENV variables, that we define here in GA. Means we can store your credentials in Github and use them.
The other option is that we tell Galaxy that we know this test is failing and then we check for a specific error message. If this error message pops up the test passed.
There was a problem hiding this comment.
Not sure I fully understand how to do that but I think the 1st option might be best ?
There was a problem hiding this comment.
No, indeed it's not a suite. But it's all on the same thematic. If I change folders i'll need to close and open a new PR ? we finish here and then I do that ? |
|
So I put my secrets as github variables and it does not seem to work so I am guessing I did not do it right ? |
| return False, message | ||
| else: | ||
| return False, "Error: Command must be 'subset' or 'get'" | ||
|
|
There was a problem hiding this comment.
What if a user puts in any of --force-download --username --password ...
There was a problem hiding this comment.
Ah let's try I'll see and add the correct messages
There was a problem hiding this comment.
Just tried this is no issue if the user writes down this too it works fine admittedly that the user wrote his credentials in the environment. if not It says the password or username is invalid. Is this enough ?
|
@bgruening Anything else I can improve ? |
|
but tests are failing ... |
|
Yes thts' because I don't know how to correctly put my secret I think no ? I tried but it's not sucessfull |
How have you done it? You need to define two variables in your repo: Settings --> Environments |
I think that's what I missing the adding in the github workflow Actually @bgruening did you add into the settings of this repo ? because I can add for my repo my credentials but I don't have the right here ... or @yvanlebras can you check ? |
|
Ok. Someone having admin of this repo needs to set the environment variables (but since you are using your own credentials, it would be better to give you admin access; I cannot do it on this repo). For the workflows, I think you need to add the variables in the env:
GALAXY_FORK: galaxyproject
GALAXY_BRANCH: release_23.2
MAX_CHUNKS: 40
CMEMS_PASSWORD: ${{ secrets.CMEMS_PASSWORD }}
CMEMS_USERNAME: ${{ secrets.CMEMS_USERNAME }} |
Hi @bgruening and @yvanlebras !!
I just finished trying out this new tool to directly retrieve data from the Copernicus Marine Data Store (that goes along @annefou works on atmosphere and climate stores)
I tested it with my own credentials it works fine. Now, I don't know how to test it by setting up the credentials in the user > preferences > manage information part through planemo.
I tried to reuse both Anne's work and the work that was done on another copernicus marine tool medenv (I am not sure if the configfile part is necessary here I kept it from the medenv tool)
Let me know what you think of it !!
Thanks