Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copernicus marine data retrieving tool #126

Closed
wants to merge 60 commits into from
Closed
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
56f91c0
add batch tool diva
Marie59 Jul 15, 2024
f10ecd4
add test files
Marie59 Jul 15, 2024
7cd9998
Set JULIA_DEPOT_PATH to get a writable Julia HOME
bgruening Jul 23, 2024
8924551
fix some test settings more needed
bgruening Jul 23, 2024
0caabeb
fix dates
Marie59 Jul 24, 2024
a8da9eb
Merge branch 'galaxyecology:master' into ocean
Marie59 Jul 31, 2024
877ae31
add test files
Marie59 Jul 31, 2024
656bbc0
fix test & improve tool
Marie59 Jul 31, 2024
36fd13f
fix test
Marie59 Jul 31, 2024
531fb1b
fix typo
Marie59 Jul 31, 2024
ef5b8a2
reduced test files
Marie59 Jul 31, 2024
aea8347
Delete tools/ocean/data_from_Eutrophication_Med_profiles_2022_unrestr…
Marie59 Jul 31, 2024
1f5de95
Delete tools/ocean/DIVAnd_netcdf_output.netcdf
Marie59 Jul 31, 2024
4a8decf
reduced test files in right folder
Marie59 Jul 31, 2024
750b55b
update test
Marie59 Jul 31, 2024
05cbdaa
update test avoid big file
Marie59 Aug 1, 2024
c6ec6cd
Delete tools/ocean/test-data/gebco_30sec_8.nc
Marie59 Aug 1, 2024
7f2e88c
Update tools/ocean/divandfull.xml
Marie59 Aug 1, 2024
e152059
Update tools/ocean/divandfull.xml
Marie59 Aug 1, 2024
b9155de
Delete tools/ocean/test-data/DIVAnd_netcdf_output.netcdf
Marie59 Aug 1, 2024
231217e
cleaning up julia script
Marie59 Aug 1, 2024
a5e2f54
Merge branch 'galaxyecology:master' into ocean
Marie59 Aug 6, 2024
aaa595a
change label output
Marie59 Aug 6, 2024
9877fb3
Update divandfull.xml
Marie59 Aug 6, 2024
f81e272
Update divandfull.xml
Marie59 Aug 6, 2024
e597c1b
Add Copernicus marine data store tool
Marie59 Sep 6, 2024
0eeec82
Update .shed.yml
Marie59 Sep 6, 2024
68119e0
Update tools/ocean/copernicusmarine.xml
bgruening Sep 6, 2024
ea7d27d
rollback divandfull.xml
Marie59 Sep 6, 2024
f7e5c04
add checks to avoid dengerous commandlines
Marie59 Sep 9, 2024
f7a9dc0
fix lint
Marie59 Sep 9, 2024
f3155e9
fix lint
Marie59 Sep 9, 2024
a6757da
add test failure
Marie59 Sep 9, 2024
e64ae35
add credentials
Marie59 Sep 9, 2024
363a69e
Update tools/ocean/copernicusmarine.xml
Marie59 Sep 9, 2024
42b28f7
try to fix lint help
Marie59 Sep 9, 2024
50fd246
fix ugly help
Marie59 Sep 9, 2024
2705dbe
fix with black
Marie59 Sep 19, 2024
56566a5
fix lint
Marie59 Sep 19, 2024
9408fbd
final lint fix !
Marie59 Sep 19, 2024
830d57f
add credentials in env
Marie59 Sep 20, 2024
16aee90
add real test
Marie59 Sep 20, 2024
1927518
oups
Marie59 Sep 20, 2024
10438f2
Update copernicusmarine.xml
Marie59 Oct 1, 2024
8ef41c5
add some more forbidden strings
Marie59 Oct 1, 2024
b805ae2
fix lint
Marie59 Oct 1, 2024
294b62c
Update check.py
Marie59 Oct 1, 2024
ac236fa
add pwd checks
Marie59 Oct 10, 2024
f0bf025
add credentials
Marie59 Oct 21, 2024
1c40798
add credentials
Marie59 Oct 21, 2024
12880b7
Update copernicusmarine.xml
bgruening Oct 22, 2024
a6afb87
Merge branch 'master' into ocean
bgruening Oct 22, 2024
963bd17
Update copernicusmarine.xml
bgruening Oct 22, 2024
2cc958e
Update copernicusmarine.xml
bgruening Oct 22, 2024
445d1f4
what is going on here
bgruening Oct 22, 2024
e5c0884
Update pr.yaml
bgruening Oct 22, 2024
8892007
Merge branch 'master' into ocean
bgruening Oct 22, 2024
b390b95
Update pr.yaml
bgruening Oct 22, 2024
59c2e18
last idea ... narf
bgruening Oct 22, 2024
a26ed78
Merge branch 'master' into ocean
bgruening Oct 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tools/ocean/.shed.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ categories:
- Ecology
owner: ecology
remote_repository_url: https://github.com/galaxyecology/tools-ecology/tree/master/tools/ocean
homepage_url: https://github.com/Marie59/FE-ft-ESG/tree/main/argo
homepage_url: https://github.com/Marie59/FE-ft-ESG/tree/main/ocean
long_description: |
Access, process, visualise oceanographic data for the Earth System
type: unrestricted
Expand Down
75 changes: 75 additions & 0 deletions tools/ocean/check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
import os
import subprocess
import sys


def validate_command(command):
# Example validation: Ensure the command does not contain
# potentially dangerous substrings
forbidden_substrings = ['rm -rf', 'sudo', 'dd if=', 'curl', 'wget']
Marie59 marked this conversation as resolved.
Show resolved Hide resolved
for substring in forbidden_substrings:
if substring in command:
message = f"Error: Command has forbidden substring '{substring}'"
return False, message

# Check if the command starts with 'copernicusmarine'
if not command.startswith('copernicusmarine'):
Marie59 marked this conversation as resolved.
Show resolved Hide resolved
return False, "Error: Command must start with 'copernicusmarine'"

# Remove 'copernicusmarine' from the start
command = command[len('copernicusmarine'):].strip()

# Check for specific commands and their arguments
if command.startswith('subset'):
# Check for required arguments for 'subset' command
if not ('--dataset-id' in command or '--dataset-url' in command):
message = "Error: 'subset' command must have '--dataset-id' or '--dataset-url'"
return False, message
elif command.startswith('get'):
# Check for required arguments for 'get' command
if not ('--dataset-id' in command or '--dataset-url' in command):
message = "Error: 'get' command must have '--dataset-id' or '--dataset-url'"
return False, message
elif command.startswith('login') or command.startswith('describe'):
message = "This tool only accepts 'subset' and 'get' commands."
return False, message
else:
return False, "Error: Command must be 'subset' or 'get'"

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if a user puts in any of --force-download --username --password ...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah let's try I'll see and add the correct messages

Copy link
Collaborator Author

@Marie59 Marie59 Oct 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just tried this is no issue if the user writes down this too it works fine admittedly that the user wrote his credentials in the environment. if not It says the password or username is invalid. Is this enough ?

return True, None


def main():
# Check if a filename argument is provided
if len(sys.argv) != 2:
print("Usage: python check.py <config_file>")
sys.exit(1)

# Get the filename from the command line argument
config_file = sys.argv[1]

# Check if the file exists
if not os.path.isfile(config_file):
print(f"Error: File '{config_file}' does not exist.")
sys.exit(1)

# Read the content of the file
with open(config_file, 'r') as file:
command = file.read().strip()

# Validate the command
is_valid, error_message = validate_command(command)
if not is_valid:
print(error_message)
sys.exit(1)

# Execute the command
try:
subprocess.run(command, shell=True, check=True)
except subprocess.CalledProcessError as e:
print(f"Error: Command failed with exit code {e.returncode}")
sys.exit(1)


if __name__ == "__main__":
main()
93 changes: 93 additions & 0 deletions tools/ocean/copernicusmarine.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
<tool id="copernicusmarine" name="Copernicue Marine Data Store" version="@TOOL_VERSION@+galaxy@VERSION_SUFFIX@" profile="22.01" license="MIT">
<description>retrieve marine data</description>
<macros>
<token name="@TOOL_VERSION@">1.3.3</token>
<token name="@VERSION_SUFFIX@">0</token>
</macros>
<requirements>
<requirement type="package" version="@TOOL_VERSION@">copernicusmarine</requirement>
</requirements>
<command detect_errors="exit_code"><![CDATA[
#set $cmems_username = $__user__.extra_preferences.get('cmems_username', "")
#set $cmems_password = $__user__.extra_preferences.get('cmems_password', "")
#if $cmems_username != ""
echo 'Error. Set your CMEMS credentials via: User -> Preferences -> Manage Information' &&
#end if

#set $filename="config_file_with_my_command"
echo -e '$input_text' > $filename --force-download --username $cmems_username --password $cmems_password &&

python '$__tool_directory__/check.py' $filename

]]></command>
<configfiles>
<configfile name="cmems_credentials">
#set $cmems_username = $__user__.extra_preferences.get('cmems_username', "")
#set $cmems_password = $__user__.extra_preferences.get('cmems_password', "")
cmems_username: $cmems_username
cmems_password: $cmems_password
</configfile>
</configfiles>
<inputs>
<param name="input_text" label="Paste API Request" type="text" area="true">
<sanitizer invalid_char="">
<valid initial="string.letters,string.digits">
<add value="_" />
<add value="-" />
<add value=" " />
<add value=":" />
<add value="." />
</valid>
</sanitizer>
<validator type="regex">[0-9a-zA-Z_]+</validator>
Marie59 marked this conversation as resolved.
Show resolved Hide resolved
</param>
</inputs>
<outputs>
<data name="output_netcdf" label="Copernicus marine data" from_work_dir="./*.nc" format="netcdf"/>
</outputs>
<tests>
<test expect_failure="true" expect_exit_code="1">
<param name="input_text" value="copernicusmarine subset --dataset-id cmems_mod_med_phy_my_4.2km-climatology_P1M-m --variable bottomT_avg --variable bottomT_std --variable mlotst_avg --variable mlotst_std --variable so_avg --variable so_std --variable thetao_avg --variable thetao_std --variable uo_avg --variable uo_std --variable vo_avg --variable vo_std --variable zos_avg --variable zos_std --start-datetime 1993-12-01T00:00:00 --end-datetime 1993-12-01T00:00:00 --minimum-longitude 10.283266521135577 --maximum-longitude 12.139348881644054 --minimum-latitude 38.461936025366256 --maximum-latitude 39.82957565942514 --minimum-depth 1.0182366371154785 --maximum-depth 1.0182366371154785"/>
</test>
</tests>
<help><![CDATA[
============================
Copernicus Marine Data Store
============================

** Context **
This tool is a wrapper to retrieve data from the Copernicus Marine Environment Monitoring Service (CMEMS).

- It allows to retrieve data from the Copernicus Marine Service.
- Any user willing to use this tool needs to `create a new account <https://data.marine.copernicus.eu/login>`_.
- Set your Copernicus CMEMS API Key via: User > Preferences > Manage Information
- Enter your username and password for Copernicus CMEMS
- Compose your request directly on Copernicus Marine Data Store
- Choose there which data interest you click on the download button
- Then on the top right click again on the big download butto
- Log in
- Click on "Automate"
- You should have a pop-up window called "Automate download"
- Copy the ">_Command-Line Interface" proposed there
- Back on Galaxy paste it in the input field "Paste API Request".

For more information on the Command-Line Interface (CLI) go on `Copernicus Marine Toolbox CLI - Subset <https://help.marine.copernicus.eu/en/articles/7972861-copernicus-marine-toolbox-cli-subset>`

** Input **
Command line from the Copernicus marine services copy paste as a text.

** Output **
A netcdf file containing the the data chose by the user from the Copernicus Marine Data Store.

]]></help>
<citations>
<citation type="bibtex">
@misc{Copernicus,
title={Copernicus Marine Data Store},
url={https://data.marine.copernicus.eu/products},
note={E.U. Copernicus Marine Service Information},
author={Copernicus, E.U.}
}
</citation>
</citations>
</tool>
Loading