-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Copernicus marine data retrieving tool #126
Changes from 48 commits
56f91c0
f10ecd4
7cd9998
8924551
0caabeb
a8da9eb
877ae31
656bbc0
36fd13f
531fb1b
ef5b8a2
aea8347
1f5de95
4a8decf
750b55b
05cbdaa
c6ec6cd
7f2e88c
e152059
b9155de
231217e
a5e2f54
aaa595a
9877fb3
f81e272
e597c1b
0eeec82
68119e0
ea7d27d
f7e5c04
f7a9dc0
f3155e9
a6757da
e64ae35
363a69e
42b28f7
50fd246
2705dbe
56566a5
9408fbd
830d57f
16aee90
1927518
10438f2
8ef41c5
b805ae2
294b62c
ac236fa
f0bf025
1c40798
12880b7
a6afb87
963bd17
2cc958e
445d1f4
e5c0884
8892007
b390b95
59c2e18
a26ed78
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
import os | ||
import subprocess | ||
import sys | ||
|
||
|
||
def validate_command(command): | ||
# Example validation: Ensure the command does not contain | ||
# potentially dangerous substrings | ||
forbidden_substrings = ["rm", | ||
"rm -rf", | ||
"sudo", | ||
"dd if=", | ||
"curl", | ||
"wget", | ||
";", | ||
"&&"] | ||
for substring in forbidden_substrings: | ||
if substring in command: | ||
message = f"Error: Command has forbidden substring '{substring}'" | ||
return False, message | ||
|
||
# Check if the command starts with 'copernicusmarine' | ||
if not command.startswith("copernicusmarine"): | ||
return False, "Error: Command must start with 'copernicusmarine'" | ||
|
||
# Remove 'copernicusmarine' from the start | ||
command = command[len("copernicusmarine"):].strip() | ||
|
||
# Check for specific commands and their arguments | ||
if command.startswith("subset"): | ||
# Check for required arguments for 'subset' command | ||
if not ("--dataset-id" in command or "--dataset-url" in command): | ||
message = ( | ||
"Error: 'subset' command must have '--dataset-id' or " | ||
"'--dataset-url'" | ||
) | ||
return False, message | ||
elif command.startswith("get"): | ||
# Check for required arguments for 'get' command | ||
if not ("--dataset-id" in command or "--dataset-url" in command): | ||
message = ( | ||
"Error: 'get' command must have '--dataset-id' or " | ||
"'--dataset-url'" | ||
) | ||
return False, message | ||
elif command.startswith("login") or command.startswith("describe"): | ||
message = "This tool only accepts 'subset' and 'get' commands." | ||
return False, message | ||
else: | ||
return False, "Error: Command must be 'subset' or 'get'" | ||
|
||
return True, None | ||
|
||
|
||
def main(): | ||
# Check if a filename argument is provided | ||
if len(sys.argv) != 2: | ||
print("Usage: python check.py <config_file>") | ||
sys.exit(1) | ||
|
||
# Get the filename from the command line argument | ||
config_file = sys.argv[1] | ||
|
||
# Check if the file exists | ||
if not os.path.isfile(config_file): | ||
print(f"Error: File '{config_file}' does not exist.") | ||
sys.exit(1) | ||
|
||
# Read the content of the file | ||
with open(config_file, "r") as file: | ||
command = file.read().strip() | ||
|
||
# Validate the command | ||
is_valid, error_message = validate_command(command) | ||
if not is_valid: | ||
print(error_message) | ||
sys.exit(1) | ||
|
||
# Execute the command | ||
try: | ||
subprocess.run(command, shell=True, check=True) | ||
except subprocess.CalledProcessError as e: | ||
print(f"Error: Command failed with exit code {e.returncode}") | ||
sys.exit(1) | ||
|
||
|
||
if __name__ == "__main__": | ||
main() |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,114 @@ | ||
<tool id="copernicusmarine" name="Copernicue Marine Data Store" version="@TOOL_VERSION@+galaxy@VERSION_SUFFIX@" profile="22.01" license="MIT"> | ||
<description>retrieve marine data</description> | ||
<macros> | ||
<token name="@TOOL_VERSION@">1.3.3</token> | ||
<token name="@VERSION_SUFFIX@">0</token> | ||
</macros> | ||
<requirements> | ||
<requirement type="package" version="@TOOL_VERSION@">copernicusmarine</requirement> | ||
</requirements> | ||
<command detect_errors="exit_code"><![CDATA[ | ||
#import re | ||
#set $cmems_username = $__user__.extra_preferences.get('cmems_username', "") | ||
#set $cmems_password = $__user__.extra_preferences.get('cmems_password', "") | ||
|
||
#if $cmems_username == "" or $cmems_password == "" | ||
#set $cmems_username = os.getenv('CMEMS_USERNAME', '') | ||
#set $cmems_password = os.getenv('CMEMS_PASSWORD', '') | ||
Marie59 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
echo "Using global ENVs" && | ||
|
||
#if re.match(r"^(?=.*[a-z])(?=.*[A-Z])(?=.*\d).{8,}$", $cmems_password): | ||
echo "Password meets the required criteria" && | ||
#else: | ||
echo "Password does not meet the required criteria" && | ||
#end if | ||
|
||
#end if | ||
|
||
#if $cmems_username != "" | ||
echo 'Error. Set your CMEMS credentials via: User -> Preferences -> Manage Information' && | ||
#end if | ||
|
||
#set $filename="config_file_with_my_command" | ||
echo -e '$input_text' > $filename --force-download --username $cmems_username --password $cmems_password && | ||
|
||
python '$__tool_directory__/check.py' $filename | ||
]]></command> | ||
<configfiles> | ||
<configfile name="cmems_credentials"> | ||
#set $cmems_username = $__user__.extra_preferences.get('cmems_username', "") | ||
#set $cmems_password = $__user__.extra_preferences.get('cmems_password', "") | ||
cmems_username: $cmems_username | ||
cmems_password: $cmems_password | ||
</configfile> | ||
</configfiles> | ||
<inputs> | ||
<param name="input_text" label="Paste API Request" type="text" area="true"> | ||
<sanitizer invalid_char=""> | ||
<valid initial="string.letters,string.digits"> | ||
<add value="_" /> | ||
<add value="-" /> | ||
<add value=" " /> | ||
<add value=":" /> | ||
<add value="." /> | ||
<add value=" " /> | ||
</valid> | ||
</sanitizer> | ||
<validator type="regex">[0-9a-zA-Z_:\-\. ]+</validator> | ||
</param> | ||
</inputs> | ||
<outputs> | ||
<data name="output_netcdf" label="Copernicus marine data" from_work_dir="./*.nc" format="netcdf"/> | ||
</outputs> | ||
<tests> | ||
<test expect_num_outputs="1"> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could test this tools with ENV variables, that we define here in GA. Means we can store your credentials in Github and use them. The other option is that we tell Galaxy that we know this test is failing and then we check for a specific error message. If this error message pops up the test passed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not sure I fully understand how to do that but I think the 1st option might be best ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
<param name="input_text" value="copernicusmarine subset --dataset-id cmems_mod_med_phy_my_4.2km-climatology_P1M-m --variable bottomT_avg --variable bottomT_std --variable mlotst_avg --variable mlotst_std --variable so_avg --variable so_std --variable thetao_avg --variable thetao_std --variable uo_avg --variable uo_std --variable vo_avg --variable vo_std --variable zos_avg --variable zos_std --start-datetime 1993-12-01T00:00:00 --end-datetime 1993-12-01T00:00:00 --minimum-longitude 10.283266521135577 --maximum-longitude 12.139348881644054 --minimum-latitude 38.461936025366256 --maximum-latitude 39.82957565942514 --minimum-depth 1.0182366371154785 --maximum-depth 1.0182366371154785"/> | ||
<output name="output_netcdf"> | ||
<assert_contents> | ||
<has_text text="_NCProperties"/> | ||
</assert_contents> | ||
</output> | ||
</test> | ||
</tests> | ||
<help><![CDATA[ | ||
============================ | ||
Copernicus Marine Data Store | ||
============================ | ||
|
||
** Context ** | ||
This tool is a wrapper to retrieve data from the Copernicus Marine Environment Monitoring Service (CMEMS). | ||
|
||
- It allows to retrieve data from the Copernicus Marine Service. | ||
- Any user willing to use this tool needs to `create a new account <https://data.marine.copernicus.eu/login>`_. | ||
- Set your Copernicus CMEMS API Key via: User > Preferences > Manage Information | ||
- Enter your username and password for Copernicus CMEMS | ||
- Compose your request directly on Copernicus Marine Data Store | ||
- Choose there which data interest you click on the download button | ||
- Then on the top right click again on the big download butto | ||
- Log in | ||
- Click on "Automate" | ||
- You should have a pop-up window called "Automate download" | ||
- Copy the ">_Command-Line Interface" proposed there | ||
- Back on Galaxy paste it in the input field "Paste API Request". | ||
|
||
For more information on the Command-Line Interface (CLI) go on `Copernicus Marine Toolbox CLI - Subset <https://help.marine.copernicus.eu/en/articles/7972861-copernicus-marine-toolbox-cli-subset>` | ||
|
||
** Input ** | ||
Command line from the Copernicus marine services copy paste as a text. | ||
|
||
** Output ** | ||
A netcdf file containing the the data chose by the user from the Copernicus Marine Data Store. | ||
|
||
]]></help> | ||
<citations> | ||
<citation type="bibtex"> | ||
@misc{Copernicus, | ||
title={Copernicus Marine Data Store}, | ||
url={https://data.marine.copernicus.eu/products}, | ||
note={E.U. Copernicus Marine Service Information}, | ||
author={Copernicus, E.U.} | ||
} | ||
</citation> | ||
</citations> | ||
</tool> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if a user puts in any of
--force-download --username --password ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah let's try I'll see and add the correct messages
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just tried this is no issue if the user writes down this too it works fine admittedly that the user wrote his credentials in the environment. if not It says the password or username is invalid. Is this enough ?