Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix fatal errors in GxIT and ansible tutorials #4370

Merged
merged 4 commits into from
Oct 3, 2023
Merged

Fix fatal errors in GxIT and ansible tutorials #4370

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f99e2a1
Fix errors in GxIT and ansible tutorials
tuncK Oct 3, 2023
01e8a49
typo fixed
tuncK Oct 3, 2023
1b9f010
revert galaxy version upgrades
tuncK Oct 3, 2023
94f77d8
Add cross-links between SSL boxes
hexylena Oct 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions topics/admin/tutorials/ansible-galaxy/tutorial.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -897,7 +897,7 @@ The configuration is quite simple thanks to the many sensible defaults that are
> `galaxy_commit_id` | `release_23.0` | The git reference to check out, which in this case is the branch for Galaxy Release 23.0
> `galaxy_force_checkout` | `true` | If we make any modifications to the Galaxy codebase, they will be removed. This way we know we're getting an unmodified Galaxy and no one has made any unexpected changes to the codebase.
> `miniconda_prefix` | {% raw %}`"{{ galaxy_tool_dependency_dir }}/_conda"`{% endraw %} | We will manually install conda as well. Normally Galaxy will attempt to auto-install this, but since we will set up a production-ready instance with multiple handlers, there is the chance that they can become deadlocked.
> `miniconda_version` | `4.12.0` | Install a specific miniconda version, the latest one at the time of writing that was tested and working.
> `miniconda_version` | `23.9` | Install a specific miniconda version, the latest one at the time of writing that was tested and working.
> `miniconda_channels` ` | `['conda-forge', 'defaults']` | Use the community-maintained conda-forge channel in addition to the standard defaults channel of Conda.
>
> > <tip-title>Different Galaxy Releases!</tip-title>
Expand All @@ -923,7 +923,7 @@ The configuration is quite simple thanks to the many sensible defaults that are
> +galaxy_commit_id: release_23.0
> +galaxy_force_checkout: true
> +miniconda_prefix: "{{ galaxy_tool_dependency_dir }}/_conda"
> +miniconda_version: 4.12.0
> +miniconda_version: 23.9
> +miniconda_channels: ['conda-forge', 'defaults']
> {% endraw %}
> ```
Expand Down Expand Up @@ -956,7 +956,7 @@ The configuration is quite simple thanks to the many sensible defaults that are
> +++ b/group_vars/galaxyservers.yml
> @@ -10,3 +10,17 @@ galaxy_force_checkout: true
> miniconda_prefix: "{{ galaxy_tool_dependency_dir }}/_conda"
> miniconda_version: 4.12.0
> miniconda_version: 23.9
> miniconda_channels: ['conda-forge', 'defaults']
> +
> +galaxy_config:
Expand Down Expand Up @@ -1974,7 +1974,7 @@ For this, we will use NGINX (pronounced "engine X" /ˌɛndʒɪnˈɛks/ EN-jin-EK
> > -nginx_conf_ssl_certificate_key: /etc/ssl/user/privkey-www-data.pem
> > ```
> > {% endraw %}
> >
> > Please also see the changes in [the other SSL tip box](#details-running-this-tutorial-i-without-i-ssl-1)
> {: .details}
>
> 4. Create the directory `templates/nginx` (staying in galaxy directory, after which groups_vars, roles will be siblings of templates), where we will place our configuration files which should be templated out to the server.
Expand Down Expand Up @@ -2082,6 +2082,8 @@ For this, we will use NGINX (pronounced "engine X" /ˌɛndʒɪnˈɛks/ EN-jin-EK
>
> > <details-title>Running this tutorial <i>without</i> SSL</details-title>
> >
> > Please be sure to also make the changes in [the other SSL box](#details-running-this-tutorial-i-without-i-ssl).
> >
> > In your `galaxy.j2` in the above step, you should change the `listen` parameter:
> >
> > {% raw %}
Expand All @@ -2092,7 +2094,7 @@ For this, we will use NGINX (pronounced "engine X" /ˌɛndʒɪnˈɛks/ EN-jin-EK
> > +listen *:80 default_server;
> > ```
> > {% endraw %}
> >
> >
> {: .details}
>
> 6. Run the playbook. At the very end, you should see output like the following indicating that Galaxy has been restarted:
Expand Down Expand Up @@ -2232,7 +2234,7 @@ Finally, we have explicitly mapped the tool `bwa` to run in the `local_env` envi
> --- a/group_vars/galaxyservers.yml
> +++ b/group_vars/galaxyservers.yml
> @@ -11,6 +11,24 @@ miniconda_prefix: "{{ galaxy_tool_dependency_dir }}/_conda"
> miniconda_version: 4.12.0
> miniconda_version: 23.9
> miniconda_channels: ['conda-forge', 'defaults']
>
> +# Galaxy Job Configuration
Expand Down
73 changes: 50 additions & 23 deletions topics/dev/tutorials/interactive-tools/tutorial.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ There are three elements to a GxIT - an application script, a Docker container a

## How do Interactive Tools work?

Interactive tools are a special breed of Galaxy tool which is relatively
Interactive tools are a special breed of Galaxy tools, which is relatively
new to the Galaxy ecosystem - they are a work in progress!
GxITs enable the user to run an entire web application through Galaxy, which
opens as a new tab in the browser. This can enable users to explore and manipulate
Expand All @@ -107,7 +107,7 @@ run. They then get some output file(s) when the tool run is complete. In an
Interactive Tool, however, the users are provided with a graphical web interface
allowing them to interact with their data in real time. This is great for
visualising data, but if it is possible to provide the same
functionality with a regular tool (e.g. by rendering an HTML file as an output)
functionality with a regular tool (e.g. by rendering an HTML file as an output),
then an Interactive Tool might not be necessary.

If you are sure that a static
Expand Down Expand Up @@ -162,16 +162,16 @@ cycle.

The application that we will wrap in this tutorial is a simple web tool which
allows the user to upload `csv` and `tsv` files, manipulate them and download
them. Our application is based on an R Shiny App hosted with Shiny server.
them. Our application is based on an R Shiny App hosted with a Shiny server.

Note that there is no link between this Interactive Tool and the Galaxy history.
More complex applications might be able to read and write outputs to the user's
history to create a more integrated experience - see the
[Additional components section](#galaxy-history-interaction)
to see an example of how this can be done.

Our example can already be found [online](https://github.com/Lain-inrae/geoc-gxit).
In the following sections, we will study how it was built.
Our example application can already be found [online](https://github.com/Lain-inrae/geoc-gxit).
In the following sections, we will study how it can be built into a GxIT.


> <hands-on-title></hands-on-title>
Expand Down Expand Up @@ -248,7 +248,7 @@ FROM rocker/shiny
# set author
MAINTAINER Lain Pavot <[email protected]>

## we copy the installer and run it before copying the entier project to prevent
## we copy the installer and run it before copying the entire project to prevent
## reinstalling everything each time the project has changed

COPY ./gxit/install.R /tmp/
Expand Down Expand Up @@ -540,8 +540,8 @@ Let's check this integration on your machine. You can use a VM if you prefer not
> # Get the galaxy project. A new directory named "galaxy" will be created.
> # This directory contains the whole project
> git clone https://github.com/galaxyproject/galaxy
> # Checkout the last stable version (v21.09 as the time of writing)
> cd galaxy && git checkout v21.09
> # Checkout the last stable version (v23.1 as the time of writing)
> cd galaxy && git checkout release_23.1
> ```
{: .hands_on}

Expand All @@ -554,10 +554,9 @@ Let's check this integration on your machine. You can use a VM if you prefer not
> cd ~/GxIT/galaxy/config
> # Create custom config files
> cat galaxy.yml.interactivetools > galaxy.yml
> cat job_conf.xml.interactivetools > job_conf.xml
> cat tool_conf.xml.sample > tool_conf.xml
> ```
> In `galaxy.yml`, add the `galaxy_infrastructure_url` parameter to the galaxy section:
> In `galaxy.yml`, ensure that the `galaxy_infrastructure_url` parameter is present under the galaxy section:
> ```yaml
> galaxy:
> galaxy_infrastructure_url: http://localhost:8080
Expand All @@ -576,6 +575,34 @@ Let's check this integration on your machine. You can use a VM if you prefer not
> And of course, you have no obligation to put your GxITs in this section.
> You can put them in any section.
>
> Next, create a simple `job_conf.xml` with the following contents, which basically specifies how Galaxy should execute a job. It is also possible to execute the tools directly from the local environment, but here we would like the tool to make use of the container we have just made.
> ```xml
> <?xml version="1.0"?>
> <job_conf>
> <plugins>
> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/>
> </plugins>
> <destinations default="docker_dispatch">
> <destination id="local" runner="local"/>
> <destination id="docker_local" runner="local">
> <param id="docker_enabled">true</param>
> <param id="docker_volumes">$defaults</param>
> <param id="docker_sudo">false</param>
> <param id="docker_net">bridge</param>
> <param id="docker_auto_rm">true</param>
> <param id="require_container">true</param>
> <param id="container_monitor">true</param>
> <param id="docker_set_user"></param>
> <param id="docker_run_extra_arguments">--add-host localhost:host-gateway</param>
> </destination>
> <destination id="docker_dispatch" runner="dynamic">
> <param id="type">docker_dispatch</param>
> <param id="docker_destination_id">docker_local</param>
> <param id="default_destination_id">local</param>
> </destination>
> </destinations>
> </job_conf>
> ```
> Finally, copy your GxIT wrapper to the Interactive Tool directory:
> ```sh
> cp ~/my_filepath/interactivetool_tabulator.xml ~/GxIT/galaxy/tools/interactive/
Expand All @@ -592,10 +619,10 @@ Go to the Galaxy directory and:
Galaxy is available at [http://localhost:8080/](http://localhost:8080/) and you should be able to use your GxIT.
Congrats!

# Installing in Galaxy
# Deployment in a running Galaxy instancce

We now have all the required components, we can install the tool in our
configured Galaxy instance.
We now have all the required components and tested in a local Galaxy instance above, we can install the tool in our
configured Galaxy instance for immediate production use.
This is as simple as dropping the tool XML in the right location inside
the Galaxy core application directory, and adding the tool to our
`tool_conf_interactive.xml` file.
Expand Down Expand Up @@ -629,7 +656,7 @@ the Galaxy core application directory, and adding the tool to our
>
> ```sh
> # Open the Interactive Tools config file for editing:
> sudo nano /srv/galaxy/config/tool_conf_interactive.xml
> sudo nano /srv/galaxy/config/tool_conf.xml
> ```
>
> 4. This configuration file should have been created when
Expand All @@ -649,7 +676,7 @@ the Galaxy core application directory, and adding the tool to our
> 5. Now we just need to restart the Galaxy server to refresh the tool registry
>
> ```sh
> sudo service galaxy restart
> sudo galaxyctl restart
> ```
>
{: .hands_on}
Expand All @@ -662,7 +689,7 @@ Have a look in the web interface of your Galaxy instance. You should find the ne
> - the Galaxy tool XML
> - the Docker image
>
> We have already pushed the Docker image to the cloud (thought it should be hosted on an [approved registry](#push-the-image) for production use).
> We have already pushed the Docker image to the cloud (though it should be hosted on an [approved registry](#push-the-image) for production use).
>
> All that's left is to distribute the tool XML. This would conventionally be done through the ToolShed. But the ToolShed doesn't support GxITs yet! This leaves us only two options for distributing the tool XML:
> - Make a pull request against [Galaxy core](https://github.com/galaxyproject/galaxy) to include the XML file under `tools/interactive/`
Expand All @@ -675,7 +702,7 @@ Have a look in the web interface of your Galaxy instance. You should find the ne
> The steps that we took in this section can be easily incorporated into an Ansible playbook for deploying GxITs to a Galaxy server. This means that you can manage and deploy a GxIT as part of your Galaxy instance without merging into the `galaxyproject/galaxy` repository (or a fork of it).
>
> The [Interactive Tools admin tutorial]({% link topics/admin/tutorials/interactive-tools/tutorial.md %})
> demonstrates how this can be acheived by adding our tool XML to the "local tools" section of the Ansible Playbook. However, for our GxIT to show up in the correct tool panel we'll need to add an extra config file: `local_tool_conf.xml`.
> demonstrates how this can be acheived by adding our tool XML to the "local tools" section of the Ansible Playbook. However, for our GxIT to show up in the correct tool panel, we need to add an extra config file: `local_tool_conf.xml`.
>
>
> 1. Copy the GxIT tool XML to `files/galaxy/tools/interactivetool_tabulator.xml` in your Ansible directory
Expand Down Expand Up @@ -732,19 +759,19 @@ The most obvious way to test a tool is simply to run it in the Galaxy UI, straig
The GxIT that we wrapped in this tutorial was a simple example, and you should now understand what is required to create an Interactive Tool for Galaxy. However, there are a few additional components that can enhance the reliability and user experience of the tool. In addition, more complex applications may require some additional components or workarounds the create the desired experience for the user.

## Run script
In the case of our `Tabulator` application, the run script is simply the R script that renders our Shiny App. It is quite straightforward to call this from our Galaxy tool XML. However, some web apps might require more elaborate commands to be run. In this situation there are a number of solutions demonstrated in the `<command>` section of [existing GxITs](https://github.com/galaxyproject/galaxy/tree/dev/tools/interactive):
In the case of our `Tabulator` application, the run script is simply the R script that renders our Shiny App. It is quite straightforward to call this from our Galaxy tool XML. However, some web apps might require more elaborate commands to be run. In this situation, there are a number of solutions demonstrated in the `<command>` section of [existing GxITs](https://github.com/galaxyproject/galaxy/tree/dev/tools/interactive):
- [Guacamole Desktop](https://github.com/galaxyproject/galaxy/blob/dev/tools/interactive/interactivetool_guacamole_desktop.xml): application startup with `startup.sh`
- [HiCBrowser](https://github.com/galaxyproject/galaxy/blob/dev/tools/interactive/interactivetool_hicbrowser.xml): application startup with `supervisord`
- [AskOmics](https://github.com/galaxyproject/galaxy/blob/dev/tools/interactive/interactivetool_askomics.xml): configuration with Python and Bash scripts, followed by `start_all.sh` to run the application.

## Templated config files
Using the `<configfiles>` section in the tool XML, we can enable complex user configuration for the application by templating a run script or configuration file to be read by the application. In this application for example, we could use a `<configfiles>` section to template user input into the `app.R` script that runs the application within the Docker container. This could enable the user to customize the layout of the app before launch.
Using the `<configfiles>` section in the tool XML, we can enable complex user configuration for the application by templating a run script or configuration file to be read by the application. In this application, for example, we could use a `<configfiles>` section to template user input into the `app.R` script that runs the application within the Docker container. This could enable the user to customize the layout of the app before launch.

## Reserved environment variables

There are a few environment variables
that are accessible in the command section of the tool XML - these can be handy when writing your tool script.
[check the docs](https://docs.galaxyproject.org/en/latest/dev/schema.html#reserved-variables) for a full reference on the tool XML.
[Check the docs](https://docs.galaxyproject.org/en/latest/dev/schema.html#reserved-variables) for a full reference on the tool XML.

```sh
$__tool_directory__
Expand All @@ -753,7 +780,7 @@ $__user_id__
$__user_email__
```

It can also be useful to create and inject environment variables into the tool context. This can be acheived using the `<environment variables>` tag in the tool XML. The RStudio GxIT again provides an example of this:
It can also be useful to create and inject environment variables into the tool context. This can be acheived using the `<environment variables>` tag in the tool XML. The [RStudio GxIT](https://github.com/galaxyproject/galaxy/blob/b180b7909dc3fe2750fbc8b90214e201eb276794/tools/interactive/interactivetool_rstudio.xml#L12) again provides an example of this:

```xml
<environment_variables>
Expand Down Expand Up @@ -781,7 +808,7 @@ We have demonstrated how to pass an input file to the Docker container. But what

## Self-destruct script

Unlike regular tools, web applications will run indefinitely until terminated. With Galaxy's legacy "Interactive Environments", this used to result in "zombie" containers hanging around and clogging up the Galaxy server. You may notice a `terminate.sh` script in some older GxITs as a workaround to this problem, but the new GxIT architecture handles container termination for you. This script is no longer required or reccommended.
Unlike regular tools that exit after the execution of the underlying command is complete, web applications will run indefinitely until terminated. With Galaxy's legacy "Interactive Environments", this used to result in "zombie" containers hanging around and clogging up the Galaxy server. You may notice a `terminate.sh` script in some older GxITs as a workaround to this problem, but the new GxIT architecture handles container termination for you. This script is no longer required nor recommended.


# Troubleshooting
Expand All @@ -790,6 +817,6 @@ Having issues with your Interactive Tool? Here are a few ideas for how to troubl

- Getting an error in the Galaxy History? Click on the "view" icon to see details of the tool run, including the tool command, `stdout` and `stderr`.
- If the tool's `stdout`/`stderr` is not enough, consider modifying the Docker image to make it more verbose. Add print/log statements and assertions. Write an application log to a file that can be collected as Galaxy output.
- Try running the container with Docker directly on your development machine. If the application doesn't work independantly it certainly won't work inside Galaxy!
- Try running the container with Docker directly on your development machine. If the application doesn't work independently, it certainly won't work inside Galaxy!
- If you need to debug the Docker container itself, it can be useful to write output/logging to a [mounted volume](https://docs.docker.com/storage/volumes/) that can be inspected after the tool has run.
- You can also open a `bash` terminal inside the container to check the container state while the application is running: `docker exec -it mycontainer /bin/bash`