Training material for Galaxy and Pulsar on Surf Research Cloud

CONTRIBUTORS.yaml

@@ -707,6 +707,12 @@ dyusuf:
     affiliations:
       - uni-freiburg
 
+dometto:
+    name: Dawa Ometto
+    joined: 2024-12
+    affiliations:
+      - surf
+
 eancelet:
     name: Estelle Ancelet
     email: [email protected]
@@ -1694,6 +1700,12 @@ miRlyKayleigh:
     email: [email protected]
     joined: 2022-12
 
+mirelaminkova:
+    name: Mirela Minkova
+    joined: 2024-12
+    affiliations:
+      - surf
+
 moffmade:
     name: Cory Maughmer
     joined: 2018-06
@@ -2696,6 +2708,10 @@ yvesvdb:
     affiliations:
       - ifb
 
+yuliiaorlova:
+    name: Yuliia Orlova
+    joined: 2024-12
+
 zargham-ahmad:
     name: Zargham Ahmad
     email: [email protected]

ORGANISATIONS.yaml

@@ -318,6 +318,14 @@ swiss-tph:
     github: false
     ror: "03adhka07"
 
+surf:
+    short_name: SURF
+    name: SURF
+    url: https://www.surf.nl/
+    avatar: "/training-material/shared/images/surf.png"
+    github: false
+    ror: "009vhk114"
+
 tb-capt:
     short_name: TB-CAPT
     name: TB-CAPT

_layouts/event.html

@@ -173,13 +173,14 @@ <h3>Get set up for the course!</h3>
     <p>Follow the steps below to get set up with everything you need to participate in the course!</p>
 
     <!-- Galaxy setup -->
-    {% unless event.infrastructure.no_galaxy %}
+    {% if event.infrastructure.servers %}
     <div class="card setup-card">
 	  <div class="row no-gutters">
 		<div class="col-sm-3">
 		  <img src="{% link assets/images/galaxylogo.png %}" class="card-img" alt="Galaxy logo" style="max-height: 10em">
 		</div>
 		<div class="col-sm-9">
+
 		  <div class="card-body">
 			<h5 class="card-title">Create a Galaxy Account</h5>
 			<p class="card-text">
@@ -200,7 +201,7 @@ <h5 class="card-title">Create a Galaxy Account</h5>
 		</div>
 	  </div>
 	</div>
-	{% endunless %}
+	{% endif %}
 
     <!-- TIaaS setup -->
     {% if event.infrastructure.tiaas %}

_plugins/api.rb

@@ -279,7 +279,9 @@ def generate(site)
 
 
 Jekyll::Hooks.register :site, :post_read do |site|
-  Gtn::Hooks.by_tool(site)
+  if Jekyll.env == 'production'
+    Gtn::Hooks.by_tool(site)
+  end
 end
 
 # Basically like `PageWithoutAFile`, we just write out the ones we'd created earlier.

bin/schema-event.yaml

@@ -320,10 +320,6 @@ mapping:
                 type: str
                 description: |
                   Custom markdown formatted text to display. Note that Jekyll things like `{% snippet %}` or `{% include %}` will not work here.
-          no_galaxy:
-            type: bool
-            description: |
-              Set to `true` to disable the automatic "register for a galaxy account" card.
     tags:
         type: seq
         sequence:

events/2025-01-28-src.md

@@ -0,0 +1,106 @@
+---
+layout: event
+
+# Status of this page, remove these lines when ready
+draft: true  # will hide your event from the GTN events list
+status: wip  # add a banner that warns that the contents of the page are still subject to change
+
+# Description of your event
+title: Galaxy at SURF Research Cloud workshop
+description: |
+  Half-day workshop demonstrating the local Galaxy instance built for users of SURF Research Cloud. 
+
+#cover-image:         # image for your corse, put in 'events/images' folder
+#cover-image-alt:     # supply alt text describing your image
+
+# Practical Information
+date_start: 2025-01-28
+
+cost: free # Or, e.g. 150 EUR
+audience: This event is targeted towards researchers in the Netherlands who already have experience working with Galaxy. 
+contact_email: [email protected]
+async: false # if asynchronous, we will not display the time columns on the program
+mode: onsite # online, onsite, or hybrid
+
+registration:
+  message: Please send an email to [email protected] to register for the workshop!
+  deadline: 2025-01-10
+  open: true # set this if registration is not open yet
+
+# Location of the event
+# For online events, just the 'name' is enough
+location:
+  name: SURF Amsterdam   # can be e.g. "Online" for online
+  address: Science Park 140
+  city: Amsterdam
+  country: The Netherlands
+  postcode: 1098 XG
+
+
+
+# People involved, organisers, speakers, funders, etc
+# Must be defined in CONTRIBUTORS.yaml file
+contributions:
+  organisers:
+  - mirelaminkova
+  instructors:
+  - mirelaminkova
+  - yuliiaorlova
+  funding:
+  - surf 
+
+
+# Galaxy and other infrastructure that will be used for your event.
+# This will be used to create the setup instructions for participants
+infrastructure:
+  tiaas: false    # tiaas = Training Infrastructure as a Service, and can be requested (for free) from all major Galaxies
+  custom:  # optional, any other setup instructions you want to add to the "Setup" tab
+    description: |
+      Before joining the course, please make sure to:
+        - Bring a laptop with at least 8GB of RAM.
+        - Do the [Intro to Galaxy](http://training.galaxyproject.org/topics/introduction/tutorials/galaxy-intro-short/tutorial.html) tutorial if you are not yet familiar with Galaxy
+        - Up-to-date browser
+        - Experience in using SURF Research Cloud 
+
+# Program of your course
+# Add GTN tutorials by supplying the topic and tutorial name
+# For non-GTN sessions, add a "type:custom" session and description
+program:
+  - section: "Galaxy & Pulsar on SURF"  # section title is optional
+    description: |
+      During the workshop, you will see a short demo and get a chance to test the catalog item yourself.
+    tutorials:
+      - name: SURF SRC introduction
+        type: custom
+        time: "13:00 - 13:40"
+
+      - name: surf-research-cloud-galaxy
+        topic: admin
+        time: "13:40 - 14:10"
+
+      - name: surf-research-cloud-pulsar
+        topic: admin
+        time: "14:10 - 14:40"
+
+      - type: custom
+        name: Coffee Break
+        time: "14:40"
+      - name: mapping
+        type: custom
+        time: "14:40 - 15:40"
+
+      - name: an-introduction-to-genome-assembly
+        type: custom
+        time: "15:40 - 16:10"
+
+      - type: custom
+        name: General feedback from participants 
+        time: "16:10"
+
+      - type: custom
+        name: Borreltje
+        time: "16:30 - 17:00"
+
+---
+
+During the workshop, you will see a short demo and get a chance to test the catalog item yourself. This will be followed by a discussion about your needs as Galaxy researchers, helping us work toward a better future for the Dutch research community. We will conclude the day with drinks and networking with fellow Galaxy NL users.

topics/admin/tutorials/surf-research-cloud-galaxy/tutorial.md

@@ -0,0 +1,182 @@
+---
+layout: tutorial_hands_on
+
+title: "Galaxy usage on SURF Research Cloud"
+zenodo_link: ""
+questions:
+  - How do I start a Galaxy instance on SURF Research Cloud?
+  - How do I link to external storage from my Galaxy workspace?
+  - How do I install tools?
+objectives:
+  - Understand how to work with a Galaxy on-demand instance in SURF Research Cloud
+requirements:
+  - type: "none"
+    title: Access to the SURF Research Cloud
+  - type: "none"
+    title: SRAM authentication
+  - type: "none"
+    title: An SSH key connected to your account
+  - type: "none"
+    title: Have a basic understanding of how Galaxy works 
+time_estimation: "30m"
+level: Introductory
+contributions:
+  authorship:
+  - mirelaminkova
+  - hexylena
+  - dometto
+  funding:
+  - surf
+
+edam_ontology:
+- topic_0605 # Informatics
+- topic_3071 # Data Management
+
+abbreviations:
+  SRC: SURF Research Cloud
+  GDPR: General Data Protection Regulations
+  SRAM: SURF Research Access Management
+  CO: Collaborative Organisation
+
+follow_up_training:
+  - type: "internal"
+    topic_name: admin
+    tutorials:
+      - surf-research-cloud-pulsar
+---
+
+Using Galaxy via the {SRC} allows researchers to start Galaxy instances on-demand and analyze their data in a secure environment following the {GDPR}. The instance provides secure authentication, where users must have a SURF Research account prior to this tutorial, have set the {SRAM} authentication method, and connect an SSH key to their accounts. In case you are not familiar with {SRC} and need help in setting up your accounts, please follow the instructions on the [SURF Knowledge Base](https://servicedesk.surf.nl/wiki/display/WIKI/SURF+Research+Cloud)
+
+In this training the aim is to focus on:
+
+- Understanding how to start Galaxy on {SRC}
+- Attaching external volumes for storage
+
+Galaxy instances can be started and stopped on demand, depending on personal cases and requirements. Inside the SRC members should have access to all publicly available catalog items. If you are not able to create a catalog item, please [contact SURF servicedesk](mailto:[email protected]).
+
+> <agenda-title></agenda-title>
+>
+> 1. TOC
+> {:toc}
+>
+{: .agenda}
+
+
+# Prerequisites
+
+This tutorial assumes you are member of a {CO} in {SRAM} that has access to {SRC} and a wallet with budget in SRC with enough sources to create Galaxy and Pulsar catalog items. (For more information please refer to the [SURF Knowledge Base](https://servicedesk.surf.nl/wiki/display/WIKI/Budgets%2C+wallets%2C+contracts).
+
+You must have previous experience working with data inside Galaxy.
+
+# Starting a Galaxy instance inside SRC step-by-step
+
+> <hands-on-title>Access the SRC</hands-on-title>
+> 1. The first and most important step is to have access to the SURF Research Cloud. 
+> 2. You will need to login to the [portal](https://portal.live.surfresearchcloud.nl).
+>    ![login screen for SURF research cloud](./images/1.png)
+>
+> 3. Assuming you are a Dutch researcher, you should expect to see your educational institution listed on the authentication provider screen.
+>    ![login with screen, showing an authentication provider being chosen.](./images/2.png)
+> 4. Log in with your institute credentials and follow the SRAM authentication on your phone. Inside the environment, you can see if there are any active virtual machines, the options to create a workspace, new storage or request new wallet for your projects.
+>    ![SRC dashboard, options like create new workspace, storage, or request a new wallet are available. one workspace is listed as running: galaxy test tim.](./images/3.png)
+{: .hands_on}
+
+With this, you're ready to get started launching resources within {SRC}.
+
+## Create an external storage
+
+The purpose of the storage is to attach it to your workspace, where you will save your data. This way, if the machine is deleted, your data will not be lost. We strongly advise anyone using the SRC to create storages and attach them to their machines!
+
+
+> <hands-on-title>Create a Storage Volume</hands-on-title>
+> 1. Locate the "Create new storage" option 
+>    ![top half of the SRC dashboard showing options like create new workspace, storage, or request a new wallet.](./images/4.png)
+> 2. Click on "Create new".
+> 3. Select the collaborative organisation you want the storage to be part of.
+> 4. Select the desired cloud provider and the size of the volume you want to use.
+>    > <tip-title>Unsure of the size required?</tip-title>
+>    > If you are unsure about the size of the storage you need, contact your administrator. Consider also consulting [Galaxy from an Administrator's Point of View]({% link topics/admin/tutorials/introduction/slides.html %}#10)
+>    {: .tip}
+>
+>    ![create your storage dashboard showing a selection of providers like surf and aws and azure, and then a selection of storage sizes below ranging from 5GB to 1.5TB](./images/5.png)
+> 
+> 6. Name your storage however you like and press on submit.
+>
+>    ![form to provide name and preview what will be created.](./images/6.png)
+>
+> 7. You will be redirected to a the main page, where you will be able to track the status of the storage creation. 
+>
+>    ![bottom half of home page showing the Storage tab, and a resource named storage-galaxy in state creating.](./images/7.png)
+{: .hands_on}
+
+Once the storage is deployed, you can attach it to any of your machines! 🎉
+
+Be aware, that in order to attach an external storage to an existing machine, you would need to pause the workspace first! (If you are not sure how to do that please refer to [External storage volumes](https://servicedesk.surf.nl/wiki/display/WIKI/External+storage+volumes))
+
+# Create a new workspace
+
+Depending on user’s rights, you can create a new workspace for your collaboration.
+
+> <tip-title>A Workspace?</tip-title>
+> Workspaces may also be known as *Virtual Machines/VMs* or *Instances* in other cloud providers.
+{: .tip}
+
+> <hands-on-title>Create a Workspace</hands-on-title>
+> 1. On the right side under the "Workspaces" click on the "Add" button.
+>
+>    ![dashboard of workspaces showing a single running workspace titled galaxy test tim, and on the right a button to add a new workspace.](./images/8.png)
+> 
+> 2. Then, you will be redirected to a new page, where you must first choose the collaborative organisation in which you want to create your workspace (in case you are a member of multiple organisations). Once chosen, you a new page will be loaded, where you will have access to all available catalog items. 
+>
+>    ![Create your workspace screen showing 9 catalog items on the first page with pages more. Visible are Galaxy Pulsar node amongst some docker and ubuntu options.](./images/9.png)
+> 
+> 3. Use the magnifying glass on the right side of the panel and search for Galaxy. Two catalog items will appear - the Galaxy instance designed for SURF and a Galaxy Pulsar node that can be connected to the instance. Select "Galaxy at SURF".
+>
+>    ![similar to previous image, but the items described above](./images/10.png)
+> 
+> 4. SURF Research Cloud allows researchers to host their catalog items on different cloud providers. The Galaxy catalog item is currently supported only on the HPC Cloud and for Ubuntu 22.04. On this page, users can select the number of cores and RAM that they want on their machine. More sizes can be added in the future, and on request. Choose wisely and in case you are not certain contact your administrator! 
+>
+>    ![workspace creation screen, choose the cloud provider is locked to SURF HPC, flavour is locked to 22.04, and the size options are available from 1GB/8CPU to 60C/750GB Hi-mem](./images/11.png)
+> 
+> 5. Select the storage you have created earlier so it is attached to the new workspace. 
+>
+>    ![create your workspace screen, storage only lists one option named storage-galaxy](./images/12.png)
+> 
+> 6. Lastly, before the workspace is deployed, you need to choose for how long the machine will run. 
+>
+>    > <tip-title>Expiration Date</tip-title>
+>    > The standard life-time of the VM is 5 days. If you need it for longer, this option can be changed once the machine is running.
+>    > Note, that once the machine is expired and deleted it cannot be restored! Plan accordingly and migrate your data in time to prevent data loss!
+>    >
+>    > This is an incredibly useful feature as it saves you from forgetting to destroy a VM. Especially for GPU nodes it can help you ensure that they disappear after your computation is complete.
+>    {:.tip}
+> 
+> 7. In this form, you can also select how to name your workspace, add a description (if you want) and specify the hostname. Then, scroll down and submit the workspace creation.
+>    ![almost there, some final details reads the heading. an expiration date is set and galaxy at surf is provided in the name. the hostname is set to galaxyatsurf and the description is blank](./images/13.png)
+{: .hands_on}
+
+With that, your workspace will start being created.
+
+![A galaxy instance named galaxy demo is in status creating.](./images/14.png "Creating the workspace can take some time. You will see on the main page the status of your workspace.")
+
+> <tip-title>Also launching Pulsar?</tip-title>
+> While you are waiting, you can create your Galaxy Pulsar node simultaneously to save time.
+{: .tip}
+
+Once the workspace is running, press on the down arrow to check any information such as the ID, the owner of the workspace, the collaboration it is part of, when it is created and when it expires and the wallet it is using. Additionally, you can pause the workspace when you are not using it. Pressing on the storage option will display the storage that has been attached to the workspace.
+
+![the details panel of the galaxy demo instance is shown including fields like an id, owner, date created and expiration date, url., etc](./images/15.png)
+
+To access the Galaxy instance, press on the "Access" button. This will redirect you to a new webpage. The {SRAM} login will be displayed again, and after authenticating, you will be redirected to the Galaxy at SURF Research Cloud page!
+
+![A galaxy instance homepage is shown, with a brand indicating that it was deployed on surf research cloud featuring a cute surfer emoji.](./images/16.png)
+
+Congratulations! You have started a Galaxy instance inside the SURF Research Cloud! 🏄‍♀️🎉
+
+If you have administrator rights, you can download tools that can be used by the members of the collaborative organisation. To obtain administrator rights you need to be part of the `src_co_admin` group inside the collaborative organisation. In case you are not sure how to download tools as a Galaxy administrator, please refer to [Installing Tools into Galaxy](https://galaxyproject.org/admin/tools/add-tool-from-toolshed-tutorial/).
+
+## Going Further with Pulsar?
+
+Another important step when starting a local Galaxy instance on {SRC} is connecting it to a Galaxy Pulsar Node, which you can do [in this Pulsar SRC tutorial]({% link topics/admin/tutorials/surf-research-cloud-pulsar/tutorial.md %}). The Pulsar is an useful component that is used to execute your tasks on remote servers/clusters. In this case, the Pulsar is started as a new workspace, allowing users to also run interactive tools.
+
+In SRC, this can also enable you to save costs. By choosing a smaller Galaxy instance, and attaching a large Pulsar node when you need to do computations, you can be sure to optimise your budget expenditures.