Merge branch 'main' into patch-3

bih-cluster/docs/admin/maintenance.md

@@ -2,6 +2,10 @@
 
 This page documents the current and known upcoming maintenance windows.
 
+## Login & Compute maintenance, October 22nd, 2024
+
+Kernel and Slurm upgrades. Keep an eye on [our forum](https://hpc-talk.cubi.bihealth.org/t/maintenance-slurm-upgrade-october-22nd/1095) for updates. 
+
 ## Login, Compute and Storage Maintenance, December 13-14, 2022
 
 All informationand updates regarding maintenance will be circulated on our forum https://hpc-talk.cubi.bihealth.org/c/announcements/5.

bih-cluster/docs/connecting/advanced-ssh/overview.md

@@ -2,5 +2,5 @@
 Here we describe custom scenarios for using SSH to connect to BIH HPC.
 To keep it consise, this section is divided into separate documents for
 
-- [Linux](./linux.md) and
+- [Linux](./unix.md) and
 - [Windows](./windows.md) users.

bih-cluster/docs/connecting/advanced-ssh/linux.md → bih-cluster/docs/connecting/advanced-ssh/unix.md

@@ -1,4 +1,4 @@
-# Connecting via SSH on Unix
+# Connecting via SSH with Unix-like Operating Systems
 ## Activating your Key in the SSH Key Agent
 !!! note
     The big Linux distributions automatically manage ssh-agent for you and unlock your keys at login time.

bih-cluster/docs/connecting/connecting.md

@@ -15,7 +15,7 @@ In case you're not familiar with SSH, you should probably start via the web port
 Follow these steps to connect to BIH HPC via the command line:
 
 1. [Register an account](../admin/getting-access.md) via your PI. :memo: 
-2. [Generate a SSH key pair :key: in Linux](generate-key/linux.md) or [Windows](generate-key/windows.md)
+2. [Generate a SSH key pair :key: in Linux](generate-key/unix.md) or [Windows](generate-key/windows.md)
 3. [Submit your public key :arrow_up: to Charite](submit-key/charite.md) or [to MDC](submit-key/mdc.md).
 4. Connect to one of the two login nodes.
 
@@ -45,7 +45,7 @@ Follow these steps to connect to BIH HPC via the command line:
     $ srun --pty bash -i
     ```
 
-6. Bonus: [Configure your SSH client :wrench: on Linux and Mac](advanced-ssh/linux.md) or [Windows](advanced-ssh/windows.md).
+6. Bonus: [Configure your SSH client :wrench: on Linux and Mac](advanced-ssh/unix.md) or [Windows](advanced-ssh/windows.md).
 7. Bonus: [Connect from external networks :flying_saucer:](./from-external.md).
 
 !!! tip "tl;dr"

bih-cluster/docs/connecting/generate-key/linux.md → bih-cluster/docs/connecting/generate-key/unix.md

@@ -1,4 +1,4 @@
-# Generating an SSH Key in Linux
+# Generating an SSH Key with Unix-like Operating Systems
 
 - You might already have one, check whether a file `~/.ssh/id_xxx.pub` is present.
 - Otherwise, create key using the following command (marking your key with your email address will make it easier to reidentify your key later on):

bih-cluster/docs/help/faq.md

@@ -387,7 +387,7 @@ In the example above, user1 has one job with one GPU running on hpc-gpu-3, user2
 Also see:
 
 - [Running graphical(X11) applications on Windows](../connecting/advanced-ssh/windows.md#x11)
-- [Running graphical(X11) applications on Linux](../connecting/advanced-ssh/linux.md#x11)
+- [Running graphical(X11) applications on Linux](../connecting/advanced-ssh/unix.md#x11)
 
 ## How can I log into a node outside of the scheduler?
 

bih-cluster/docs/how-to/software/jupyter.md

@@ -40,7 +40,7 @@ If you are running multiple server on one or more nodes, one can separate them b
 
 ## Connecting to the Running Session
 
-This is slightly trickier as we have to create a SSH connection/tunnel with potentially multiple hops in between. The easiest way is probably to configure your `.ssh/config` to automatically route your connection via the login node (and possibly MDC jail). This is described in our [Advanced SSH config documentation](../../connecting/advanced-ssh/linux.md#configure-ssh-client)
+This is slightly trickier as we have to create a SSH connection/tunnel with potentially multiple hops in between. The easiest way is probably to configure your `.ssh/config` to automatically route your connection via the login node (and possibly MDC jail). This is described in our [Advanced SSH config documentation](../../connecting/advanced-ssh/unix.md#configure-ssh-client)
 
 In short,add these lines to `~/.ssh/config` (replace curly parts):
 

bih-cluster/docs/hpc-tutorial/episode-0.md

@@ -44,7 +44,7 @@ That was easy!
 In preparation for our first steps tutorial series, we would like you to install the software for this tutorial.
 In general the users on the cluster will manage their own software with the help of conda.
 If you haven't done so so far, please [follow the instructions in installing conda](../best-practice/software-installation-with-conda.md) first.
-The only premise is that you are able to [log into the cluster](../connecting/advanced-ssh/linux.md).
+The only premise is that you are able to [log into the cluster](../connecting/advanced-ssh/unix.md).
 Make also sure that you are logged in to a computation node using `srun -p medium --time 1-00 --mem=4G --cpus-per-task=1 --pty bash -i`.
 
 Now we will create a new environment, so as to not interfere

bih-cluster/docs/index.md

@@ -28,6 +28,7 @@ Read the following set of pages (in order) to learn how to get access and connec
     Please add your publications using the cluster to [this list](misc/publication-list.md).
 
 ## News & Maintenance Announcements
+- :jack_o_lantern: October 22nd 2024: Kernel & Slurm upgrades on (almost) all nodes.
 - :ram: July 16th: New high-memory node `hpc-mem-5` with 4 TB of RAM.
 - :locomotive: Until autumn 2024: Operation Exodus – Migration of all data from GPFS to CephFS storage.
 - :maple_leaf: September 30th 2024: Unmounting of `/fast` on all non-transfer nodes.

bih-cluster/docs/overview/storage.md

@@ -3,7 +3,7 @@
 !!! info "No mounting **on** the cluster itself."
 
     For various technical and security-related reasons it is not possible to mount anything on the cluster nodes by users.
-    For mounting the cluster storage on your computer, please read [Connecting: SSHFS Mounts](../connecting/advanced-ssh/linux.md#file-system-mount-via-sshfs).
+    For mounting the cluster storage on your computer, please read [Connecting: SSHFS Mounts](../connecting/advanced-ssh/unix.md#file-system-mount-via-sshfs).
 
 This document gives an overview of the nodes and volumes on the cluster.
 

bih-cluster/docs/slurm/snakemake.md

@@ -11,9 +11,10 @@ This page describes how to use Snakemake with Slurm.
 
 We first create a new environment `snakemake-slurm` and activate it.
 We need the `snakemake` package for this.
+For snakemake 8, we additionally need [`snakemake-executor-plugin-slurm`](https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/slurm.html).
 
 ```bash
-host:~$ conda create -y -n snakemake-slurm snakemake
+host:~$ conda create -y -n snakemake-slurm 'snakemake>=8.24.1' snakemake-executor-plugin-slurm
 [...]
 #
 # To activate this environment, use
@@ -40,7 +41,7 @@ rule default:
 
 rule mkresult:
     output: "the-result.txt"
-    shell: r"sleep 1m; touch the-result.txt"
+    shell: r"sleep 2m; touch the-result.txt"
 EOF
 host:snake-slurm$ snakemake --cores=1
 [...]
@@ -51,12 +52,12 @@ host:snake-slurm$ rm the-result.txt
 
 ## Snakemake and :tada: Slurm
 
-You have two options:
+Simply use `snakemake --profile=cubi-v1` and the Snakemake resource configuration as shown below.
+This works for both snakemake version 7 *and* 8.
 
-1. Simply use `snakemake --profile=cubi-v1` and the Snakemake resource configuration as shown below. **STRONGLY PREFERRED**
-2. Use the `snakemake --cluster='sbatch ...'` command.
+(For version 7, you can also use `snakemake --cluster='sbatch ...'` command instead, but this is discouraged.)
 
-Note that we sneaked in a `sleep 1m`? In a second terminal session, we can see that the job has been submitted to SLURM indeed.
+Note that we sneaked in a `sleep 2m`? In a second terminal session, we can see that the job has been submitted to SLURM indeed.
 
 ```bash
 host:~$ squeue  -u holtgrem_c
@@ -78,11 +79,14 @@ The `cubi-v1` profile (stored in `/etc/xdg/snakemake/cubi-v1` on all cluster nod
 
 You will need Snakemake >=7.0.2 for this.
 
+For more information on resources, see [snakemake resources](https://snakemake.readthedocs.io/en/latest/snakefiles/rules.html#standard-resources) and [slurm resources](https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/slurm.html#advanced-resource-specifications).
+
 Here is how to call Snakemake:
 
 ```bash
-# snakemake --profile=cubi-v1 -j1
+# snakemake --profile=cubi-v1 --jobs 1
 ```
+When using conda, additionally use `--sdm conda` (snakemake 8) or `--use-conda` (snakemake 7).
 
 To set rule-specific resources:
 

bih-cluster/mkdocs.yml

@@ -88,14 +88,14 @@ nav:
     - "SSH Basics": connecting/ssh-basics.md
     - "Windows": "connecting/connecting-windows.md"
     - "Generate Keys":
-      - "Linux": connecting/generate-key/linux.md
+      - "Linux/macOS": connecting/generate-key/unix.md
       - "Windows": connecting/generate-key/windows.md
     - "Submit Keys":
       - "Charite": connecting/submit-key/charite.md
       - "MDC": connecting/submit-key/mdc.md
     - "Advanced SSH":
       - "Overview": connecting/advanced-ssh/overview.md
-      - "Linux": connecting/advanced-ssh/linux.md
+      - "Linux/macOS": connecting/advanced-ssh/unix.md
       - "Windows": connecting/advanced-ssh/windows.md
     - "From External": connecting/from-external.md
     - "Connection Problems": connecting/connection-problems.md