forward

Lab-specific recommendations

1. Follow the instructions in "Setup" below to configure your params.sh script and sherlock ssh config. IMPORTANT: if two people who are using forward at the same time have the same port number, there will be issues launching the job. I would recommend choosing a random port number in the range specified when setting up params.sh.
2. Copy load_modules.sh from this repo to $HOME/bin on sherlock (e.g., using scp)
3. You might also want to copy the .bashrc from this repo. It defines helpful aliases and automatically calls load_modules.sh when you log in to Sherlock.
4. Then you should be all set to launch a jupyter notebook by running bash start.sh jupyter <cluster-dir> where cluster-dir is the directory you want the notebook to be executed from. In this command, jupyter will reference the file /sbatches/jupyter.sbatch which requests CPU/GPU resources (you can change this).
5. Don't forget to stop the notebook when you're done by running bash end.sh jupyter. You can resume a notebook if your computer goes to sleep and port forwarding stops by running bash resume.sh jupyter.
6. Ideally only use an A100 if you really need it for your computations. Otherwise, try to connect to cpu.

So far, I've been running my notebooks on the swl1 partition, but if we start having resource conflicts then you may have to change params.sh to use another partition like owners or gpu.

Running forward with virtual environments (written by Amber and Xavier)

Here is a sort of hacky way to get jupyter notebooks running inside a virtual environment on Sherlock. If anyone has another way to do this without having to automatically load one virtual environment, please contribute!

Steps:

• Create a virtual environment (ideally in GROUP HOME, but can also do HOME)
• In your bin/activate file for your venv, you should include these lines to make pip actually install in your venv (see the best_practices.md file in the lab manual)

# aliases to make pip work
alias pip="PYTHONUSERBASE=<path-to-your-venv>/python pip"
export PYTHONPATH=$<path-to-your-venv>/python/lib/python3.9/site-packages:$PYTHONPATH
export PATH=<path-to-your-venv>/python/bin:$PATH

• pip install jupyter in the virtual environment
• In your load_modules.sh, include the line source <PATH_TO_VENV>/bin/activate. This will automatically activate your venv when you log into Sherlock. Note that this will only work if in your .bashrc you are calling source load_modules.sh and also note that you need to be loading the correct venv that you want to use!
• Then when you launch forward, your notebook should automatically connect to the venv you've created!

What is this?

Forward sets up an sbatch script on your cluster resource and port forwards it back to your local machine! Useful for jupyter notebook and tensorboard, amongst other things.

• start.sh is intended for submitting a job and setting up ssh forwarding
• start-node.sh will submit the job and give you a command to ssh to the node, without port forwarding

The folder sbatches contains scripts, organized by cluster resource, that are intended for use and submission. It's up to you to decide if you want a port forwarded (e.g., for a jupyter notebook) or just an instruction for how to connect to a running node with your application.

Tiny Tutorials

Here we will provide some "tiny tutorials" to go along with helping to use the software. These are tiny because there are many possible use cases!

• Jupyter with GPU on Sherlock: A recipe for interactive computing using custom Jupyter kernels on Stanford's Sherlock.
• Using sherlock/py3-jupyter and copying notebook first from your host to use a notebook module (python 3) on the Sherlock cluster at Stanford Version 0.0.1.
• Running an R Kernel in a Jupyter Notebook
• Using containershare with repo2docker-julia a repo2docker-julia Singularity container deployed on Sherlock using Version 0.0.1

Setup

For interested users, a few tutorials are provided on the Research Computing Lessons site. Brief instructions are also documented in this README.

For farmshare - please navigate to the README located in sbatches/farmshare/README.md.

Clone the Repository

Clone this repository to your local machine.

You will then need to create a parameter file. To do so, follow the prompts at:

bash setup.sh

You can always edit params.sh later to change these configuration options.

Parameters

• RESOURCE should refer to an identifier for your cluster resource that will be recorded in your ssh configuration, and then referenced in the scripts to interact with the resource (e.g., ssh sherlock).
• PARTITION If you intend to use a GPU (e.g., sbatches/py2-tensorflow.sbatch the name of the PARTITION variable should be "gpu."
• CONTAINERSHARE (optional) is a location on your cluster resource (typically world readable) where you might find containers (named by a hash of the container name in the library that are ready to go! If you are at Stanford, leave this to be default. If you aren't, then ask your cluster admin about setting up a containershare
• CONNECTION_WAIT_SECONDS refers to how many seconds the start.sh script waits before setting up port forwarding. If your cluster runs slow, or is particularly busy, this can be set at 30.

If you want to modify the partition flag to have a different gpu setup (other than --partition gpu --gres gpu:1) then you should set this entire string for the partition variable.

SSH config

You will also need to at the minimum configure your ssh to recognize your cluster (e.g., sherlock) as a valid host. We have provided a hosts folder for helper scripts that will generate recommended ssh configuration snippets to put in your ~/.ssh/config file. Based on the name of the folder, you can intuit that the configuration depends on the cluster host. Here is how you can generate this configuration for Sherlock:

bash hosts/sherlock_ssh.sh

Host sherlock
    User put_your_username_here
    Hostname sh-ln01.stanford.edu
    GSSAPIDelegateCredentials yes
    GSSAPIAuthentication yes
    ControlMaster auto
    ControlPersist yes
    ControlPath ~/.ssh/%l%r@%h:%p

Using these options can reduce the number of times you need to authenticate. If you don't have a file in the location ~/.ssh/config then you can generate it programatically:

bash hosts/sherlock_ssh.sh >> ~/.ssh/config

Do not run this command if there is content in the file that you might overwrite! One downside is that you will be foregoing sherlock's load balancing since you need to be connecting to the same login machine at each step.

SSH Port Forwarding Considerations

Depending on your cluster, you will need to identify whether the compute nodes (not the login nodes) are isolated from the outside world or not (i.e can be ssh'd into directly). For Sherlock, they are isolated. For FarmShare they are not. This is important when we are setting up the ssh command to port forward from the local machine to the compute node.

For HPC's where the compute node is isolated from the outside world (as is the case with sherlock), the ssh command basically establishes a tunnel to the login node, and then from the login node establishes another tunnel to the compute node. In this case we write a command where we port forward to the login node, and then the compute node, which is accessible from the login node. The entire command might look like this:

$ ssh -L $PORT:localhost:$PORT ${RESOURCE} ssh -L $PORT:localhost:$PORT -N "$MACHINE" &

In the command above, the first half is executed on the local machine ssh -L $PORT:localhost:$PORT ${RESOURCE}, which establishes a port forwarding to the login node. The next line ssh -L $PORT:localhost:$PORT -N "$MACHINE" & is run from the login node, and port forwards it to the compute node, since you can only access the compute node from the login nodes.

For HPC's where the compute node is not isolated from the outside world (as is the case with Farmshare) the ssh command for port forwarding first establishes a connection the login node, but then continues to pass on the login credentials to the compute node to establish a tunnel between the localhost and the port on the compute node. The ssh command in this case utilizes the flag -K which forwards the login credentials to the compute node:

$ ssh "$DOMAINNAME" -l $FORWARD_USERNAME -K -L  $PORT:$MACHINE:$PORT -N  &

The drawback of this method is that when the start.sh script is run, you will have to authenticate twice (once at the beginning to check if a job is running on the HPC, and when the port forwarding is setup). This is the case for FarmShare.

In the setup.sh file, we have added an option $ISOLATECOMPUTENODE, which is a boolean operator. For users of FarmShare, and Sherlock, this value is set automatically. For your own default cluster, you will be prompted whether the compute node is isolated or not, please write true or false (case sensitive) for your resource depending on its properties. You may have to consult the documentation or ask the HPC manager.

Notebooks

Notebooks have associated sbatch scripts that are intended to start a jupyter (or similar) notebook, and then forward the port back to your machine. If you just want to submit a job, (without port forwarding) see the job submission section. For notebook job submission, you will want to use the start.sh script.

Notebook password

If you have not set up notebook authentication before, you will need to set a password via jupyter notebook password on your cluster resource.
Make sure to pick a secure password!

Job Submission

Job submission can mean executing a command to a container, running a container, or writing your own sbatch script (and submitting from your local machine). For standard job submission, you will want to use the start-node.sh script. If your cluster has a containershare, you can use the containershare-notebook set of scripts to have a faster deployment (without needing to pull).

Usage

# Choose a containershare notebook, and launch it! On Sherlock, the containers are already in the share
bash start.sh sherlock/containershare-notebook docker://vanessa/repo2docker-julia

# Run a Singularity container that already exists on your resource (recommended)
bash start-node.sh singularity-run /scratch/users/vsochat/share/pytorch-dev.simg

# Execute a custom command to the same Singularity container
bash start-node.sh singularity-exec /scratch/users/vsochat/share/pytorch-dev.simg echo "Hello World"

# Run a Singularity container from a url, `docker://ubuntu`
bash start-node.sh singularity-run docker://ubuntu

# Execute a custom command to the same container
bash start-node.sh singularity-exec docker://ubuntu echo "Hello World"

# Execute your own custom sbatch script
cp myscript.job sbatches/
bash start-node.sh myscript

As a service for Stanford users, @vsoch provides a containershare of ready to go containers to use on Sherlock! The majority of these deploy interactive notebooks, however can also be run without (use start-node.sh instead of start.sh). If you want to build your own container for containershare (or request a container) see the README in the repository that serves it.

# Run a containershare container with a notebook
bash start.sh sherlock/containershare-notebook docker://vanessa/repo2docker-julia

If you would like to request a custom notebook, please reach out.

Usage

# To start a jupyter notebook in a specific directory ON the cluster resource
bash start.sh jupyter <cluster-dir>

# If you don't specify a path on the cluster, it defaults to your ${SCRATCH}
bash start.sh jupyter /scratch/users/<username>

# To start a jupyter notebook with tensorflow in a specific directory
bash start.sh py2-tensorflow <cluster-dir>

# If you want a GPU node, make sure your partition is set to "gpu."
# To start a jupyter notebook (via a Singularity container!) in a specific directory
bash start.sh singularity-jupyter <cluster-dir>

Want to create your own Singularity jupyter container? Use repo2docker and then specify the container URI at the end.

bash start.sh singularity.jupyter <cluster-dir> <container>

# You can also run a general singularity container!
bash start.sh singularity <cluster-dir> <container>

# To start tensorboard in a specific directory (careful here and not recommended, as is not password protected)
bash start.sh start <cluster-dir>

# To stop the running jupyter notebook server
bash end.sh jupyter

If the sbatch job is still running, but your port forwarding stopped (e.g. if your computer went to sleep), you can resume with:

bash resume.sh jupyter`

Debugging

Along with some good debugging notes here, common errors are below.

Connection refused after start.sh finished

Sometimes you can get connection refused messages after the script has started up. Just wait up to a minute and then refresh the opened web page, and this should fix the issue.

Terminal Hangs when after start.sh

Sometimes when you have changes in your network, you would need to reauthenticate. In the same way you might get a login issue here, usually opening a new shell resolves the hangup.

Terminal Hangs on "== Checking for previous notebook =="

This is the same bug as above - this command specifically is capturing output into a variable, so if it hangs longer than 5-10 seconds, it's likely hit the password prompt and would hang indefinitely. If you issue a standard command that will re-prompt for your password in the terminal session, you should fix the issue.

$ ssh sherlock pwd

slurm_load_jobs error: Socket timed out on send/recv operation

This error is basically saying something to the effect of "slurm is busy, try again later." It's not an issue with submitting the job, but rather a ping to slurm to perform the check. In the case that the next ping continues, you should be ok. However, if the script is terminate, while you can't control the "busyness" of slurm, you can control how likely it is to be allocated a node, or the frequency of checking. Thus, you can do either of the following to mitigate this issue:

choose a partition that is more readily available

In your params.sh file, choose a partition that is likely to be allocated sooner, thus reducing the queries to slurm, and the chance of the error.

offset the checks by changing the timeout between attempts

The script looks for an exported variable, TIMEOUT and sets it to be 1 (1 second) if not defined. Thus, to change the timeout, you can export this variable:

export TIMEOUT=3

While the forward tool cannot control the busyness of slurm, these two strategies should help a bit.

mux_client_forward: forwarding request failed: Port forwarding failed

Similarly, if your cluster is slow, you may get this error after "== Setting up port forwarding ==". To fix this, increase your CONNECTION_WAIT_SECONDS.

I ended a script, but can't start

As you would kill a job on Sherlock and see some delay for the node to come down, the same can be try here! Try waiting 20-30 seconds to give the node time to exit, and try again.

How do I contribute?

First, please read the contributing docs. Generally, you will want to:

• fork the repository to your username
• clone your fork
• checkout a new branch for your feature, commit and push
• add your name to the CONTRIBUTORS.md
• issue a pull request!

Adding new sbatch scripts

You can add more sbatch scripts by putting them in the sbatches directory.