adding farmshare, still under development

README.md

@@ -13,27 +13,25 @@ for use and submission. It's up to you to decide if you want a port forwarded (e
 or just an instruction for how to connect to a running node with your application.
 
 ## Setup
-For interested users, a few tutorials are provided:
-
- - [sherlock jupyter](https://vsoch.github.io/lessons/sherlock-jupyter/) 
- - [sherlock tensorflow](https://vsoch.github.io/lessons/jupyter-tensorflow/)
- - [sherlock singularity jupyter](https://vsoch.github.io/lessons/sherlock-singularity)
-
+For interested users, a few tutorials are provided on the [Research Computing Lessons](https://vsoch.github.io/lessons) site.
 Brief instructions are also documented in this README.
 
 ### Clone the Repository
 Clone this repository to your local machine.
 
 You will then need to create a parameter file.  To do so, follow the prompts at:
 
-`bash setup.sh`
+```bash
+bash setup.sh
+```
 
-You can always edit params.sh later to change these configuration options. 
+You can always edit `params.sh` later to change these configuration options. 
 
 #### Parameters
 
  - **RESOURCE** should refer to an identifier for your cluster resource that will be recorded in your ssh configuration, and then referenced in the scripts to interact with the resource (e.g., `ssh sherlock`).
  - **PARTITION** If you intend to use a GPU (e.g., [sbatches/py2-tensorflow.sbatch](sbatches/py2-tensorflow.sbatch) the name of the PARTITION variable should be "gpu."
+ - **CONTAINERSHARE** (optional) is a location on your cluster resource (typically world readable) where you might find containers (named by a hash of the container name in the [library]() that are ready to go! If you are at Stanford, leave this to be default. If you aren't, then ask your cluster admin about [setting up a containershare](https://www.github.com/vsoch/containershare)
 
 If you want to modify the partition flag to have a different gpu setup (other than `--partition gpu --gres gpu:1`) then you should set this **entire** string for the partition variable.
 
@@ -72,6 +70,7 @@ balancing since you need to be connecting to the same login machine at each
 step.
 
 # Notebooks
+
 Notebooks have associated sbatch scripts that are intended to start a jupyter (or similar)
 notebook, and then forward the port back to your machine. If you just want to submit a job,
 (without port forwarding) see [the job submission](#job-submission) section. For 
@@ -88,11 +87,15 @@ Make sure to pick a secure password!
 Job submission can mean executing a command to a container, running a container, or 
 writing your own sbatch script (and submitting from your local machine). For 
 standard job submission, you will want to use the [start-node.sh](start-node.sh) script.
-For a notebook type job (anything with jupyter and 
+If your cluster has a containershare, you can use the `containershare-notebook`
+set of scripts to have a faster deployment (without needing to pull).
 
 ## Usage
 
 ```bash
+# Choose a containershare notebook, and launch it! On Sherlock, the containers are already in the share
+bash start.sh sherlock/containershare-notebook docker://vanessa/repo2docker-julia
+
 # Run a Singularity container that already exists on your resource (recommended)
 bash start-node.sh singularity-run /scratch/users/vsochat/share/pytorch-dev.simg
 

sbatches/farmshare/README.md

@@ -0,0 +1,3 @@
+# Farmshare
+
+Hi friend! These haven't been tested fully yet. Do you want to help? Please work with @vsoch!

sbatches/farmshare/singularity-exec.sbatch

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+# The singularity-run script will run a container, and it would use the
+# container's runscript as entrypoint. To execute a command, 
+# use singularity-exec
+
+CONTAINER=${1}
+shift
+
+if [ "$#" -gr 0 ]; then
+    NOTEBOOK_DIR=${1}
+    shift 1
+else
+    NOTEBOOK_DIR=${SCRATCH}
+fi
+
+cd $NOTEBOOK_DIR
+
+export SINGULARITY_CACHEDIR=/farmshare/user_data/${USERNAME}/.singularity
+singularity exec ${CONTAINER} "${@}"

sbatches/farmshare/singularity-jupyter.sbatch

@@ -0,0 +1,33 @@
+#!/bin/bash
+
+# Usage
+
+# 1. Default Jupyter notebook (with your scratch to work in)
+# $ bash start.sh singularity-jupyter
+
+# 2. Default Jupyter notebook with custom working directory
+# $ bash start.sh singularity-jupyter /scratch/users/<username>
+
+# 3. Select your own jupyter container on Sherlock!
+# $ bash start.sh singularity-jupyter /scratch/users/<username> /path/to/container
+
+# 4. Or any singularity container...
+# $ bash start.sh singularity /path/to/container <args>
+
+PORT=$1
+NOTEBOOK_DIR=${2:-${SCRATCH}}
+CONTAINER=${3:-docker://vanessa/repo2docker}
+
+export SINGULARITY_CACHEDIR=/farmshare/user_data/${USERNAME}/.singularity
+echo "Container is ${CONTAINER}"
+echo "Notebook directory is ${NOTEBOOK_DIR}"
+cd ${NOTEBOOK_DIR}
+
+# Create .local folder for default modules, if doesn't exist
+if [ ! -d "${HOME}/.local" ];
+  then
+    echo "Creating local python modules folder to map at ${HOME}/.local";
+    mkdir -p "${HOME}/.local";
+fi
+
+singularity exec --home ${HOME} --bind ${HOME}/.local:/home/username/.local ${CONTAINER} jupyter notebook --no-browser --port=$PORT --ip 0.0.0.0

sbatches/farmshare/singularity-run.sbatch

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+# The singularity-run script will run a container, and it would use the
+# container's runscript as entrypoint. To execute a command, 
+# use singularity-exec
+
+CONTAINER=${1}
+shift
+
+if [ "$#" -gr 0 ]; then
+    NOTEBOOK_DIR=${1}
+    shift 1
+else
+    NOTEBOOK_DIR=${SCRATCH}
+fi
+
+cd $NOTEBOOK_DIR
+
+export SINGULARITY_CACHEDIR=/farmshare/user_data/${USERNAME}/.singularity
+singularity run ${CONTAINER} "${@}"

sbatches/sherlock/containershare-notebook.sbatch

@@ -1,26 +1,51 @@
 #!/bin/bash
 
-# This batch script is intended to run a containershare container, typically
-# deploying a Docker container as Singularity from the containershare
-# registry at https://vsoch.github.io/containershare. You can select
-# containers that have jupyter notebooks (e.g., julia, jupyter, or similar)
-# Open an issue there to request a custom container.
+# Run a Singularity container notebook, meaning content is in /home/joyvan and
+# your $HOME/.local is mounted to the container's $HOME/.local to make
+# extra modules available.
 
-# Usage
+# --- Usage
+# bash start.sh sherlock/singularity-notebook <container>
 
-# 1. Containershare Julia (with your scratch to work in)
-# $ bash start.sh containershare-notebook docker://vanessa/repo2docker-julia
+# --- <container>
+# This batch script is intended to run a singularity notebook container,
+# and this can mean:
+
+# A Singularity container image FILE on your scratch
+# $ bash start.sh sherlock/singularity-notebook /scratch/users/vsochat/julia.simg
+
+# A Docker unique resource identifier
+# $ bash start.sh sherlock/singularity-notebook docker://<username>/<repository>:<tag>
+
+# A Singularity Hub unique resource identifier
+# $ bash start.sh sherlock/singularity-notebook shub://<username>/<repository>:<tag>
+
+# --- Container Expectations
+# In all cases, it's expected that the notebook is in /home/joyvan (jupyter
+# standard). Your local installation directory (at $HOME/.local) will be 
+# mapped to the container so that modules you have installed locally will 
+# be usable in the notebook.
+
+# --- Containershare: You can select
+# If you need a set of notebooks to use, or bootstrap, check out the 
+# containershare provided by @vsoch: https://vsoch.github.io/containershare 
+# These are all Docker containers that are used by this tool by pulling
+# them onto the cluster as Singularity containers with have jupyter notebooks 
+# (e.g., julia, jupyter, or similar)
+
+# Please Open an issue there to request a custom container! :)
 
 # Ensure we have at least port and container
 if (( $# < 2 )); then
     echo "Please provide minimally PORT and CONTAINER"
-    echo "containershare.sbatch <port> <container>"
+    echo "singularity-notebook.sbatch <port> <container>"
     exit 1
 fi
 
 PORT=$1
 CONTAINER="${2}"
 NOTEBOOK_DIR="${3:-${SCRATCH}}"
+CONTAINERSHARE=${4:-/scratch/users/vsochat/share}
 
 module use system
 module load singularity
@@ -32,15 +57,30 @@ cd ${NOTEBOOK_DIR}
 # If it's not a file, try pulling it
 if [ ! -f "${CONTAINER}" ]
     then
-    echo "Container ${CONTAINER} not found on filesystem, attempting pull..."
+
+    # The container name is the hash of the string
     CONTAINER_NAME=$(echo -n "${CONTAINER}" | md5sum | awk '{ print $1 }').simg
 
-    # Pull the container, if it doesn't exist.
-    if [ ! -f "${SINGULARITY_CACHEDIR}/${CONTAINER_NAME}" ]
+    # Attempt 1: look in the containershare
+    echo "Attempt 1: Looking for ${CONTAINERSHARE}/${CONTAINER_NAME}"
+    if [ -f "${CONTAINERSHARE}/${CONTAINER_NAME}" ]
         then
-        singularity pull --name "${CONTAINER_NAME}" "${CONTAINER}"
+        echo "Container ${CONTAINER} found in containershare!"
+        CONTAINER="${CONTAINERSHARE}/${CONTAINER_NAME}"
+    else
+
+        # Attempt 2: Look in the user's cache
+        if [ ! -f "${SINGULARITY_CACHEDIR}/${CONTAINER_NAME}" ]
+            then
+            echo "Attempt 2: Container ${CONTAINER} not found on filesystem, attempting pull..."
+            echo "singularity pull --name ${CONTAINER_NAME} ${CONTAINER}"
+            singularity pull --name "${CONTAINER_NAME}" "${CONTAINER}"
+        fi
+
+        # Pull the container, if it doesn't exist.
+        CONTAINER="${SINGULARITY_CACHEDIR}/${CONTAINER_NAME}"
+
     fi
-    CONTAINER="${SINGULARITY_CACHEDIR}/${CONTAINER_NAME}"
 fi
 
 # If still doesn't exist, exit

sbatches/sherlock/singularity-notebook.sbatch

@@ -0,0 +1,77 @@
+#!/bin/bash
+
+# Run a Singularity container notebook, meaning content is in /home/joyvan and
+# your $HOME/.local is mounted to the container's $HOME/.local to make
+# extra modules available.
+
+# --- Usage
+# bash start.sh sherlock/singularity-notebook <container>
+
+# --- <container>
+# This batch script is intended to run a singularity notebook container,
+# and this can mean:
+
+# A Singularity container image FILE on your scratch
+# $ bash start.sh sherlock/singularity-notebook /scratch/users/vsochat/julia.simg
+
+# A Docker unique resource identifier
+# $ bash start.sh sherlock/singularity-notebook docker://<username>/<repository>:<tag>
+
+# A Singularity Hub unique resource identifier
+# $ bash start.sh sherlock/singularity-notebook shub://<username>/<repository>:<tag>
+
+# --- Container Expectations
+# In all cases, it's expected that the notebook is in /home/joyvan (jupyter
+# standard). Your local installation directory (at $HOME/.local) will be 
+# mapped to the container so that modules you have installed locally will 
+# be usable in the notebook.
+
+# Ensure we have at least port and container
+if (( $# < 2 )); then
+    echo "Please provide minimally PORT and CONTAINER"
+    echo "singularity-notebook.sbatch <port> <container>"
+    exit 1
+fi
+
+PORT=$1
+CONTAINER="${2}"
+NOTEBOOK_DIR="${3:-${SCRATCH}}"
+
+module use system
+module load singularity
+export SINGULARITY_CACHEDIR="${SCRATCH}/.singularity"
+echo "Container is ${CONTAINER}"
+echo "Notebook directory is ${NOTEBOOK_DIR}"
+cd ${NOTEBOOK_DIR}
+
+# If it's not a file, try pulling it
+if [ ! -f "${CONTAINER}" ]
+    then
+
+    # Attempt 1: look in the containershare
+    echo "Container ${CONTAINER} not found on filesystem, attempting pull..."
+    CONTAINER_NAME=$(echo -n "${CONTAINER}" | md5sum | awk '{ print $1 }').simg
+
+    # Pull the container, if it doesn't exist.
+    if [ ! -f "${SINGULARITY_CACHEDIR}/${CONTAINER_NAME}" ]
+        then
+        singularity pull --name "${CONTAINER_NAME}" "${CONTAINER}"
+    fi
+    CONTAINER="${SINGULARITY_CACHEDIR}/${CONTAINER_NAME}"
+fi
+
+# If still doesn't exist, exit
+if [ ! -f "${CONTAINER}" ]
+    then
+    echo "Issue obtaining ${CONTAINER}."
+    exit 1
+fi
+
+# Create .local folder for default modules, if doesn't exist
+if [ ! -d "${HOME}/.local" ];
+    then
+        echo "Creating local python modules folder to map at ${HOME}/.local";
+        mkdir -p "${HOME}/.local";
+fi
+
+singularity exec --home "${HOME}" --bind ${HOME}/.local:/home/joyvan/.local "${CONTAINER}" jupyter notebook --no-browser --port=$PORT --ip 0.0.0.0

setup.sh

@@ -27,13 +27,23 @@ echo
 read -p "${RESOURCE} partition (default: normal) > "  PARTITION
 PARTITION=${PARTITION:-normal}
 
+echo
+SHARE="/scratch/users/vsochat/share"
+echo "A containershare (https://vsoch.github.io/containershare is a library of
+containers that are prebuilt for you, and provided on your cluster resource. if you
+are at Stanford, leave this to be the default. If not, ask your HPC administrator
+about setting one up, and direct them to https://www.github.com/vsoch/containershare."
+echo
+read -p "container shared folder (default for Stanford: ${SHARE}) > " CONTAINERSHARE
+CONTAINERSHARE=${CONTAINERSHARE:-${SHARE}}
+
 echo
 
 MEM=20G
 
 TIME=8:00:00
 
-for var in USERNAME PORT PARTITION RESOURCE MEM TIME
+for var in USERNAME PORT PARTITION RESOURCE MEM TIME CONTAINERSHARE
 do
     echo "$var="'"'"$(eval echo '$'"$var")"'"'
 done >> params.sh

start-node.sh

@@ -51,7 +51,7 @@ echo
 echo "== Submitting sbatch =="
 
 SBATCH_NAME=$(basename $SBATCH)
-command="${RESOURCE} sbatch
+command="sbatch
     --job-name=$NAME
     --partition=$PARTITION
     --output=$RESOURCE_HOME/forward-util/$NAME.out
@@ -61,7 +61,7 @@ command="${RESOURCE} sbatch
     $RESOURCE_HOME/forward-util/$SBATCH_NAME $PORT \"${@:2}\""
 
 echo ${command}
-ssh ${command}
+ssh ${RESOURCE} ${command}
 
 # Tell the user how to view error/output logs
 instruction_get_logs