Running at NCI

TCRM at the NCI

Here we describe how to run TCRM on one of the machines at the NCI. TCRM has been successfully run on raijin with up to 256 cores.

Installing TCRM

TCRM can be installed by cloning the GitHub repository:

git clone http://github.com/GeoscienceAustralia/tcrm

Compile the C extensions:

cd tcrm
python installer/setup.py build_ext -i

Raijin

Setting up the environment

Load up the latest versions of Python, OpenMPI and netCDF:

~$ module load python/2.7.15
~$ module load openmpi/1.6.3
~$ module load netcdf/4.3.0
~$ module load hdf5/1.8.7
~$ module load pypar

Check that they are loaded:

~$ module list
Currently Loaded Modulefiles:
1) pbs                     6) python/2.7.15
2) dot                     7) netcdf/4.3.0
3) intel-fc/12.1.9.293     8) pypar/30May-2.7.6-1.8
4) intel-cc/12.1.9.293     9) szip/2.1
5) openmpi/1.6.3          10) hdf5/1.8.7

Build a local copy of the netcdf4-python library. Install netcdf4-python either using pip:

pip install netCDF4 --user

Or by building the latest copy of the netcdf4-python library:

git clone https://github.com/Unidata/netcdf4-python
cd netcdf4-python/
python setup.py install --user

Install the remaining required python modules locally:

pip -v install -r requirements.txt --user

Install matplotlib Basemap

Download basemap tar-ball and unpack in a tmp directory on NCI:

tar -xzvf basemap-1.0.7.tar.gz
cd basemap-1.0.7
export GEOS_DIR=$GEOS_ROOT
cc=gcc python setup.py build
python setup.py install --user

Creating a job file

Create a job file called tcrm with the following content:

#!/bin/bash
#PBS -q normal
#PBS -l walltime=00:45:00,mem=3GB
#PBS -l ncpus=64
#PBS -j oe

module purge
module load openmpi/1.8.4
module load python/2.7.6
module load python/2.7.6-matplotlib
module load pypar/26Feb15-2.7.6-1.8.4
module load geos
module load gdal/1.11.1-python

NCPUS=${PBS_NCPUS-8}

cd ~/tcrm
mpirun -n $NCPUS python tcrm.py -c ./example/port_hedland.ini

Running a job

Submit the job:

qsub tcrm