Skip to content

Step by step guide

Jump to bottom Edit New page
boegel edited this page Apr 4, 2012 · 11 revisions
## Introduction

This step-by-step guide will guide you through putting together a self-contained compiler toolkit, and using that toolkit to build a software package. It is assumed you've already configured EasyBuild.

For more information on what a compiler toolkit is and why EasyBuild uses them, see the Compiler toolkits wiki page.

We will put together the goalf toolkit, which consists of:

  • the GNU Compiler Collection (GCC), a set of open-source C/C++/Fortran compilers named gcc, g++ and gfortran
  • the OpenMPI library, which provides support for building MPI (Message Passing Interface) applications
  • the ATLAS and LAPACK libraries, which provide highly tuned linear algebra routines
  • the FFTW library, which provides fast discrete Fourier transform routines
  • the ScaLAPACK library, which provides MPI-enabled LAPACK routines

Note that the name we give to the toolkit is arbitrary; you might as well name it myToolkit.

The build times specified for each of the toolkit components below are estimates, based on the build times observed on a laptop with a quad-core Intel Core i5 2.4GHz processor and 8GB of physical memory, running Fedora 16 Linux (64-bit).

## Step 1: Compilers

The first step is build the set of compilers that will be used in our toolkit, because we will build the libraries that are part of the toolkit using these compilers.

For compilers that are part of a toolkit, we aim to make them self-contained, i.e., we try and make sure they depend as little as possible on any external (system) libraries.

GCC supports a so-called bootstrap build, in which the compilers are being built in 3 stages: once using the system compiler, once with the compilers obtained from stage 1, and then again using the compilers from stage 3.

This bootstrap build procedure is implemented by the GCC easyblock that comes with EasyBuild, so you don't need to worry about it.

Just provide EasyBuild with a specification file that described which GCC version you want to build, and you'll obtain a self-contained GCC build.

Step 1.1 Download GCC

For this guide, we will be using GCC v4.6.3. Download the source tarball gcc-4.6.3.tar.gz from http://ftpmirror.gnu.org/gcc/gcc-4.6.3/gcc-4.6.3.tar.gz to the source path you configured, in a subdirectory named GCC (or g/GCC, up to you).

Because GCC has a couple of dependencies, you'll need to download them as well.

Put the following source tarballs in the same directory as the GCC source tarball, e.g., <sourcePath>/GCC:

The GCC easyblock will make sure these dependencies are taken care of.

Step 1.2 Create specification file for GCC

Create a specification file GCC-4.6.3.eb with the following contents:

name="GCC"
version='4.6.3'

homepage='http://gcc.gnu.org/'
description="The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, 
             as well as libraries for these languages (libstdc++, libgcj,...)."

toolkit={'name':'dummy','version':'dummy'}

sources=['%s-%s.tar.gz'%(name.lower(),version),
         'gmp-5.0.4.tar.bz2',
         'mpfr-3.0.1.tar.gz',
         'mpc-0.9.tar.gz',
        ]

languages=['c','c++','fortran','lto']

moduleclass='compiler'

# building GCC sometimes fails if make parallelism is too high, so let's limit it
maxparallel=4

Most of entries in the specification files should be clear (if not, see Specification files).

Some remarks:

  • The dummy toolkit specifies that we'll be using the system compilers to kickstart the GCC bootstrap build. However, after completing the build procedure, the resulting GCC will be self-contained and will not have used the system compilers anymore when building the last GCC binaries.

  • The languages config entry is specific to the GCC easyblock, and specifies for which programming languages support should be enabled.

  • Because of problems with parallel building for some of the libraries on which GCC depends, we limit parallel building to a maximum of 4. EasyBuild auto-detects how many cores you have in your system, and will build in parallel accordingly.

Step 1.3 Build and install GCC

Instruct EasyBuild to build GCC by providing it the specification file:

<path>/easybuild/easybuild.sh GCC-4.6.3.eb

Building GCC v4.6.3 with a bootstrap build as performed by the GCC easyblock takes about 35 minutes.

Step 1.4 Verify installation

Once the installation is completed, you should see a message like COMPLETED: Installation ended successfully appearing, both on the command line output and in the log file created by EasyBuild.

To verify that EasyBuild has produced a working GCC build, load the GCC/4.6.3 module provided and check the version:

module load GCC/4.6.3
gcc --version | grep ^gcc

This should yield something like gcc (GCC) 4.6.3 as output.

## Step 2: MPI library

The next step is to build OpenMPI, that will serve as MPI library in the goalf compiler toolkit.

Again, just download the source, create a specification file and build/install OpenMPI using EasyBuild.

Step 2.1: Download OpenMPI

Download the openmpi-1.4.5.tar.gz source tarball from http://www.open-mpi.org/software/ompi/v1.4/downloads/openmpi-1.4.5.tar.gz, and store it in a directory named <sourcePath>/OpenMPI (or <sourcePath>/o/OpenMPI).

Step 2.2: Create specification file for OpenMPI

Create a specification file OpenMPI-1.4.5-no-OFED.eb with the following contents:

name='OpenMPI'
version='1.4.5'
versionsuffix="-no-OFED" # no InfiniBand support, so add version suffix

homepage='http://www.open-mpi.org/'
description="The Open MPI Project is an open source MPI-2 implementation."

toolkit={'name':'GCC','version':'4.6.3'}

sources=['%s-%s.tar.gz'%(name.lower(),version)]

configopts='--with-threads=posix --enable-shared '

moduleclass='lib'

sanityCheckPaths = {
                    'files':["bin/%s" % binfile for binfile in ["ompi_info", "opal_wrapper", "orterun"]] +
                            ["lib/lib%s.so" % libfile for libfile in ["mca_common_sm", "mpi_cxx", "mpi_f77" ,"mpi_f90",  
                                                                      "mpi", "openmpi_malloc", "open-pal", "open-rte"]],
                    'dirs':["include/openmpi/ompi/mpi/cxx"]
                   }

Some specific remarks with regard to this specification file:

  • We specify that EasyBuild should add a version suffix -no-OFED for this build, because we are not enabling InfiniBand (IB) support (using the OFED stack) in this OpenMPI build procedure. We disabled this because enabling IB support requires several other system libraries to be installed, and we prefer not to bother you with those for the sake of this demo.

  • We are using the GCC built in the previous step as a toolkit for building OpenMPI, as specified by the toolkit config entry.

  • OpenMPI follows the more-or-less standard configure/make/make install build procedure, and thus there's no real need for a dedicated OpenMPI easyblock. Because there is not easyblock available by default, EasyBuild will fall back to the default Application easyblock that implements this configure/make/make install build procedure.

  • Non-default configure options are specified using the configopts config entry, and we specify which files and directories are expected to be installed using the sanityCheckPaths config entry.

Step 1.3 Build and install OpenMPI

Instruct EasyBuild to build OpenMPI by providing it the specification file:

<path>/easybuild/easybuild.sh OpenMPI-1.4.5-no-OFED.eb

Building and installing OpenMPI should only take about 7 minutes.

## Step 3: BLAS and LAPACK libraries

(more soon)

## Step 4: FFTW library

(more soon)

## Step 5: ScaLAPACK library

(more soon)

## Step 6: Compiler toolkit

(more soon)

## Step 7: Building software with compiler toolkit

(more soon)

## Step 8: Updating the compiler: automatic dependency resolution

(more soon)

Add a custom footer
Add a custom sidebar
Clone this wiki locally