Manual:Specific supercomputers

From OctopusWiki

Here the general instructions for building Octopus are supplemented by instructions for some specific large supercomputers. Included also is how to configure the optional ETSF_IO library, and how you can run the testsuite in parallel.

1 United States
2 Europe

[edit]

United States

[edit]

Franklin

Cray XT4 at National Energy Research Scientific Computing Center (NERSC), Lawrence Berkeley National Laboratory, USA

http://www.nersc.gov/nusers/systems/franklin/

module load gsl
module load fftw/3.1.1
module load netcdf
./configure --prefix=`pwd` --enable-mpi CC=cc FC=ftn FCFLAGS="-O2 -Munroll=c:1 -Mnoframe -Mlre -Mscalarsse -Mcache_align -Mflushz" GSL_CONFIG=$GSL_DIR/gsl-config --enable-newuoa

To build ETSF_IO:

module load netcdf
./configure --prefix=`pwd` --with-netcdf-module-path="$NETCDF_DIR/src/f90"

To run the testsuite in parallel (only for 3.2.0 and later):

#!/bin/bash
#PBS -q debug
#PBS -l mppwidth=5
#PBS -l walltime=0:30:00
#PBS -V

cd $HOME/octopus
export MPIEXEC=`which aprun`
export TEMPDIRPATH=$SCRATCH
make check &> $PBS_O_WORKDIR/makecheck

(27 October 2009)

[edit]

Kraken

Cray XT5 at National Institute for Computational Sciences (NICS), University of Tennessee-Knoxville / Oak Ridge National Laboratory, USA

Part of National Science Foundation's TeraGrid

http://www.nics.tennessee.edu/computing-resources/kraken

module load gsl
module load fftw/3.1.1
module load netcdf/4.0.1.0
module load gdlib
./configure --prefix=`pwd` --enable-mpi CC=cc FC=ftn FCFLAGS=-fast --with-netcdf="$NETCDF_FLIB" --enable-newuoa

To build ETSF_IO:

module load netcdf/4.0.1.0
./configure --prefix=`pwd` --with-netcdf-module-path="$NETCDF_DIR/include" --with-netcdf-ldflags="$POST_LINK_OPTS"

To run the testsuite in parallel: (the code must be compiled and installed in $SCRATCH, not $HOME)

#!/bin/bash
#PBS -l size=12,walltime=0:30:00
#PBS -V

cd $SCRATCH/octopus
export MPIEXEC=`which aprun`
export TEMPDIRPATH=$SCRATCH/tmp
make -e check &> $PBS_O_WORKDIR/makecheck

(12 February 2010)

[edit]

Lonestar

Dell Linux cluster at Texas Advanced Computing Center (TACC), University of Texas, Austin, USA

Part of National Science Foundation's TeraGrid

http://services.tacc.utexas.edu/index.php/lonestar-user-guide

module load gsl
module load mkl
module load netcdf
module load fftw3
./configure --prefix=`pwd` --enable-mpi FCFLAGS=-O3 --with-blas="-L$TACC_MKL_LIB -lmkl_em64t" --with-fft-lib="-L$TACC_FFTW3_LIB -lfftw3" --with-netcdf="-L$TACC_NETCDF_LIB -lnetcdf -I$TACC_NETCDF_INC" --enable-newuoa --disable-gdlib

To build ETSF_IO:

module load netcdf 
./configure --prefix=`pwd` FC=mpif90 --with-netcdf-module-path=$TACC_NETCDF_INC --with-netcdf-ldflags=-L$TACC_NETCDF_LIB FCFLAGS=-O3

To run the testsuite in parallel:

Not really possible to do properly.

(26 October 2009)

[edit]

Lawrencium

Dell Linux cluster at Lawrence Berkeley National Laboratory, USA

Cluster available only to Lawrence Berkeley National Laboratory researchers

http://lrc.lbl.gov/html/Lawrencium.html

module load netcdf/4.0-intel
module load gsl
module load mkl
module load fftw/3.1.2-intel
module load gdlib
./configure --prefix=`pwd` --enable-mpi FCFLAGS=-O3 --with-blas="-L$MKL_HOME/lib/em64t -lmkl_em64t -lguide -pthread" --with-fft-lib="-L/global/software/centos-5.x86_64/modules/fftw/3.1.2-intel/lib -lfftw3" --with-netcdf="-L$NETCDF/lib -lnetcdf -lnetcdf_c++ -I$NETCDF/include" --enable-newuoa

To build ETSF_IO:

module load netcdf 
./configure --prefix=`pwd` --with-netcdf-module-path=$NETCDF/include/ --with-netcdf-ldflags="-L$NETCDF/lib/ -lnetcdf"

(29 October 2009)

[edit]

Europe

[edit]

Jugene

FZJ-JSC IBM Blue Gene/P, JÜLICH SUPERCOMPUTING CENTRE (JSC), Germany

May also work in another Blue Gene/P computers

General information: http://www.fz-juelich.de/jsc/jugene

Usage information: http://www.fz-juelich.de/jsc/jugene/usage/

module load gsl;
module load lapack;
autoreconf -i;
export FC_INTEGER_SIZE=4;
export CC_FORTRAN_INT=int;
export CC=mpixlc_r;
export CFLAGS='-g -O3 -qarch=450d';
export FC=mpixlf90_r;
export FCFLAGS='-g -O3 -qarch=450d -qxlf90=autodealloc -qessl -qsmp=omp';
export LIBS_BLAS="-lesslsmpbg -L/opt/ibmmath/essl/4.4/lib -lesslbg";
export LIBS_LAPACK="-L$LAPACK_LIB -llapack";
export LIBS_FFT="-L/bgsys/local/fftw3/lib/ -lfftw3 -lm";
./configure --prefix=$outdir \
     --host=powerpc32-unknown-linux-gnu \
     --build=powerpc64-unknown-linux-gnu \
     --disable-gdlib \
     --with-gsl-prefix=$GSL_DIR \
     --enable-mpi --enable-openmp;

To run you have to change to $WORK directory and run LoadLeveler. Below is an example of the tutorial (http://www.tddft.org/programs/octopus/wiki/index.php/Tutorial:Benzene_molecule) executing in 32 nodes in the SMP mode (4 threads per chip). The execution mode is hybrid; OpenMP/MPI.

Here is a example job input script:

# @ job_name = Octopus_Sample_1
# @ comment = "BGP Job by Size"
# @ error = $(job_name).$(jobid).out
# @ output = $(job_name).$(jobid).out
# @ environment = COPY_ALL;
# @ wall_clock_limit = 00:30:00
# @ notification = error
# @ notify_user = foo@gmail.com
# @ job_type = bluegene
# @ bg_size = 32
# @ queue
mpirun  -exe $OCTOPUS_HOME/bin/octopus_mpi -mode SMP -verbose 1

To execute the above script:

llsubmit executing_script

[edit]

Finis Terrae

Finis Terrae Itanium computer installed at CESGA (Galicia, Spain), is an integrated system with shared memory nodes with an SMP NUMA architecture.

http://www.cesga.es/content/view/917/115/lang,en/

Script to build ETSF_IO-1.0.2 under bash:

#!/bin/bash
module load ifort/11.0.074 netcdf/4.0

export F77="ifort"
export FFLAGS="-O3 -ip -ftz"

export FC="ifort"
export FCFLAGS="-O3 -ip -ftz"

./configure --prefix=/dir/install/etsf_io-1.0.2 \ 
 --with-netcdf-module-path="/dir/install/netcdf-4.0/include" --with-netcdf-ldflags="-L/dir/install/netcdf-4.0/lib -lnetcdf -lnetcdff -lnetcdf_c++"

Script to build Octopus-3.2.0 in serial mode under bash:

#!/bin/bash
module load gsl/1.12 icc/11.0.074 ifort/11.0.074 fftw/3.1.2 mkl/10.1.0 impi/3.2.0.011 netcdf/4.0 sparskit/2 etsf_io/1.0.2

export CC="icc"
export CFLAGS="-O2 -ip -ftz" 
export LDFLAGS="-lguide -lpthread -lm"

export F77="ifort"
export FFLAGS="-O2 -ip -ftz" 

export FC="ifort"
export FCFLAGS="-O2 -ip -ftz" 
# without MPI
./configure  --prefix=/dir/install/octopus-3.2.0  --disable-python  --disable-gdlib  \ 
 --with-blas="-L/dir/install/intel/intel11.0/Compiler/11.0/069/mkl/lib/64 -lmkl -lguide -lpthread"  \ 
 --with-lapack="-L/dir/install/intel/intel11.0/Compiler/11.0/069/mkl/lib/64 -lmkl_lapack -lguide -lpthread"  \ 
 --with-fft=fftw3 \ 
 --with-etsf_io="-L/dir/install/etsf_io-1.0.2/lib/ -letsf_io -I/dir/install/etsf_io-1.0.2/include/intel/" \
 --with-netcdf="-L/dir/install/netcdf-4.0/lib -lnetcdf -lnetcdff -lnetcdf_c++ -I/dir/install/netcdf-4.0/include" \ 
 --with-sparskit="-L/dir/install/SPARSKIT-2/lib -lskit"

The tests in serial mode run properly:

    Passed:  36 / 38
    Skipped: 2 / 38 

Everything seems to be OK
Total run-time of the testsuite: 00:11:13

Script to build Octopus-3.2.0 in parallel mode under bash:

#!/bin/bash
module load gsl/1.12 icc/11.0.074 ifort/11.0.074 fftw/3.1.2 mkl/10.1.0 impi/3.2.0.011 netcdf/4.0 sparskit/2 etsf_io/1.0.2

export CC="mpiicc"
export CFLAGS="-O2 -ip -ftz"
export LDFLAGS="-lguide -lpthread -lm"

export F77="mpiifort"
export FFLAGS="-O2 -ip -ftz"

export FC="mpiifort"
export FCFLAGS="-O2 -ip -ftz"
# with MPI
./configure --prefix=/dir/install/octopus-3.2.0-mpi --disable-python --disable-gdlib --enable-mpi \
 --with-blas="-L/dir/install/intel/intel11.0/Compiler/11.0/069/mkl/lib/64 -lmkl -lguide -lpthread"  \ 
 --with-lapack="-L/dir/install/intel/intel11.0/Compiler/11.0/069/mkl/lib/64 -lmkl_lapack -lguide -lpthread"  \ 
 --with-fft=fftw3 \ 
 --with-etsf_io="-L/dir/install/etsf_io-1.0.2/lib/ -letsf_io -I/dir/install/etsf_io-1.0.2/include/intel/" \
 --with-netcdf="-L/dir/install/netcdf-4.0/lib -lnetcdf -lnetcdff -lnetcdf_c++ -I/dir/install/netcdf-4.0/include" \ 
 --with-sparskit="-L/dir/install/SPARSKIT-2/lib -lskit"

If you have a lot of failed tests in parallel mode, fail:

...
Test run failed with exit code 65280
...
cat: td.general/multipoles: No such file or directory
...
   Passed:  1 / 38
   Skipped: 3 / 38
   Failed:  34 / 38
...

Maybe you must set the MPIEXEC variable to mpirun instead fo mpiexec and you must use a SHARED directory:

#!/bin/bash
module load gsl/1.12 icc/11.0.074 ifort/11.0.074 fftw/3.1.2 mkl/10.1.0 impi/3.2.0.011 netcdf/4.0 sparskit/2 etsf_io/1.0.2
mkdir -p $HOMESFS/tmp

export TEMPDIRPATH=$HOMESFS/tmp
echo TEMPDIRPATH: $TEMPDIRPATH

export MPIEXEC=`which mpirun`
echo MPIEXEC: $MPIEXEC

make check

It seems to work:

   Passed:  34 / 38
   Skipped: 3 / 38
   Failed:  1 / 38
   testfile                                          # failed testcases
   --------------------------------------------------------------------
   finite_systems_3d/21-lobpcg.test                  2

[edit]

MareNostrum

MareNostrum is a supercomputer in Europe, the most powerful in Spain. This is one of the seven supercomputers of the Spanish Supercomputing Network. The supercomputer consists of 2,560 JS21 blade computing nodes, each with 2 dual-core IBM 64-bit PowerPC 970MP processors running at 2.3 GHz for 10,240 CPUs in total. The computing nodes of MareNostrum communicate primarily through Myrinet.

You may need to compile GSL and FFTW libraries before starting then installation.

Here is a script to build Octopus:

#!/bin/bash

export LD_LIBRARY_PATH=/usr/lib
export CC=mpicc
export FC=mpif90

export CFLAGS="-q64 -O3 -qtune=ppc970 -qarch=ppc970 -qcache=auto -qnostrict -qignerrno -qinlglue"
export FCFLAGS=$CFLAGS" -qessl -qxlf90=autodealloc" 

export LIBS_BLAS="-L/usr/lib64 -lessl" 

./configure   \
   --with-lapack=/gpfs/apps/LAPACK/lib64/liblapack.a \
   --disable-gsltest \
   --disable-gdlib \
   --with-gsl-prefix=$outdirGSL \
   --with-fft-lib=$outdirFFT/lib/libfftw3.a  \
   --prefix=$outdir --enable-mpi --disable-f90-forall

As you can see in the script the --with-gsl-prefix has to point to your GSL and --with-fft-lib to your FFTW. You should compile those with the same CFLAGS and FCFLAGS. Here is an example of configuring GSL (we use GSL-1.11):

./configure --prefix=$outdirGSL CC=gcc -m64 --enable-static --disable-shared

We use FFTW-3.1.3. To configure FFTW3 just write:

./configure --prefix=$outdirFFT

Recently Marenostrum has changed to the SLURM manager. To execute a job you have to submit it with jobsubmit (more details in the User's Guide).

Retrieved from "http://www.tddft.org/programs/octopus/wiki/index.php/Manual:Specific_supercomputers"

Manual:Specific supercomputers

From OctopusWiki

Contents

United States

Franklin

Kraken

Lonestar

Lawrencium

Europe

Jugene

Finis Terrae

MareNostrum

Views

Personal tools

Navigation

Search

Toolbox