// Compilers

Compilers at NCCS

To accommodate the needs of a broad range of user groups, multiple versions of compilers from different vendors are provided on the Discover systems, including:

Vendor C C++ Fortran Finding Module
GNU Compiler gcc g++ gfortran module av comp/gcc
Intel Compiler icc icpc ifort module av comp/intel
PGI Compiler pgcc pgc++ pgf77/pgf90 module av comp/pgi
NAG Compiler N/A N/A nagfor module av comp/nag

For in depth examples in using modules to select compilers, click here.

Compiler Options

For compiler options, use appropriate man pages. For example: $ man nagfor
.. or ..
$ man gcc
.. or ..
$ man pgfortran
.. or ..
$ man ifort

MPI Libraries

The NCCS provides support for compiling and linking against various MPI libraries, including Intel MPI, SGI MPT, and HPCX.

Vendor Supported Compilers Finding Modules
Intel MPI GCC/Intel/PGI module av mpi/impi
SGI MPT GCC/Intel/PGI module av mpi/sgi-mpt
HPCX GCC/Intel/PGI module av mpi/hpcx

Note: The module system will not display any MPI modules unless you first have a compiler module loaded. Also, because of the OPA interconnect, only "mpi/impi" modules are supported on Skylake nodes.

MPI modules provided by above can be used with all compiler types: GNU, Intel, PGI and NAG.

MPI modules provides an environment that allows usage of MPI libraries and corressponding MPI enabled compilers: mpif90, mpicc, mpicxx.

To learn more about MPI enabled compiler and view the corresponding link lines used by the MPI based compiler, use: $ mpif90 -show
$ mpicc -show
$ mpicxx -show

Loading MPI libraries

Using Intel compilers: $ module load comp/intel/19.1.3.304
$ module load mpi/impi/19.1.3.304 Note: Users are recommended to use the same versions numbers for compilers and MPI.

Using PGI compiler: $ module load comp/pgi/20.4
$ module load mpi/impi/19.1.3.304 Note: PGI compilers are no longer being produced, this is the last release.

Note: Due to fortran module dependencies on particular compilers, users are recommended to replace "use mpi" source code line with: include 'mpif.h'

Numerical Libraries

Intel MKL (Math Kernal Library) is the primary numerical library package supported on the NCCS systems. MKL offers highly optimized, thread-safe math routines with good performance.

MKL libraries are already included in the LD_LIBRARY_PATH after you load a module for a recent version of the Intel Compiler (version >= 11). Hence there is no need to load "lib/mkl-*" if you load the module for intel compiler 11 and up.

Please note that you will need to load the module "lib/mkl-*" if you use either PGI or GNU compiler and want to use the MKL.

The table below shows different scenarios to choose the threading and run time libraries (RTL) when linking your code with the MKL (static cases only in the table). All the MKL libraries are located under $MKLPATH.

Compiler Application Threaded Threading lib RTL recommended Comment
Intel Does not matter libmkl_intel_thread.a libiomp5.so Dynamically linking RTL is always recommended
PGI Yes libmkl_pgi_thread.a or libmkl_sequential.a None. PGI supplied Use of libmkl_sequential.a removes threading from Intel MKL calls
PGI No libmkl_sequential.a None --
GNU Yes libmkl_gnu_thread.a libiomp5.so or GNU OpenMP run time library libiomp5 offers superior scaling performance
GNU No libmkl_sequential.a None --
Other Yes libmkl_intel_thread.a libiomp5.so --
Other No libmkl_sequential.a None --

To decide which options to use during the link and compile stages when you invoke MKL routines, Intel provided a very helpful Link Advisor website.

Loading MKL with Intel compilers

Intel MKL are bundled all Intel compilers, starting with version 11. It is recommended that you use the most recent compiler version. Use: $ module load comp/intel/18.0.5.274

The MKL package provides two environment variables: MLKPATH and MLKROOT. The following commands show the value of these environment variables: $ echo $MKLPATH
[path]/intel/2018/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64/
$ echo $MKLROOT
[path]/intel/2018/compilers_and_libraries_2018.5.274/linux/mkll

Users can find example code using the following path, once they have loaded the appropriate module: $ cd $MKLROOT/examples

Examples for compiling Basic Linear Algebra Subprograms(BLAS) programs using different kind of compilers:

  1. Using Intel compiler:
    • Set your LIBS variable to:LIBS = -L$MKLPATH -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -ldl

    • Assuming your source files are sasumx.f common_func.f, ad executable is sasumx.out; run: $ ifort sasumx.f common_func.f $LIBS -o sasumx.out


  2. Using GNU compiler (such as gfortran):
    Users may use the default GNU compiler or load a module using "module av comp/gcc"
    • MKL functionality is available through lib modules, which can be listed using "module av lib"

    • An example of loading a lib module:$ module load lib/mkl/18.0.5.274Note: Loading the module above will add the following to the LD_LIBRARY_PATH to: /usr/local/intel/2018/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64
      /usr/local/intel/2018/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin

    • Set the $LIBS1 and LIBS2 variable to: LIBS1="-L/usr/local/intel/2018/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64 -lmkl_gf_lp64 -lmkl_intel_thread -lmkl_core"
      LIBS2="-L/usr/local/intel/2018/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin -liomp5 -lpthread -ldl"

    • Assuming your source files are sasumx.f common_func.f, ad executable is sasumx.out; run: $ gfortran -m64 -fcray-pointer sasumx.f common_func.f $LIBS1 $LIBS2 -o sasumx.out


  3. Using PGI compiler:
    • Once a PGI compiler module is loaded, set the LIBS1 and LIBS2 variable to: LIBS1="-L/usr/local/intel/2018/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core"
      LIBS2="-L/usr/local/intel/2018/compilers_and_libraries_2018.5.274/linux/compiler/lib/intel64_lin -liomp5 -lpthread -ldl"

    • Assuming your source files are sasumx.f common_func.f, ad executable is sasumx.out; run:$ pgf95 sasumx.f common_func.f $LIBS1 $LIBS2 -o sasumx.out