.. _gsi_install: 2. Software Installation ========================= Introduction ------------- The DTC GSI is a community distribution of NOAAs operational GSI. The community GSI expands the portability of the operational code by adding a flexible build system which provides support for many common platforms and compilers. GSI 3.7 builds and runs on most standard Linux platforms using Intel, PGI, or Gnu compilers. This chapter describes how to build and install the DTC community GSI software on your local Linux computing resources. The GSI build process consists of three general steps: - Obtaining the source code for GSI. - Setting the appropriate environment variables for the GSI build. - Configuring and compiling the GSI source code using CMake. This chapter is organized as follows: Section `1.2 <#ch2_obtainingcode>`__ describes how to obtain the source code. Section `1.3 <#ch2_directorystructure>`__ covers the directory structure and supplemental NCEP libraries included with the distribution. Section `1.4 <#ch2_cmake>`__ starts with an brief overview of CMake and continues with an example of the build process in section `1.4.1 <#ch2_cmake_start_here>`__. Section `1.4.2 <#ch2_cmake_default_options>`__ describes all of the current optional settings for the build. Sections `1.5 <#ch2_externallibs>`__ and `1.5.1 <#ch2_compilers_tested>`__ cover the system requirements (tools, libraries, and environment variable settings) and currently supported platforms in detail. Section `1.6 <#ch2_gettinghelp>`__ discusses what to do if you have problems with the build and where to get help. Advanced topics, such as customizing the build, porting to new platforms, and debugging can be found in the GSI Advanced Users Guide. .. _ch2_obtainingcode: Obtaining and Setting Up the Source Code ---------------------------------------- The community GSI resources, including source code, build system, utilities, practice data, and documentation, are available from the DTC community GSI users website, located at http://www.dtcenter.org/com-GSI/users/index.php The source code is available by first selecting the tab on the vertical menu located on the left column of the page, and then selecting the submenu. New users must first register before downloading the source code. Returning users only need to enter their registration email address to log in. After accessing the download page, select the link to the ``comGSIv3.7_EnKFv1.3.tar`` tarball to download the most recent version of the source code (November 2018). Selecting the newest release of the community GSI is critical for having the most recent capabilities, versions of supplemental libraries, and bug fixes. Full support is only offered for the three most recent code releases. To analyze satellite radiance observations, GSI requires the use of CRTM coefficients. It is crucial to use **only** the version of CRTM coefficients provided on the GSI website. Due to their large size, these are available as a separate tarfile. They can be downloaded by selecting the link to the tarball for the ``CRTM 2.3.0 Big Endian coefficients`` from the web page. For all compilers, use the big endian byte order coefficients found in the first CRTM link. The download page also contains links to the fixed files necessary for running global GSI: - Global configuration (fix files to run Global GSI) The community GSI version 3.7 comes in a tar file named ``comGSIv3.7_EnKFv1.3.tar``. The tar file may be unpacked by using the UNIX commands: :: gunzip comGSIv3.7_EnKFv1.3.tar.gz tar -xvf comGSIv3.7_EnKFv1.3.tar This creates the top level GSI directory ``comGSIv3.7_EnKFv1.3/``. After downloading the source code, and prior to building, the user should check the known issues link on the download page of the DTC website to determine if any bug fixes or platform specific customizations are needed. .. _ch2_directorystructure: Directory Structure, Source Code and Supplemental Libraries ----------------------------------------------------------- The GSI system includes the GSI source code, the build system, supplemental libraries, fixed files, and run scripts. Again, with V3.7 of GSI the directory structure has been slightly changed to accommodate the new CMake build system. The following table lists the system components found inside the root GSI directory. .. table:: Description of top level directories ============== =============================================================== Directory Name Content ============== =============================================================== cmake/ CMake build rules and configure files. fix/ Fixed input files required by a GSI analysis, such as \ background error covariances, observation error tables; \ Excluding the CRTM coefficients libsrc/ Supplemental library source code, required for the CMake build modulefiles/ Include files created by the build system README.cmake Default README file for the CMake build system. scripts/ Sample run scripts src/ GSI source code and makefiles src/enkf EnKF source code and makefiles util/ Tools for GSI diagnostics ============== =============================================================== [ch2_tble1] For convenience, supplemental NCEP libraries for building GSI are included in the ``libsrc/`` directory. These libraries will be built prior to compiling GSI. These supplemental libraries are listed in the table below. .. table:: List of NCEP libraries used by GSI ============== ================================================== Directory Name Content ============== ================================================== bacio/ NCEP BACIO library bufr/ NCEP BUFR library crtm/ JCSDA community radiative transfer model GSD/ GSD Cloud analysis library ncdiag/ Misc NetCDF libraries nemsio/ NEMS I/O library sfcio/ NCEP GFS surface file i/o module sigio/ NCEP GFS atmospheric file i/o module sp/ NCEP spectral - grid transforms w3emc/ NCEP/EMC W3 library (date/time manipulation, GRIB) w3nco/ NCEP/NCO W3 library (date/time manipulation, GRIB) wrflib WRF IO libraries ============== ================================================== [ch2_tble2] .. _ch2_cmake: CMake Build System ------------------ A new unified build system based on CMake has been added to the GSI code. While the new build system is a significant departure from the previous DTC build system, CMake is a very powerful cross-platform open-source build system that has many capabilities to make building GSI easier. The CMake build infrastructure consists of a top level directory with the name ``cmake/`` and configuration files in each directory named ( ``CMakeLists.txt``). This replaces the old ``arch/`` directory. CMake relies on a two step command line process, similar to the old UNIX "configure" and "compile." Command line arguments are used to specify paths and compilers. By default CMake is configured to build the source code "out-of-place," meaning that it does not populate the GSI directory with the build. The location for the build must be specified by the user. Once a compiler has been chosen, CMake generates local makefiles by invoking the cmake command with the proper arguments. One of those arguments selects that a local build of the NCEP libraries needed by GSI will be conducted prior to the source code being built. The final step is to invoke a parallel build (make) of the code. A significant advantage with using CMake to build the code, is that CMake automatically generates code dependencies each time a build is invoked, allowing the use of a parallel make as the final step of the build. This greatly reduces the time it take to complete the build. Because of the parallel compiling, the time to complete the CMake build is typically a quarter of the time needed for the serial DTC "configure" and "compile" to complete. Once the build is complete, the two executables a ``gsi.x`` and ``enkf_gfs.x`` are placed in the ``bin`` directory of the build directory. Note that the name of the executables and their location differs from the traditional DTC build. Summary of CMake build steps: #. Set up the build environment - Select a specific compiler by loading modules or setting paths as needed - Set the environment path for the NetCDF library (The HDF5 library is also needed if your NetCDF version is 4+). - Set the LaPack/MKL mathematics libraries as needed. #. Create a directory for your build and change into it. #. Run the cmake command with the appropriate arguments. #. Run make #. The completed executables ``gsi.x`` and ``enkf_gfs.x`` will be located in the build directory within the ``bin`` directory. .. _ch2_cmake_start_here: Example build using the Intel Fortran Compiler ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This section details how to build GSI on the NCAR supercomputer Cheyenne using the Intel V18.0.1 fortran compiler, the 2018 MKL libraries, NetCDF v4.6.1, and the 2018 Intel MPI libraries. - Set up your build environment :: module purge module load cmake/3.9.1 module load intel/18.0.1 ncarenv/1.2 ncarcompilers/0.4.1 mkl/2018.0.1 module load netcdf/4.6.1 module load impi/2018.1.163 - Next, create a build directory and move into it. :: mkdir ./build cd build - Set environment variables specifying the ``C``, ``C++``, and Fortran compilers. Also set the path location for the top of the source code directory. These can be set in your dot files if you plan to always use the same compiler and source code base. For bash, :: export CC=icc export CXX=icpc export FC=mpif90 export CORE_DIR /path_to_the_source_code_directory/ and for csh :: setenv CC icc setenv CXX icpc setenv FC mpif90 setenv CORE_DIR /path_to_the_source_code_directory/ Alternatively, these can be set on the cmake command line. - From within the build directory, invoke the cmake command :: cmake -DBUILD_CORELIBS=ON $CORE_DIR Or if you wish to specify everything on the command line :: cmake -DBUILD_CORELIBS=ON -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icpc \ -DCMAKE_Fortran_COMPILER=mpif90 $CORE_DIR The makefiles customized to your platform and the environment you have set are now created. - Run make (parallel compile) on eight cores :: make -j8 Because CMake has already figured out the dependencies, the make can run in parallel. Lets review the relevant arguments for cmake when building GSI. The first is ``-DBUILD_CORELIBS=ON``. This argument directs CMake to look in the ``libsrc/`` directory to build the NCEP libraries needed for GSI. The argument ``CMAKE_C_COMPILER`` specifies the ``C`` compiler to use with the build. For the Intel compiler, use ``icc``. The argument ``CMAKE_CXX_COMPILER`` specifies the ``C++`` compiler to use with the build. For the Intel compiler, use ``icpc``. The argument ``CMAKE_Fortran_COMPILER`` specifies the Fortran compiler to use with the build. Because this is an parallel MPI code, on Cheyenne use ``mpif90`` instead of ``ifort``. Typically, the only time one would explicitly specify the compilers on the command line is when you wish to override a default choice set in your dot files or if you were to create a batch script to conduct the build. The argument ``$CORE_DIR`` indicates the location of the build directory. This can be set to an environment variable or entered explicitly on the command line. The final statement ``make -j 8`` invokes a parallel call to *make* using eight processors. This speeds up the compilation considerably. To run the cmake command with a compiler other than Intel only requires replacing the values for ``CMAKE_C_COMPILER``, ``CMAKE_CXX_COMPILER``, and ``CMAKE_Fortran_COMPILER`` that correspond to the new compiler. Table `[ch2_tble3] <#ch2_tble3>`__ lists these values. .. table:: The compiler specific values for the cmake compiler argument ======== ======== ========== ========== Compiler ``FC`` ``C`` ``C++`` ======== ======== ========== ========== Intel ``icc`` ``icpc`` ``mpif90`` PGI ``pgcc`` ``pgcc++`` ``mpif90`` GNU ``gcc`` ``g++`` ``mpif90`` ======== ======== ========== ========== [ch2_tble3] .. _ch2_cmake_default_options: Default Options ~~~~~~~~~~~~~~~ The CMake build has a number of options to when building. These are enacted by adding the option with the prefix ``-D`` to the command line call to cmake. For instance, in the example above, the ``BUILD_CORELIBS`` option is enacted by adding ``-DBUILD_CORELIBS=ON`` to the cmake call. The currently available command line build options are: - ``BUILD_ENKF``: builds the enkf executable (default is ON) - ``BUILD_CORELIBS``: Attempts to build the core libraries froim source - ``USE_WRF``: Builds GSI with WRF dependencies (default is ON) - ``BUILD_GFS``: Builds ENKF using GFS (default is ON) - ``BUILD_WRF``: Builds ENKF using WRF (default is OFF) - ``BUILD_NMMB``: Builds ENKF using NMMB (default is OFF) Note that by default GSI is built with the regional WRF IO bindings, and EnKF is built with GFS. See the readme file *README.cmake* in the top directory for a current list of options; or look in the CMakeLists.txt in the top directory and search for options. .. _ch2_externallibs: Software Requirements and Compiler Specific Notes ------------------------------------------------- The source code for GSI is written in FORTRAN, FORTRAN 90, and C. In addition, the parallel executables require some flavor of MPI and OpenMP for the distributed memory parallelism. The I/O relies on the NetCDF I/O libraries. And CMake is needed to build the code. The basic requirements for building and running the GSI system are the following: - CMake V3+ - A FORTRAN compiler that supports the 2003 or newer standard - C compiler - MPI v1.2+ - OpenMP - NetCDF V4.2+ and HDF5 - LAPACK and BLAS mathematics libraries, or equivalent In addition, GSI and EnKF require linear algebra libraries such as LAPACK and BLAS. The Intel compiler usually comes with a vendor provided mathematics library known as the *Mathematics Kernel Libraries* or MKL for short. While *most* installations of the Intel compiler come with the MKL libraries installed not all do. In addition, if it is not installed correctly, the ifort compiler does not automatically load the library. It is therefore necessary to set the LAPACK_PATH variable to the location of the MKL libraries when using the Intel compiler. You may need to ask your system administrator for the correct path to these libraries. IBM systems typically come installed with the LAPACK equivalent ESSL library that links automatically. Likewise, the PGI compiler often comes with a vendor provided version of LAPACK that links automatically with the compiler. The Gnu compiler is the only one that does not come with any installed version of the LAPACK and BLAS libraries. Because all but the last of these tools and libraries are typically the purview of system administrators to install and maintain, they are lumped together here as part of the basic system requirements. .. _ch2_compilers_tested: Compilers Tested for Release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Version 3.7 of the DTC community GSI system extensively tested with a variety of compilers and versions of those compilers, on standard Linux platforms. To summarize, Intel compiler versions 16-18 should build and run without issue. Early version 15 has some known issues, but 15.1, 15.3, and 15.6 should work. For the Portland Group compiler, versions 16-18 should work without issue, and Gnu version 6.3, 7.1, and 7.3 are known to work. Version 18 of Gnu has issues and is not currently supported. See table `[ch2_tble4] <#ch2_tble4>`__ for the complete list compilers and versions tested. The following Linux compiler combinations have been fully tested: .. table:: Tested Compiler Combinations ===================== ================ =============== Compiler manifacturer Compiler version MPI version ===================== ================ =============== intel \ 18.1.163 impi_5.1.2.150 \ 18.0.3.222 mvapich2_1.8 \ 18.0.3.222 impi_5.1.1.109 \ 18.0.1.163 impi_5.1.2.150 \ 18.0.1 impi_2018.1.163 \ 18.0.1 mpt_2.15f \ 18.0.1 mvapich2_2.2 \ 18.0.1 openmpi_3.1.0 \ 18.0.0.128 impi_5.1.2.150 \ 17.0.5.239 mvapich2_2.1a \ 17.0.1 impi_2017.1.132 \ 17.0.1 mpt_2.15f \ 17.0.1 mvapich2_2.2 \ 17.0.1 openmpi_3.0.1 \ 2016.2.181 impi_5.1.1.109 \ 2016.2.181 mvapich2_1.8 \ 16.1.150 mvapich2_2.1a \ 16.1.150 impi_5.1.2.150 \ 16.0.3 impi_5.1.3.210 \ 16.0.3 mpt_2.15f \ 16.0.3 openmpi_3.1.0 \ 16.0.1.150 mvapich2_2.1a \ 16.0.1 impi_5.1.2.150 \ 15.6.233 mvapich2_2.1a \ 15.3.187 mvapich2_2.1a \ 15.1.133 mvapich2_2.1a \ 14.0.2 mvapich2_2.1a pgi \ 18.5 mvapich2_2.1a \ 18.3. mvapich2_2.1a \ 18.1 mvapich2_2.1a \ 17.10 mvapich2_2.1a \ 17.9 openmpi_2.1.2 \ 17.7 mvapich2_2.1a \ 17.5 mvapich2_2.1a \ 16.10 mvapich2_2.1a \ 16.9 mvapich2_2.1a \ 16.7 mvapich2_2.1a \ 16.5 mvapich2_2.1a gnu \ 7.3.0 mvapich2_2.2 \ 7.3.0 openmpi_3.0.1 \ 7.1.0 mvapich2_2.2 \ 6.3.0 openmpi_3.0.0 ===================== ================ =============== [ch2_tble4] Unforeseen build issues may occur when using older compiler and library versions. As always, the best results will be achieved by using the most recent compiler versions. .. _ch2_gettinghelp: Getting Help and Reporting Problems ----------------------------------- Should a user experience any difficulty building GSI on their system, please first confirm that all the required software is properly installed (section `[ch2_ch2_externallibs] <#ch2_ch2_externallibs>`__). Should all these check out, feel free to contact the community GSI Help Desk for assistance at gsi-help@ucar.edu At a minimum, when reporting code building problems to the helpdesk, please include a copy of the build log with your e-mail.