2. Software Installation¶
Introduction¶
The DTC community EnKF is a community distribution of NOAAs operational ensemble Kalman filter. The community EnKF expands the portability of the operational code by adding a flexible build system and providing example run scripts that allow the system to run on many common platforms. The current version of the community EnKF builds and runs on most standard Linux platforms using the Intel, PGI, and GNU compilers.
This chapter describes how to build and install the DTC community EnKF on your local Linux computing resources. By default EnKF is built as part of the GSI build. The only difference between the steps necessary to build GSI and EnKF are that for EnKF one of three IO configurations must be chosen prior to enacting the build.
The EnKF build process consists of the three steps:
- Obtaining the source code for EnKF/GSI.
- Setting the appropriate environment variables for the build.
- Configuring and compiling the source code using CMake.
Section 2.2 describes how to obtain the source code from the DTC website. Section 2.2.1 describes the three different IO options that can be selected prior to building the code. Section 2.3 presents an outline of the build process. Section 2.3.1 illustrates an Intel build in detail, and discusses optional arguments. Section 2.3.2 describes all of the current optional settings for the build. Sections 2.4 and 2.5 cover the system requirements (tools, libraries, and environment variable settings) and currently supported platforms in detail. Section 2.6 discusses what to do if you have problems with the build and where to get help.
Obtaining the Source Code¶
The community EnKF code and the GSI code are released as a combined source code package. The current EnKF release is v1.3 and is paired with the community GSI release version 3.7. The community EnKF release is available from the DTC community EnKF users website;
http://www.dtcenter.org/EnKF/users/index.php
The community GSI/EnKF release includes the source code for both the EnKF v1.3 and the GSI v3.7 models, as well as an integrated build system, utilities, and documentation necessary to build and run the EnKF.
To download the source code from the either the GSI or the EnKF website,
select the tab along with the System sub-tab on the vertical menu
located on the left side of the main page. New users must first register
before downloading the source code. Returning users only need to enter
their registration email address to log in. After accessing the download
page, select the link to the comGSIv3.7_EnKFv1.3
tarball. Please
only use the source code provided with the comGSIv3.7_EnKFv1.3
tarball. Do not mix and match this tarball with other versions of the
community GSI code or supplemental libraries, as this will lead to
unpredictable results.
The community EnKF version 1.3 comes in a tar file named
comGSIv3.7_EnKFv1.3.tar
. The tar file may be unpacked by using the
standard UNIX commands:
gunzip comGSIv3.7_EnKFv1.3.tar.gz
tar -xvf comGSIv3.7-EnKFv1.3.tar
This creates the top level GSI directory comGSIv3.7_EnKFv1.3/
. After
downloading the source code, and prior to building, the user should
check the known issues link on the download page of DTC website to
determine if any bug fixes or platform specific customizations are
needed.
Specifics on the source code directory structure can be found in the GSI Users Guides, at the DTC community GSI users website:
http://www.dtcenter.org/com-GSI/users/index.php
Versions of EnKF¶
The EnKF code has three build time configurations for the I/O; WRF regional, GFS global, and NMMB. The EnKF analysis is identical in each case, only the capability to digest model input differs. The regional version can only digest WRF formatted I/O files. The global version can only digest spectral input from the NCEP global model, and lastly, the NMMB version can only digest NMMB files.
The default build constructs the GFS global configuration of EnKF. To enact either of the two remaining builds it is necessary to add optional build arguments to the cmake command. To build the regional EnKF with the WRF IO bindings, add the following two optional arguments to the cmake command in the build process:
-DBUILD_GFS=OFF
to switch off the global GFS version-DBUILD_WRF=ON
to turn on the regional WRF option
Similar arguments control the enacting the NMMB bindings. This will be explained in detail in section 2.3.1 and 2.3.2.
Compiling EnKF¶
A new unified build system based on CMake has been added to the EnKF/GSI code. While the new build system is a significant departure from the previous DTC build system, CMake is a very powerful cross-platform open-source build system that has many capabilities to make building easier.
The CMake build infrastructure consists of a top level directory with
the name cmake/
and configuration files in each directory named (
CMakeLists.txt
). This replaces the old arch/
directory. CMake
relies on a two step command line process, similar to the old UNIX
“configure” and “compile.” Command line arguments are used to specify
paths and compilers. By default CMake is configured to build the source
code “out-of-place,” meaning that it does not populate the source code
directory with the build. The location for the build must be specified
by the user.
Once a compiler has been chosen, CMake generates local makefiles by invoking the cmake command with the proper arguments. One of those arguments selects that a local build of the NCEP libraries needed by EnKF/GSI will be conducted prior to the source code being built. The final step is to invoke a parallel build (make) of the code.
A significant advantage with using CMake to build the code, is that CMake automatically generates code dependencies each time a build is invoked, allowing the use of a parallel make as the final step of the build. This greatly reduces the time it take to complete the build. Because of the parallel compiling, the time to complete the CMake build is typically a quarter of the time needed for the serial DTC “configure” and “compile” to complete.
Once the build is complete, the EnKF executable is placed in the bin
directory of the build directory. Note that the name of the executables
and their location differs from the traditional DTC build.
Summary of CMake build steps:
- Set up the build environment
- Select a specific compiler by loading modules or setting paths as needed
- Set the environment path for the NetCDF library (The HDF5 library is also needed if your NetCDF version is 4+).
- Set the LAPACK/MKL mathematics libraries as needed.
- Create a directory for your build and change into it.
- Run the cmake command with the appropriate arguments.
- Run make
- The executables are located in the build directory within the
bin
directory.
Example build using the Intel Fortran Compiler¶
This section details how to build EnKF on the NCAR supercomputer Cheyenne, using the Intel V18.0.1 fortran compiler, the 2018 MKL libraries, NetCDF v4.6.1, and the 2018 Intel MPI libraries.
Set up your build environment
module purge module load cmake/3.9.1 module load intel/18.0.1 ncarenv/1.2 ncarcompilers/0.4.1 mkl/2018.0.1 module load netcdf/4.6.1 module load impi/2018.1.163
Next, create a build directory and move into it.
mkdir ./build cd build
Set environment variables specifying the
C
,C++
, and Fortran compilers. Also set the path location for the top of the source code directory. These can be set in your dot files if you plan to always use the same compiler and source code base. For bash,export CC=icc export CXX=icpc export FC=mpif90 export CORE_DIR /path_to_the_source_code_directory/
and for csh
setenv CC icc setenv CXX icpc setenv FC mpif90 setenv CORE_DIR /path_to_the_source_code_directory/
These environment variables can be over ruled by using command line arguments.
From within the build directory, invoke the cmake command to build the global GFS version of EnKF.
cmake -DBUILD_CORELIBS=ON $CORE_DIR
Other versions of EnKF require using optional build arguments. To build the regional EnKF with the WRF bindings switch off the global GFS versions and turn on the regional WRF version.
cmake -DBUILD_CORELIBS=ON -DBUILD_WRF=ON -DBUILD_GFS=OFF $CORE_DIR
Or if you wish to specify everything on the command line
cmake -DBUILD_CORELIBS=ON -DBUILD_WRF=ON -DBUILD_GFS=OFF \ -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icpc \ -DCMAKE_Fortran_COMPILER=mpif90 $CORE_DIR
The makefiles customized to your platform and the environment, are now created.
Run make (parallel compile) on eight cores
make -j8
Because CMake has already figured out the dependencies, the make can run in parallel.
Lets review the relevant arguments for cmake when building EnKF. The
first is -DBUILD_CORELIBS=ON
. This argument directs CMake to look in
the libsrc/
directory to build the NCEP libraries needed for EnKF
and GSI.
Next, if we wish to build the regional EnKF with the WRF bindings, an
optional argument to turn off the global GFS version -DBUILD_GFS=OFF
is needed as well as a second argument to turn on the regional EnKF with
the WRF bindings -DBUILD_WRF=ON
.
The argument CMAKE_C_COMPILER
specifies the C
compiler to use
with the build. For the Intel compiler, use icc
. The argument
CMAKE_CXX_COMPILER
specifies the C++
compiler to use with the
build. For the Intel compiler, use icpc
. The argument
CMAKE_Fortran_COMPILER
specifies the Fortran compiler to use with
the build. Because this is an parallel MPI code, on Cheyenne use
mpif90
instead of ifort
. Typically, the only time one would
explicitly specify the compilers on the command line is when you wish to
override a default choice set in your dot files or if you were to create
a batch script to conduct the build.
The argument $CORE_DIR
indicates the location of the build
directory. This can be set to an environment variable or entered
explicitly on the command line. The final statement make -j 8
invokes a parallel call to make using eight processors. This speeds up
the compilation considerably.
To run the cmake command with a compiler other than Intel only requires
replacing the values for CMAKE_C_COMPILER
, CMAKE_CXX_COMPILER
,
and CMAKE_Fortran_COMPILER
that correspond to the new compiler.
Table [ch2_tble3] lists these values.
Compiler | FC |
C |
C++ |
---|---|---|---|
Intel | icc |
icpc |
mpif90 |
PGI | pgcc |
pgcc++ |
mpif90 |
GNU | gcc |
g++ |
mpif90 |
[ch2_tble3]
Default Options¶
The CMake build has a number of options to when building. These are
enacted by adding the option with the prefix -D
to the command line
call to cmake. For instance, in the example above, the
BUILD_CORELIBS
option is enacted by adding -DBUILD_CORELIBS=ON
to the cmake call. The currently available command line build options
are:
BUILD_ENKF
: builds the enkf executable (default is ON)BUILD_CORELIBS
: Attempts to build the core libraries froim sourceUSE_WRF
: Builds GSI with WRF dependencies (default is ON)BUILD_GFS
: Builds ENKF using GFS (default is ON)BUILD_WRF
: Builds ENKF using WRF (default is OFF)BUILD_NMMB
: Builds ENKF using NMMB (default is OFF)
Note that by default EnKF built is global with GFS. See the readme file README.cmake in the top directory for a current list of options; or look in the CMakeLists.txt in the top directory and search for options.
System Requirements and Compiler Specific Notes¶
The EnKF source code is written in FORTRAN, FORTRAN 90, and C. In addition, the parallel executables require some flavor of MPI and OpenMP for the distributed memory parallelism. The I/O relies on the NetCDF I/O libraries. And CMake is needed to build the code.
The basic requirements for building and running the GSI system are the following:
- CMake V3+
- A FORTRAN compiler that supports the 2003 or newer standard
- C compiler
- MPI v1.2+
- OpenMP
- NetCDF V4.2+ and HDF5
- LAPACK and BLAS mathematics libraries, or equivalent
In addition, GSI and EnKF require linear algebra libraries such as LAPACK and BLAS. The Intel compiler usually comes with a vendor provided mathematics library known as the Mathematics Kernel Libraries or MKL for short. While most installations of the Intel compiler come with the MKL libraries installed not all do. In addition, if it is not installed correctly, the ifort compiler does not automatically load the library. It is therefore necessary to set the LAPACK_PATH variable to the location of the MKL libraries when using the Intel compiler. You may need to ask your system administrator for the correct path to these libraries.
IBM systems typically come installed with the LAPACK equivalent ESSL library that links automatically. Likewise, the PGI compiler often comes with a vendor provided version of LAPACK that links automatically with the compiler. The Gnu compiler is the only one that does not come with any installed version of the LAPACK and BLAS libraries.
Because all but the last of these tools and libraries are typically the purview of system administrators to install and maintain, they are lumped together here as part of the basic system requirements.
Compilers Tested for Release¶
Version 1.3 of the DTC community EnKF system has been extensively tested with a variety of compilers and versions of those compilers, on standard Linux platforms. To summarize, Intel compiler versions 16-18 should build and run without issue. Early version 15 has some known issues, but 15.1, 15.3, and 15.6 should work. For the Portland Group compiler, versions 16-18 should work without issue, and Gnu version 6.3, 7.1, and 7.3 are known to work. Version 18 of Gnu has issues and is not currently supported. See table [ch2_tble4] for the complete list compilers and versions tested.
The following Linux compiler combinations have been fully tested:
Compiler manifacturer | Compiler version | MPI version |
---|---|---|
intel | ||
18.1.163 | impi_5.1.2.150 | |
18.0.3.222 | mvapich2_1.8 | |
18.0.3.222 | impi_5.1.1.109 | |
18.0.1.163 | impi_5.1.2.150 | |
18.0.1 | impi_2018.1.163 | |
18.0.1 | mpt_2.15f | |
18.0.1 | mvapich2_2.2 | |
18.0.1 | openmpi_3.1.0 | |
18.0.0.128 | impi_5.1.2.150 | |
17.0.5.239 | mvapich2_2.1a | |
17.0.1 | impi_2017.1.132 | |
17.0.1 | mpt_2.15f | |
17.0.1 | mvapich2_2.2 | |
17.0.1 | openmpi_3.0.1 | |
2016.2.181 | impi_5.1.1.109 | |
2016.2.181 | mvapich2_1.8 | |
16.1.150 | mvapich2_2.1a | |
16.1.150 | impi_5.1.2.150 | |
16.0.3 | impi_5.1.3.210 | |
16.0.3 | mpt_2.15f | |
16.0.3 | openmpi_3.1.0 | |
16.0.1.150 | mvapich2_2.1a | |
16.0.1 | impi_5.1.2.150 | |
15.6.233 | mvapich2_2.1a | |
15.3.187 | mvapich2_2.1a | |
15.1.133 | mvapich2_2.1a | |
14.0.2 | mvapich2_2.1a | |
pgi | ||
18.5 | mvapich2_2.1a | |
18.3. | mvapich2_2.1a | |
18.1 | mvapich2_2.1a | |
17.10 | mvapich2_2.1a | |
17.9 | openmpi_2.1.2 | |
17.7 | mvapich2_2.1a | |
17.5 | mvapich2_2.1a | |
16.10 | mvapich2_2.1a | |
16.9 | mvapich2_2.1a | |
16.7 | mvapich2_2.1a | |
16.5 | mvapich2_2.1a | |
gnu | ||
7.3.0 | mvapich2_2.2 | |
7.3.0 | openmpi_3.0.1 | |
7.1.0 | mvapich2_2.2 | |
6.3.0 | openmpi_3.0.0 |
[ch2_tble4]
Unforeseen build issues may occur when using older compiler and library versions. As always, the best results will be achieved by using the most recent compiler versions.
Getting Help and Reporting Problems¶
Should the user experience any difficulty building EnKF on their system, please first confirm that GSI has successfully built. Feel free to contact the community EnKF support at
enkf-help@ucar.edu
for assistance.
At a minimum, when reporting issues building the code, please include a copy of the EnKF build log.