These instructions will allow users to download the current NEMSfv3gfs source code including CCPP capabilities.  The authoritative repositories are located on github.com, in the NCAR organizational space.  We will be using the NOAA RDHPCS system, theia, for these exercises!

If you are not familiar with git, please refer to the following pages:

https://help.github.com/ —

https://www.atlassian.com/git/tutorials/comparing-workflows/forking-workflow

Start by going into a working directory in your project space, i.e.

cd ${YOUR_WORK_DIR}

Next, clone the top-level git repository and all of it's submodules, checking out the appropriate tag (emc_training_march_2019)

git clone --recursive -b emc_training_march_2019 https://github.com/NCAR/NEMSfv3gfs

Note. Potential errors while checking out the submodule NCEPLIBS-pyprodutil (e.g., "gerrit: Name or service not known") can be ignored.

You will now have a local working copy of the source code:

cd NEMSfv3gfs

Compile the code

This exercise will compile the NEMSfv3gfs code with CCPP in static mode, using the control physics suite (operational/Zhao-Carr microphysics).  An optional second step will recompile and run with the GFDL microphysics (updated control).

Using the same build mechanisms as the baseline NEMSfv3gfs, 

cd tests/

• For bash shell

./compile.sh $PWD/../FV3 theia.intel "CCPP=Y HYBRID=N STATIC=Y SUITE=suite_FV3_GFS_2017_updated.xml" ccpp1 YES NO 2>&1 | tee compile1.log

• For t/csh shell

./compile.sh $PWD/../FV3 theia.intel "CCPP=Y HYBRID=N STATIC=Y SUITE=suite_FV3_GFS_2017_updated.xml" ccpp1 YES NO |& tee compile1.log

• The compile.sh script takes up to 6 arguments. 

PATHTR BUILD_TARGET [ MAKE_OPT [ BUILD_NR ] [ clean_before ] [ clean_after ]

• so, in the example command-line given above,
  • PATHTR = $PWD/../FV3
  • BUILD_TARGET = theia.intel
  • MAKE_OPT = "CCPP=Y HYBRID=N STATIC=Y SUITE=suite_FV3_GFS_2017_updated.xml"
  • BUILD_NR (name of the fv3.exe executable) = ccpp1 (fv3_ccpp1.exe)
  • clean_before = YES
  • clean_after = NO

While the code is compiling, review the SDF file, which can be found in the ccpp/suites directory:

cat ../ccpp/suites/suite_FV3_GFS_2017_updated.xml

Note the use of interstitial code (this was/is in GFS_physics_driver!) in addition to the physics schemes.

While the code is compiling, you may start setting up the case run directory (next page).

Completed compilation

The result of the compile.sh script will be an executable named fv3_ccpp1.exe in the tests/ subdirectory.  

The log file will show how the CCPP is compiled, and what flags are used in the NEMSfv3gfs build to include CCPP.

The compilation (on theia) generally takes about 15 minutes.

 

Now that you have successfully compiled the NEMSfv3gfs code, use these instructions to run a simple case using the CCPP and a SDF (suite definition file).

This example uses a low-resolution global grid (C96), and the operational (2017) GFS physics suite.  The directory may be copied to any local/personal working directory.

Copy this example directory (theia) into your work space:

cp -r /scratch4/BMC/gmtb/emc_training_march_2019/handson_day1/fv3_ccpp_control .

cd fv3_ccpp_control 

 

Edit the batch script to use your own project account:  run_ccpp.sh

 

Link (or copy) your executable into this run directory:

ln -s ${YOUR_WORK_DIR}/NEMSfv3gfs/tests/fv3_ccpp1.exe fv3.exe

 

Submit the run script to the batch queue:

qsub run_ccpp.sh
 

NOTES:

1. Look at input.nml --- find the ccpp_suite file name listed there
2. Note that there is a CCPP SDF file (suite_FV3_GFS_2017_updated.xml) in this directory.  Because we compiled the static build, this SDF file MUST MATCH the one used to compile (see previous step)
3. Once the job has finished, the usual various log and output files are available for review

• job log output is in out and err
• forecast files are in NetCDF format, native grid (6 tiles), and dyn and phys (3d and 2d) fields, i.e.
  • phyf018.tile1.nc
  • dynf015.tile1.nc

The code used in the first example was compiled using the CCPP static build option, which has the best timing/memory performance.  However, this means that you will need to recompile to run a different suite.

 

In this exercise, you will compile and run the code using the GFDL microphysics instead of the Zhao-Carr microphysics used in the first example.

 

Compile

Repeat the steps in the Compile section of the first example, this time using the following SDF:

suite_FV3_GFS_2017_updated_gfdlmp.xml

• For bash shell

./compile.sh $PWD/../FV3 theia.intel "CCPP=Y HYBRID=N STATIC=Y SUITE=suite_FV3_GFS_2017_updated_gfdlmp.xml" ccpp2 YES NO 2>&1 | tee compile2.log

• For t/csh shell

./compile.sh $PWD/../FV3 theia.intel "CCPP=Y HYBRID=N STATIC=Y SUITE=suite_FV3_GFS_2017_updated_gfdlmp.xml" ccpp2 YES NO |& tee compile2.log

Compare this SDF with the one used in the previous exercise - you will see a change in the physics parameterizations called in the suite.

 

Run

Copy a new run directory, fv3_ccpp_gfdlmp, and follow the same steps.

cp -r /scratch4/BMC/gmtb/emc_training_march_2019/handson_day1/fv3_ccpp_gfdlmp .

cd fv3_ccpp_gfdlmp

 

Edit the batch script to use your own project account:  run_ccpp.sh

 

Link (or copy) your executable into this run directory:

ln -s ${YOUR_WORK_DIR}/NEMSfv3gfs/tests/fv3_ccpp2.exe fv3.exe

 

Submit the run script to the batch queue:

qsub run_ccpp.sh

NOTE:  the only difference between these two cases, are the SDF and the namelist input.nml.

 

OPTIONAL exercise: 

Re-compile using the "STATIC=N" option, and run each of these cases using the new executable.  With the dynamic build option (STATIC=N), different SDFs can be used at runtime with the same executable.

Be sure to use the "clean-before" option YES, so that the code and libraries are recompiled correctly!  This is the 5th argument to compile.sh

Compile

Repeat the steps in the Compile section of the first example, this time using the dynamic build (no build-time SDF needed).

• For bash shell

./compile.sh $PWD/../FV3 theia.intel "CCPP=Y HYBRID=N STATIC=N " ccpp3 YES NO 2>&1 | tee compile.log

• For t/csh shell

./compile.sh $PWD/../FV3 theia.intel "CCPP=Y HYBRID=N STATIC=N" ccpp3 YES NO |& tee compile.log

 

Run

You may re-use the previous run directories (fv3_ccpp_control or fv3_ccpp_gfdl) to run either case with this dynamic executable.

Link (or copy) your dynamic executable into this run directory:

rm fv3.exe

ln -s ${YOUR_WORK_DIR}/NEMSfv3gfs/tests/fv3_ccpp3.exe fv3.exe

When using the dynamic build options, be sure to add the ccpp library path to your LD_LIBRARY_PATH envar.  For example, in this run directory, in the file module-setup.sh file, update:

export LD_LIBRARY_PATH=${YOUR_WORK_DIR}/NEMSfv3gfs/ccpp/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}

Submit the run script to the batch queue:

qsub run_ccpp.sh