Session 1: Set Up and Run the Model

admin Tue, 10/13/2020 - 18:01

This session will enable you to build and run a simulation with initial conditions from August 29, 2019, during Hurricane Dorian. The simulation is configured for a C96 (~100km) resolution. The code base includes the UFS Weather Model, with pre- and post-processing, running in the Common Infrastructure for Modeling Earth (CIME) workflow. In order to perform the test, you will run the Medium-Range Weather Application for 48 hours.

In step 8, you will use the workflow software to make a change to the namelist that increases the concentration of cloud condensation nucleii (CCN), and perform another 48 hour run with this change. Then you will compare the total cloud cover field before and after the change.

0. Log in to Cheyenne

arbetter Fri, 10/30/2020 - 12:41

Login from a terminal (ssh -Y YourUserName@cheyenne.ucar.edu) following the instructions you received (Yubikey, Duo, other) to authenticate and complete the login.

******************************************************************************
* Welcome to Cheyenne - October 30, 2020
******************************************************************************
                 Today in the Daily Bulletin (dailyb.cisl.ucar.edu)

        - NOAA postdoc fellowship program accepting application
        - Summer internships program now accepting applications
        - PBS job script documentation updated
Quick Start: www2.cisl.ucar.edu/resources/cheyenne/quick-start-cheyenne
User environment: www2.cisl.ucar.edu/resources/cheyenne/user-environment
Key module commands: module list, module avail, module spider, module help
CISL Help: support.ucar.edu -- 303-497-2400
--------------------------------------------------------------------------------

Go to your work space:

cd $HOME

cd /glade/u/home/$USER

where $USER is your login/username, or

cd ~

where ~ is shorthand for your home directory.

1. Clone the code

arbetter Wed, 10/21/2020 - 14:16

To download the Medium-Range Weather Application, with externals pointing to its constituent components, go to your $HOME directory and enter:

mkdir UFS_training

git clone https://github.com/ufs-community/ufs-mrweather-app.git -b ufs-v1.1.0 $HOME/UFS_training/my_ufs_sandbox

Then, to get the components

cd $HOME/UFS_training/my_ufs_sandbox 

./manage_externals/checkout_externals 

This will clone the ufs-v1.1.0 branch of the ufs-mrweather-app repository and check out all of the externals and sub-modules. The process will take several minutes and show the following progress information in the screen:

Processing externals description file : Externals.cfg
Checking status of externals: model, fv3gfs_interface, cime, emc_post, nems_interface,
Checking out externals: model, fv3gfs_interface, emc_post, cime, nems_interface,

The repositories URL, tag number and their location path of external and sub-modules are defined in the text file Externals.cfg. Users can check this file and match the new cloned repositories with their local paths.

Once the externals have been been downloaded, you can proceed to the next step, setting up the environment.

2. Set up the environment

arbetter Wed, 10/21/2020 - 14:14

Prior to building the model, the environment for CIME needs to be set up. This requires five environment variables:

UFS_INPUT: Path to location of input data. A directory named ufs_inputdata should be located under this directory. This can be in a shared location so multiple users can use the data.
UFS_SCRATCH: Writable directory where output for each case will go.
UFS_DRIVER: Default is nems, do not change.
CIME_MODEL: Default is ufs, do not change.
PROJECT: Account code on machine

Enter the following commands on Cheyenne. You may need to create a directory on /glade/scratch.

cd /glade/scratch/$USER

If you see something like

-bash: cd: /glade/scratch/$USER: No such file or directory

Then

cd /glade/scratch/

mkdir $USER

This directory should persist throughout the tutorial.

For bash:

export UFS_INPUT=/glade/p/ral/jntp/UFS_training

export UFS_SCRATCH=/glade/scratch/$USER/UFS_training

export UFS_DRIVER=nems

export CIME_MODEL=ufs  

export PROJECT=NJNT0007

For csh:

setenv UFS_INPUT /glade/p/ral/jntp/UFS_training

setenv UFS_SCRATCH /glade/scratch/$USER/UFS_training

setenv UFS_DRIVER nems

setenv CIME_MODEL ufs

setenv PROJECT NJNT0007

Note: You'll need to set the environments (export/setenv) each time you log in to do work with the MR-Weather-App. You can add these lines to your .bashrc/.cshrc file (found in your home directory) or put in a shell script that you can source each time you log in to cheyenne. Some exercises in this practical session involve loading different modules or versions of modules and/or editing namelists; you can use this page to reset back to the proper build/run environment for the baseline case.

3. Create a case

arbetter Thu, 10/22/2020 - 08:52

A case is an instance of a MRW model simulation. It is determined by:

component set
model grid
machine
compiler
any other additional customizations

Invoke create_newcase as follows from the cime/scripts directory using the following format:

cd $HOME/UFS_training/my_ufs_sandbox/cime/scripts

./create_newcase --case $UFS_SCRATCH/ufs-mrweather-app-workflow.c96 --compset GFSv15p2 --res C96 --workflow ufs-mrweather

This will

Create a case directory named /glade/scratch/$USER/UFS_training/ufs-mrweather-app workflow.c96
Use the GFSv15p2 CCPP physics suite, C96 grid and the UFS weather model with pre and post-processing steps

The output can be seen here. When the case directory is created, the build is done.

Creating Case directory /glade/scratch/$USER/UFS_training/ufs-mrweather-app-workflow.c96

Click here to move to step 4.

Create the case: screen output

arbetter Fri, 10/30/2020 - 13:10

You will see output such as the following:

Compset longname is FCST_ufsatm%v15p2_SLND_SICE_SOCN_SROF_SGLC_SWAV
Compset specification file is /glade/scratch/$USER/my_ufs_sandbox/src/model/FV3/cime/cime_config/config_compsets.xml
Automatically adding SESP to compset
Compset forcing is
ATM component is UFSATM Atmosphere with:CCPP physics version 15p2
LND component is Stub land component
ICE component is Stub ice component
OCN component is Stub ocn component
ROF component is Stub river component
GLC component is Stub glacier (land ice) component
WAV component is Stub wave component
ESP component is Stub external system processing (ESP) component
Pes specification file is /glade/scratch/$USER/my_ufs_sandbox/src/model/FV3/cime/cime_config/config_pes.xml
Compset specific settings: name is RUN_STARTDATE and value is 2019-08-29
Compset specific settings: name is START_TOD and value is 0
Compset specific settings: name is COMP_CLASSES and value is ATM
Compset specific settings: name is CHECK_TIMING and value is FALSE
Machine is cheyenne
Pes setting: grid match is a%C96
Pes setting: compset_match is ufsatm%v1
Pes setting: grid is a%C96_l%null_oi%null_r%null_g%null_w%null_z%null_m%null
Pes setting: compset is FCST_ufsatm%v15p2_SLND_SICE_SOCN_SROF_SGLC_SWAV_SESP
Pes setting: tasks is {'NTASKS_ATM': 108}
Pes setting: threads is {'NTHRDS_ATM': 1}
Pes setting: rootpe is {}
Pes setting: pstrid is {}
Pes other settings: {}
Pes comments: none
Compset is: FCST_ufsatm%v15p2_SLND_SICE_SOCN_SROF_SGLC_SWAV_SESP
Grid is: a%C96_l%null_oi%null_r%null_g%null_w%null_z%null_m%null
Components in compset are: ['ufsatm', 'slnd', 'sice', 'socn', 'srof', 'sglc', 'swav', 'sesp']
Using project from env PROJECT: P48503002
No charge_account info available, using value from PROJECT
ufs model version found: 1e9cdab
Batch_system_type is pbs
job is case.chgres USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S
job is case.run USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S
job is case.gfs_post USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S
job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S
Creating Case directory /glade/scratch/$USER/ufs-mrweather-app-workflow.c96

When the case directory is created, the build is done.

4. Set up the case run script

arbetter Thu, 10/22/2020 - 08:53

Move to this case directory to set up and build the model:

cd $UFS_SCRATCH/ufs-mrweather-app-workflow.c96

This step creates the scripts needed to build and run the model along with the namelist:

./case.setup

Note the new bld and run directories under your case directory:

/glade/scratch/$USER/ufs-mrweather-app-workflow.c96/bld
/glade/scratch/$USER/ufs-mrweather-app-workflow.c96/run

The bld directory will contain the executables ufs.exe, ncep_post and chgres_cube.exe when ./case.build is executed (next step). The run directory will be populated when the case is run. Note that chgres_cube.exe and ncep_post are links to the NCEPLIBS-ufs-v1.1.0 directory, since they are built with the NCEPLIBS.

You can see more details of the case setup by running:

./preview_run

./preview_namelists

5. Build the model executable

arbetter Thu, 10/22/2020 - 08:56

This step builds ufs.exe in about 8 minutes:

./case.build

The last part of the output looks like:

Building case in directory /glade/scratch/$USER/ufs-mrweather-app-workflow.c96 ...
Building atm with output to /glade/scratch/$USER/ufs-mrweather-app workflow.c96/bld/atm.bldlog.200930-094156
ufsatm built in 484.297560 seconds
Building ufs from
/glade/p/ral/jntp/$USER/my_ufs_sandbox/cime/../src/model/NEMS/cime/cime_config/bui ldexe with output to /glade/scratch/$USER/ufs-mrweather-app
workflow.c96/bld/ufs.bldlog.200930-094156
Time spent not building: 6.931199 sec
Time spent building: 485.038825 sec
MODEL BUILD HAS FINISHED SUCCESSFULLY

Once the build has finished successfully, check bld and run to see the results of the build process.

6. Run the case

arbetter Thu, 10/22/2020 - 08:58

Modify the runtime settings in env_run.xml for a low-resolution, 48 hour forecast:

Reduce run length from 5 days to 48 hours

./xmlchange STOP_OPTION=nhours,STOP_N=48

Reduce wallclock time from 12 hours to 30 minutes

./xmlchange JOB_WALLCLOCK_TIME=00:30:00

Turn off short term archiving

./xmlchange DOUT_S=FALSE

For the Tutorial, there is a reserved queue:

./xmlchange --force JOB_QUEUE=S4857724

Run the case:

./case.submit

See https://ufs-mrweather-app.readthedocs.io/en/ufs-v1.1.0/quickstart.html for more information on this command.

After running ./case.submit, you will see:

Submits 3 jobs for chgres, the model run and post: 

Submitted job id is 4433245.chadmin1.ib0.cheyenne.ucar.edu Submitted job case.chgres with id  

4433243.chadmin1.ib0.cheyenne.ucar.edu 

Submitted job case.run with id 4433244.chadmin1.ib0.cheyenne.ucar.edu Submitted job case.gfs_post with id  

4433245.chadmin1.ib0.cheyenne.ucar.edu

Monitor your job's progress in the queue:

qstat -u $USER

If needed, you can kill your job:

qdel JobID

How to tell if your job is completed:

cd $UFS_SCRATCH/ufs-mrweather-app-workflow.c96/run

tail run/chgres_cube.yymmdd-hhmmss.log: DONE.

tail run/ufs.log.<jobid>.yymmdd-hhmmss: PROGRAM nems HAS ENDED.

tail run/oi.hhh: PROGRAM UNIFIED_POST HAS ENDED.

7. Check your results

arbetter Thu, 10/22/2020 - 11:29

Your outputs will be in the $UFS_SCRATCH/ufs-mrweather-app-workflow.c96/run directory. After a successful run, the directory will look like this:

run directory

For visually checking the results of your run, we have provided a NCL script that plots 2-m temperature (tmp2m) and total precipitation (tprcp) at 48 h.

You will need to load NCL first:

module load intel/18.0.5 ncl/6.6.2

Then get and run the script:

cd $UFS_SCRATCH/ufs-mrweather-app-workflow.c96/run

wget https://raw.githubusercontent.com/wiki/ufs-community/ufs-mrweather-app/files/plot_ufs_sfcf.ncl

ncl plot_ufs_sfcf.ncl

To visualize the resulting images in png, use the command:

display plot_ufs_phyf_tmp2m.png

display plot_ufs_phyf_tprcp.png

The sample plots are below.

Now that you completed this step, the next step is to change a namelist option and run a second test to check your understanding of how results will change.

8. Build a new case and change some variables

arbetter Fri, 10/23/2020 - 10:01

You will need to create a new case directory for this experiment (we will use ufs-mrweather-app-workflow.c96_exp - note the "_exp" this time), and setup the new case.

cd $HOME/UFS_training/my_ufs_sandbox/cime/scripts

./create_newcase --case $UFS_SCRATCH/ufs-mrweather-app-workflow.c96_exp --compset GFSv15p2 --res C96 --workflow ufs-mrweather

cd $UFS_SCRATCH/ufs-mrweather-app-workflow.c96_exp

./case.setup

At this point, you can see the current values for CCN (ccn_o = 100 and ccn_l = 300) by looking at the input.nml file in the control run:

more $UFS_SCRATCH/ufs-mrweather-app-workflow.c96/run/input.nml

You will set the values through the workflow:

cd $UFS_SCRATCH/ufs-mrweather-app-workflow.c96_exp

Use your editor to append the following to the entries at the bottom of the file user_nl_ufsatm

ccn_l = 900.

ccn_o = 300.

Next you will build the case:

./case.build

Once that completes, change the run duration to 48 hours and reduce the time request. Then submit the case.

./xmlchange STOP_OPTION=nhours,STOP_N=48

./xmlchange JOB_WALLCLOCK_TIME=00:45:00

./xmlchange --force JOB_QUEUE=S4857724

./case.submit

Once again, you can monitor your jobs in the queue:

qstat -u $USER

And if needed, you can kill your job:

qdel JobID

As before, you should see the output files in the case run directory for the experiment:

ls $UFS_SCRATCH/ufs-mrweather-app-workflow.c96_exp/run

tail run/chgres_cube.yymmdd-hhmmss.log: DONE.

tail run/ufs.log.<jobid>.yymmdd-hhmmss: PROGRAM nems HAS ENDED.

tail run/oi.hhh: PROGRAM UNIFIED_POST HAS ENDED.

9. Compare outputs

arbetter Fri, 10/23/2020 - 11:13

Changing the concentration of CCN creates a noticeable difference in the total cloud cover. Now that you've completed both runs compare the total cloud cover before and after the CCN change.

We are providing a script that plots the differences between control and test runs for 2m temperature (tmp2m), total precipitation (tprcp), and atmospheric column total cloud cover (tcdc_aveclm) at 48h.

To get and run this NCL script execute the following commands. This script will produce png files for each variable being plotted. The argument "c=" is the control run directory, and "t=" is the test run directory.

cd $UFS_SCRATCH

wget https://raw.githubusercontent.com/wiki/ufs-community/ufs-mrweather-app/files/plot_ufs_sfcf_diff.ncl

ncl plot_ufs_sfcf_diff.ncl 'c="ufs-mrweather-app-workflow.c96/run"' 't="ufs-mrweather-app-workflow.c96_exp/run"'

View the total cloud cover:

display plot_ufs_phyf_diff_tcdc_aveclm.png

Your NCL output of atmospheric column total cloud cover difference should look like this:

You can also look at the 2-meter temperature and precipitation:

display plot_ufs_phyf_diff_tmp2m.png

display plot_ufs_phyf_diff_tprcp.png

Congratulations! You've completed Session 1.