From a terminal:

And follow the instructions you received (Yubikey, Duo, other) to authenticate and complete the login.

For convenience, set some environment variables to more easily navigate in your work space on Cheyenne. In this tutorial, we will use /glade/scratch:

where $USER is your login/username, and $SR_WX_APP_TOP_DIR is the path in which you will clone the GitHub repository in the next step. Go to your work space on Cheyenne:

Estimated time: 5 minutes

The necessary source code is publicly available on GitHub.  To clone the ufs-v1.0.1 branch of the ufs-srweather-app repository:

For convenience, set:

The UFS SRW App uses components from multiple repositories, including: the regional workflow, the UFS_UTILS pre-processor utilities, the UFS Weather Model, and the Unified Post Processor (UPP).  Obtaining the necessary external repositories is simplified by the use of manage_externals:

The checkout_externals script uses the configuration file Externals.cfg in the top level directory and will clone the regional workflow, pre-processing utilities, UFS Weather Model, and UPP source code into the appropriate directories under your regional_workflow and src directories.  The process will take several minutes and show the following progress information on the screen:

Once the externals have been downloaded, you can proceed to the next step, building the executables.

Estimated time: 20-25 minutes

Instructions for loading the proper modules and/or setting the correct environment variables can be found in the env/ directory in files named build_<platform>_<compiler>.env. The commands in these files can be directly copy-pasted to the command line or the file can be sourced.

Make the following changes to build_cheyenne_intel_tcsh.env.  Change

to

The modified script can then be sourced as follows:

to set the build environment.

Build the executables as follows (for both bash and csh/tcsh users):

Run cmake to set up the Makefile, then run make:

Output from the build will be in the $SR_WX_APP_TOP_DIR/build/build.out file. When the build completes, you should see the forecast model executable NEMS.exe and eleven pre- and post-processing executables in the $SR_WX_APP_TOP_DIR/bin directory.

Estimated time: 15 minutes

Generating the workflow requires three steps:

• Set experiment parameters in config.sh
• Set Python and other environment parameters
• Run the generate_FV3LAM_wflow.sh script

The workflow requires a file called config.sh to specify the values of your experiment parameters. Make a copy of the example template config.community.sh file provided in the $SR_WX_APP_TOP_DIR/regional_workflow/ush directory (from the build directory):

Edit the config.sh file on Cheyenne for running the FV3_GFS_v15p2 physics suite for 48 hours starting at 00 UTC on 15 June 2019 on a 25 km predefined CONUS domain.  Since this is the default case, only the following variables need to be modified in config.sh:

where ACCOUNT is an account you can charge. We will be using pre-staged initialization data, so uncomment USE_USER_STAGED_EXTRN_FILES="TRUE".

EXTRN_MDL_SOURCE_BASEDIR_ICS and EXTRN_MDL_SOURCE_BASEDIR_LBCS contain the locations of the initial and lateral boundary conditions, and EXTRN_MDL_FILES_ICS and EXTRN_MDL_FILES_LBCS contain the filenames of the initial and lateral boundary conditions.

Load the appropriate Python environment for Cheyenne:

If you see something like:

This will occur if this is the first time you have used the NCAR Python Library (NPL), type y, to get the following output:

The workflow can be generated with the command:

The generated workflow will be located in ${SR_WX_APP_TOP_DIR}/../expt_dirs/${EXPT_SUBDIR} where ${SR_WX_APP_TOP_DIR} is the directory where the ufs-srweather-app was cloned in Step 1, and ${EXPT_SUBDIR} is the experiment name set above in the config.sh file: test_CONUS_25km_GFSv15p2

Estimated time: 25 min

Set the environment variable $EXPTDIR to your experiment directory to make navigation easier:

The workflow can be run using the./launch_FV3LAM_wflow.sh script which contains the rocotorun and rocotostat commands needed to launch the tasks and monitor the progress. There are two ways to run the ./launch_FV3LAM_wflow.sh script: 1) manually from the command line or 2) by adding a command to the user’s crontab.  Both options are described below.

To run the workflow manually:

Once the workflow is launched with the launch_FV3LAM_wflow.sh script, a log file named log.launch_FV3LAM_wflow will be created in $EXPTDIR.

To see what jobs are running for a given user at any given time, use the following command:

This will show any of the workflow jobs/tasks submitted by rocoto (in addition to any unrelated jobs the user may have running).  Error messages for each task can be found in the task log files located in the $EXPTDIR/log directory. In order to launch more tasks in the workflow, you just need to call the launch script again in the $EXPTDIR directory as follows:

until all tasks have completed successfully.  You can also look at the end of the $EXPT_DIR/log.launch_FV3LAM_wflow file to see the status of the workflow.  When the workflow is complete, you no longer need to issue the ./launch_FV3LAM_wflow.sh command.

To run the workflow via crontab:

For automatic resubmission of the workflow (e.g., every 3 minutes), the following line can be added to the user’s crontab:

and insert the line:

The state of the workflow can be monitored using the rocotostat command, from the $EXPTDIR directory:

The workflow run is completed when all tasks have “SUCCEEDED”, and the rocotostat command will output the following:

If something goes wrong with a workflow task, it may end up in the DEAD state:

This means that the DEAD task has not completed successfully, so the workflow has stopped.  Go to the $EXPTDIR/log directory and look at the make_grid.log file, in this case, to identify the issue. Once the issue has been fixed, the failed task can be re-run using the rocotowind command:

where -c specifies the cycle date (first column of rocotostat output) and -t represents the task name (second column of rocotostat output). After using rocotorewind, the next time rocotorun is used to advance the workflow, the job will be resubmitted.

The plotting scripts are under the directory  $SR_WX_APP_TOP_DIR/regional_workflow/ush/Python

Change to the script directory

Load the appropriate python environment for Cheyenne, 

If you see something like:

This will occur if this is the first time you have used the NCAR Python Library (NPL), type y, to get the following output:

Run the python plotting script (Estimated time: 5 minutes)

Here plot_allvar.py is the plotting script, and six command line arguments are:

1. 2019061500 is cycle date/time in YYYYMMDDHH format, 
2. 6 is the starting forecast hour
3. 12 is the ending forecast hour
4. 6 is the forecast hour increment
5. The top level of the experiment directory is /glade/scratch/$USER/expt_dirs/test_CONUS_25km_GFSv15p2
6. /glade/p/ral/jntp/UFS_SRW_app/tools/NaturalEarth is the base directory of the cartopy shapefiles 

Display the output figures

The output files (in .png format) can be found in the directory /glade/scratch/$USER/expt_dirs/test_CONUS_25km_GFSv15p2/2019061500/postprd.

The png files can be found using ls command after changing to that directory: 

The png file can be displayed by the command display:

Here is an example plot of 10m surface wind at F006:

Running with batch script

(Estimated time: 25 min wall time for 48 forecast hours, every 3 hours)

There are some batch scripts under the same directory with the python scripts. If the batch script qsub_job.sh is being used to run the plotting script, multiple environment variables need to be set prior to submitting the script besides the python environment:

For bash:

For csh/tcsh:

The HOMErrfs gives the location of regional workflow and EXPTDIR points to the experiment directory (test_CONUS_25km_GFSv15p2). In addition, the account to run the batch job in qsub_job.sh needs to be set.  You might wish to change the FCST_INC in the batch script for different plotting frequency, and increase the walltime to 25 minutes (for 48 forecast hours, every 3 hours).

Then the batch job can be submitted by the following command:

The running log will be in the file $HOMErrfs/ush/Python/plot_allvars.out when the job completes.

The output files (in .png format) will be located in the same directory as the command line run: $EXPTDIR/2019061500/postprd