Preliminary work: METplus setup

Preliminary work: METplus setup

Welcome to the online METplus tutorial!

In this preliminary session, you'll be guided through the necessary steps that need to happen before executing the tutorial's METplus commands. Additionally, we'll go over some of the common METplus wrapper configuration settings, timing control, and the directory structure of the METplus wrappers.

If you have already started working on the tutorial and are rejoining with the same METplus terminal session active (i.e. you have not changed any of the environmental configuration settings), you do not need to redo this section and can jump directly to the session of interest.

If you are just starting out, or you are returning with a new terminal session, you will need to revisit this session to get your environment correctly set. So select the next page of "METplus setup" to get started!

 

 

jopatz Mon, 08/28/2023 - 15:17

METplus Setup

METplus Setup

METplus Overview

METplus is a set of Python modules that have been developed with the flexibility to run the MET applications for various use cases or scenarios. The goal is to simplify the running of MET for scientists. Currently, the primary means of achieving this is through the use of METplus configuration files, aka "conf files." It is designed to provide a framework in which additional use cases can be added. The conf file implementation utilizes a Python package called produtil that was developed by NOAA/NCEP/EMC for the HWRF system.

Please be sure to follow the instructions in order.

METplus Useful Links

The following links are just for reference, and not required for this practical session. METplus releases are available on GitHub along with sample data and instructions.

METplus User's Guide
METplus Releases on GitHub

The source code for the METplus components are publicly available in the following GitHub repositories:

New features are developed and bugs are tracked using GitHub issues in each repository.

admin Mon, 06/24/2019 - 15:58

METplus: Initial setup

METplus: Initial setup

Prerequisites: Software

The Requirements section in the Software Installation chapter of the METplus User's Guide lists the software and Python packages that are required to run the METplus wrappers. Note that there is a core set of requirements needed to run the METplus wrappers and additional requirements needed to utilize some of the more advanced features.

 

Prerequisites: Environment

This tutorial assumes MET 11.0.1 and METplus 5.0.1 have been installed on the machine used to run the exercises and that the sample input data is also available. If this is not the case, please navigate to the Software Installation chapter of the METplus User's Guide.

The following instructions are required so the commands in this tutorial can be copied and run without modification. The steps involve creating a working directory to store files used/generated by the tutorial and configuring a simple script that can be run to set up the shell environment. Once configured correctly, the script can be run upon returning to the tutorial content to easily resume progress.

Click on the link that corresponds to the environment you are setting up.
If you are running the tutorial instructions on your own computer, select the bash or csh instructions depending on which shell you prefer. We recommend using bash if you do not have a preference.

Pre-Configured Environments

Setting up the Tutorial Environment on Hera (NOAA)

Setting up the Tutorial Environment on Jet (NOAA)

Setting up the Tutorial Environment on Cheyenne (NCAR)

Setting up the Tutorial Environment on Seneca (NCAR)

User Configured Environments

Setting up the Tutorial Environment (bash)

Setting up the Tutorial Environment (csh)

 

admin Mon, 06/24/2019 - 15:59

Setting up the Tutorial Environment on Cheyenne (NCAR)

Setting up the Tutorial Environment on Cheyenne (NCAR)

Setting up the Tutorial Environment on Cheyenne (NCAR)

The following instructions should be run if configuring the shell environment to run the METplus Tutorial on Cheyenne (NCAR). If you are running on your own computer or a NOAA machine that has been set up to run the tutorial, please go back and click the appropriate link for those instructions.

Create a Working Directory

Create a directory called 'METplus-5.0.0_Tutorial' to hold all of the files you will create during the tutorial. This can be any directory that you have write permission.

In the following instructions, change "/path/to" to the directory you chose: EDIT AFTER COPYING and BEFORE HITTING RETURN!
Change directory to the location where you want to put your tutorial files.
cd /path/to
Create a directory called METplus-5.0.0_Tutorial.
This directory will contain all of your tutorial work, including configuration files, output data, and any other notes you'd like to keep.
mkdir METplus-5.0.0_Tutorial
Change directory into the tutorial directory.
cd METplus-5.0.0_Tutorial

Create Directories for Configuration Files and Output Data

mkdir user_config
mkdir output

Obtain the Tutorial Setup Script

Copy the tutorial setup shell script to configure your runtime environment for use with METplus. You will also copy over a METplus .conf file that you will use to run:
cp /glade/p/ral/jntp/MET/METplus/Tutorial_Files/METplus-5.0.0_Tutorial_Files/tutorial-5.0.0.conf ./tutorial.conf
Run the command that corresponds your default shell.
DO NOT RUN BOTH SETS OF COMMANDS.
If you are unsure which shell you are using, run the command "echo $SHELL" to check.
If you are using bash:
cp /glade/p/ral/jntp/MET/METplus/Tutorial_Files/METplus-5.0.0_Tutorial_Files/METplus-5.0.0_TutorialSetup.cheyenne.bash ./METplus-5.0.0_TutorialSetup.sh
If you are using csh:
cp /glade/p/ral/jntp/MET/METplus/Tutorial_Files/METplus-5.0.0_Tutorial_Files/METplus-5.0.0_TutorialSetup.cheyenne.csh ./METplus-5.0.0_TutorialSetup.sh
Next navigate to the Verify Environment is Set Correctly page.
mccabe Thu, 07/29/2021 - 09:50

Setting up the Tutorial Environment on Jet (NOAA)

Setting up the Tutorial Environment on Jet (NOAA)

Setting up the Tutorial Environment on Jet (NOAA)

The following instructions should be run if configuring the shell environment to run the METplus Tutorial on Jet (NOAA). If you are running on your own computer or an NCAR machine that has been set up to run the tutorial, please go back and click the appropriate link for those instructions.

Create a Working Directory

Create a directory called 'METplus-5.0.0_Tutorial' to hold all of the files you will create during the tutorial. This can be any directory that you have write permission.

In the following instructions, change "/path/to" to the directory you chose: EDIT AFTER COPYING and BEFORE HITTING RETURN!
Change directory to the location where you want to put your tutorial files.
cd /path/to
Create a directory called METplus-5.0.0_Tutorial. This directory will contain all of your tutorial work, including configuration files, output data, and any other notes you'd like to keep.
mkdir METplus-5.0.0_Tutorial
Change directory into the tutorial directory.
cd METplus-5.0.0_Tutorial

Create Directories for Configuration Files and Output Data

mkdir user_config
mkdir output

Obtain the Tutorial Setup Script

Copy the tutorial setup shell script to configure your runtime environment for use with METplus. You will also copy over a METplus .conf file that you will use to run:
cp /lfs1/HFIP/dtc-hurr/METplus/METplus-5.0.0_Tutorial_Files/METplus-5.0.0_TutorialSetup.jet.`basename $SHELL` ./METplus-5.0.0_TutorialSetup.sh
cp /lfs1/HFIP/dtc-hurr/METplus/METplus-5.0.0_Tutorial_Files/tutorial-5.0.0.conf ./tutorial.conf

 

Next navigate to the Verify Environment is Set Correctly page.
mccabe Wed, 11/17/2021 - 13:07

Setting up the Tutorial Environment on Seneca (NCAR)

Setting up the Tutorial Environment on Seneca (NCAR)

Setting up the Tutorial Environment on Seneca (NCAR)

The following instructions should be run if configuring the shell environment to run the METplus Tutorial on Seneca (NCAR). If you are running on your own computer or a NOAA machine that has been set up to run the tutorial, please go back and click the appropriate link for those instructions.

Create a Working Directory

Create a directory called 'METplus-5.0.0_Tutorial' to hold all of the files you will create during the tutorial. This can be any directory that you have write permission.

In the following instructions, change "/path/to" to the directory you chose: EDIT AFTER COPYING and BEFORE HITTING RETURN!
Change directory to the location where you want to put your tutorial files.
cd /path/to
Create a directory called METplus-5.0.0_Tutorial.
This directory will contain all of your tutorial work, including configuration files, output data, and any other notes you'd like to keep.
mkdir METplus-5.0.0_Tutorial
Change directory into the tutorial directory.
cd METplus-5.0.0_Tutorial

Create Directories for Configuration Files and Output Data

mkdir user_config
mkdir output

Obtain the Tutorial Setup Script

Copy the tutorial setup shell script to configure your runtime environment for use with METplus. You will also copy over a METplus .conf file that you will use to run:
cp /d1/projects/METplus/METplus-5.0.0_Tutorial_Files/tutorial-5.0.0.conf ./tutorial.conf
cp /d1/projects/METplus/METplus-5.0.0_Tutorial_Files/METplus-5.0.0_TutorialSetup.seneca.`basename $SHELL` ./METplus-5.0.0_TutorialSetup.sh
 

(OPTIONAL) Set up conda Environment for METplus Analysis and s2s Tutorial Sections

Due to the need for external package dependencies for the METplus Analysis and s2s sections, the following instructions are also necessary. If you do not plan on executing the commands of those sections, you can skip these steps.

First, ensure that you are running the bash environment:

bash

Now, verify that you have a .condarc file in your /home/user directory. If the file does not exist, create one. In that file, you will need the following entries, in the exact same format (dashes and colons):

channels:
- defaults
- conda-forge
- anaconda

Verify that you are using the most recent version of anaconda3 in your .bashrc file. Verify that your conda_setup in your .bashrc file is using /usr/local/anaconda3/bin/conda. You should see a code block similar to following:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/usr/local/anaconda3/bin/conda' 'shell.bash' 'hook'2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/usr/local/anaconda3/etc/profile.d/conda.sh" ]; then
. "/usr/local/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/usr/local/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<

If you do have to make any changes to your .bashrc file, it would be best at this point to sign out, then sign back into your Seneca connection from the terminal. Errors have been found to happen even after sourcing your .bashrc file for the updated content.

From the command line, enter the following:

/usr/local/anaconda3/bin/conda create -n metplus_analysis_tutorial_env python=3.8
mamba -y

If this was successful, you will see the following at the end of the output streamed to the screen:

Downloading and Extracting Packages
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate metplus_analysis_tutorial_env
#
# To deactivate an active environment, use
#
# $ conda deactivate

This creates a basic Python 3.8 conda environment named ‘metplus_analysis_tutorial_env’ that contains mamba (to accelerate installing the remaining Python packages).

Activate this environment by running the following from the command line:

conda activate metplus_analysis_tutorial_env

Install the following packages to the conda environment by running the following from the command line (using mamba to speed up the installation):

mamba install imageio=2.19.3 imutils=0.5.4 matplotlib=3.5.2 metpy=1.3.1 netcdf4=1.6.2 numpy=1.24.2 pandas=1.5.1 pint=0.19.2 plotly=5.9.0 pytest=7.2.1 python-kaleido=0.2.1 pyyaml=6.0 scikit-image=0.19.3 scipy=1.8.1 lxml=4.9.1 eofs=1.4.0 scikit-learn=1.2.0 -y

You will see many lines of information streaming to your screen. You will see following output to your screen if everything was successful:

Preparing transaction: done
Verifying transaction: done
Executing transaction: /
Installed package of scikit-learn can be accelerated using scikit-learn-intelex.
More details are available here: https://intel.github.io/scikit-learn-intelex

For example:
$ conda install scikit-learn-intelex
$ python -m sklearnex my_application.py
done

Now you are ready to run the METplus Analysis tutorials.

Next navigate to the Verify Environment is Set Correctly page.
mccabe Wed, 06/30/2021 - 12:24

Setting up the Tutorial Environment on Hera (NOAA)

Setting up the Tutorial Environment on Hera (NOAA)

Setting up the Tutorial Environment on Hera (NOAA)

The following instructions should be run if configuring the shell environment to run the METplus Tutorial on Hera (NOAA). If you are running on your own computer or an NCAR machine that has been set up to run the tutorial, please go back and click the appropriate link for those instructions.

Create a Working Directory

Create a directory called 'METplus-5.0.0_Tutorial' to hold all of the files you will create during the tutorial. This can be any directory that you have write permission.

In the following instructions, change "/path/to" to the directory you chose: EDIT AFTER COPYING and BEFORE HITTING RETURN!
Change directory to the location where you want to put your tutorial files.
cd /path/to
Create a directory called METplus-5.0.0_Tutorial. This directory will contain all of your tutorial work, including configuration files, output data, and any other notes you'd like to keep.
mkdir METplus-5.0.0_Tutorial
Change directory into the tutorial directory.
cd METplus-5.0.0_Tutorial

Create Directories for Configuration Files and Output Data

mkdir user_config
mkdir output

Obtain the Tutorial Setup Script

Copy the tutorial setup shell script to configure your runtime environment for use with METplus. You will also copy over a METplus .conf file that you will use to run:
cp /scratch1/BMC/dtc/METplus/METplus-5.0.0_Tutorial_Files/METplus-5.0.0_TutorialSetup.hera.`basename $SHELL` ./METplus-5.0.0_TutorialSetup.sh
cp /scratch1/BMC/dtc/METplus/METplus-5.0.0_Tutorial_Files/tutorial-5.0.0.conf ./tutorial.conf

 

Next navigate to the Verify Environment is Set Correctly page.
mccabe Wed, 06/30/2021 - 12:22

Setting up the Tutorial Environment (bash)

Setting up the Tutorial Environment (bash)

Setting up the Tutorial Environment (bash)

The following instructions should be run if configuring the bash environment to run the METplus Tutorial on your own computer. If you are running on an NCAR or NOAA machine that has been set up to run the tutorial, please go back and click the appropriate link for those instructions.

Create a Working Directory

Create a directory called 'METplus-5.0.0_Tutorial' to hold all of the files you will create during the tutorial. This can be any directory that you have write permission.

In the following instructions, change "/path/to" to the directory you chose: EDIT AFTER COPYING and BEFORE HITTING RETURN!
Change directory to the location where you want to put your tutorial files.
cd /path/to
Create a directory called METplus-5.0.0_Tutorial.
This directory will contain all of your tutorial work, including configuration files, output data, and any other notes you'd like to keep.
mkdir METplus-5.0.0_Tutorial
Change directory into the tutorial directory.
cd METplus-5.0.0_Tutorial

Create Directories for Configuration Files and Output Data

mkdir user_config
mkdir output

Obtain the Tutorial Setup Script

Copy the tutorial setup shell script to configure your runtime environment for use with METplus. You will also copy over a METplus .conf file that you will use to run:
wget https://dtcenter.org/sites/default/files/community-code/metplus/tutorial-data/METplus-5.0.0_TutorialSetup.bash.sh.txt -O ./METplus-5.0.0_TutorialSetup.sh
wget https://dtcenter.org/sites/default/files/community-code/metplus/tutorial-data/tutorial-5.0.0.conf -O ./tutorial.conf

Configure the Tutorial Setup Script

Open the tutorial setup script with the editor of your choice and modify the values for METPLUS_BUILD_BASE, MET_BUILD_BASE, and METPLUS_DATA.
vi METplus-5.0.0_TutorialSetup.sh
METPLUS_BUILD_BASE is the full path to the METplus installation (/path/to/METplus-X.Y)
MET_BUILD_BASE is the full path to the MET installation (/path/to/met-X.Y)
METPLUS_DATA is the location of the sample test data directory

 

Next navigate to the Verify Environment is Set Correctly page.
mccabe Wed, 06/30/2021 - 12:06

Setting up the Tutorial Environment (csh)

Setting up the Tutorial Environment (csh)

Setting up the Tutorial Environment (csh)

The following instructions should be run if configuring a csh environment to run the METplus Tutorial on your own computer. If you are running on an NCAR or NOAA machine that has been set up to run the tutorial, please go back and click the appropriate link for those instructions.

Create a Working Directory

Create a directory called 'METplus-5.0.0_Tutorial' to hold all of the files you will create during the tutorial. This can be any directory that you have write permission.

In the following instructions, change "/path/to" to the directory you chose: EDIT AFTER COPYING and BEFORE HITTING RETURN!
Change directory to the location where you want to put your tutorial files.
cd /path/to
Create a directory called METplus-5.0.0_Tutorial.
This directory will contain all of your tutorial work, including configuration files, output data, and any other notes you'd like to keep.
mkdir METplus-5.0.0_Tutorial
Change directory into the tutorial directory.
cd METplus-5.0.0_Tutorial

Create Directories for Configuration Files and Output Data

mkdir user_config
mkdir output

Obtain the Tutorial Setup Script

Copy the tutorial setup shell script to configure your runtime environment for use with METplus. You will also copy over a METplus .conf file that you will use to run:
wget https://dtcenter.org/sites/default/files/community-code/metplus/tutorial-data/METplus-5.0.0_TutorialSetup.csh.sh.txt -O ./METplus-5.0.0_TutorialSetup.sh
wget https://dtcenter.org/sites/default/files/community-code/metplus/tutorial-data/tutorial-5.0.0.conf -O ./tutorial.conf

Configure the Tutorial Setup Script

Open the tutorial setup script with the editor of your choice and modify the values for METPLUS_BUILD_BASE, MET_BUILD_BASE, and METPLUS_DATA.
vi METplus-5.0.0_TutorialSetup.sh
METPLUS_BUILD_BASE is the full path to the METplus installation (/path/to/METplus-X.Y)
MET_BUILD_BASE is the full path to the MET installation (/path/to/met-X.Y)
METPLUS_DATA is the location of the sample test data directory

 

Next navigate to the Verify Environment is Set Correctly page.
mccabe Wed, 06/30/2021 - 12:21

Verify Environment is Set Correctly

Verify Environment is Set Correctly

Verify Environment is Set Correctly

Run the Tutorial Setup Script

Navigate to your METplus tutorial directory and source the environment file to apply the settings to the current shell. Each time you log in, you will have to source this file again.
In the following instructions, change "/path/to" to the path to your tutorial directory. EDIT AFTER COPYING and BEFORE HITTING RETURN!
cd /path/to/METplus-5.0.0_Tutorial
source METplus-5.0.0_TutorialSetup.sh

The tutorial setup script sets the paths for METPLUS_TUTORIAL_DIR, METPLUS_BUILD_BASE, MET_BUILD_BASE, and METPLUS_DATA. It also appends the $PATH environment variable to include the directory where the METplus scripts are located. If necessary, it may also load modules needed for the METplus software to run correctly.

Check Path

Make sure that all of the environment variables are set to the appropriate values and that the path is set up to locate the METplus components.

Run the 'which run_metplus.py' command.
If you did everything correctly, the full path displayed should be the script in the shared location, ${METPLUS_BUILD_BASE}:
which run_metplus.py
Run the 'point_stat' command without any arguments.
This should display the usage information for the application.
point_stat

You should see the usage statement for Point-Stat. The version number listed should correspond to the version listed in MET_BUILD_BASE. If it does not, you will need to either reload the met module, or add ${MET_BUILD_BASE}/bin to your PATH.

Check that the environment variables required to run the tutorial instructions are set correctly.

$METPLUS_TUTORIAL_DIR

The directory you created to store all of your tutorial files

echo ${METPLUS_TUTORIAL_DIR}

Example value:

/home/metplus_user/METplus_Tutorial
ls ${METPLUS_TUTORIAL_DIR} -1

Example contents:

METplus-5.0.0_TutorialSetup.sh
output/
tutorial.conf
user_config/

$MET_BUILD_BASE

The directory where MET is installed

echo ${MET_BUILD_BASE}

Example value:

/home/metplus_user/met-11.0.1
ls ${MET_BUILD_BASE} -1

Example contents

bin
share
Note: there may be more files/directories than shown here

Check contents of MET bin directory

ls ${MET_BUILD_BASE}/bin -1

Example contents:

ascii2nc
ensemble_stat
gen_ens_prod
gen_vx_mask
gis_dump_dbf
gis_dump_shp
gis_dump_shx
grid_diag
grid_stat
gsid2mpr
gsidens2orank
ioda2nc
lidar2nc
madis2nc
mode
mode_analysis
modis_regrid
mtd
pb2nc
pcp_combine
plot_data_plane
plot_mode_field
plot_point_obs
point2grid
point_stat
regrid_data_plane
rmw_analysis
series_analysis
shift_data_plane
stat_analysis
tc_dland
tc_gen
tc_pairs
tc_rmw
tc_stat
wavelet_stat
wwmca_plot
wwmca_regrid

$METPLUS_BUILD_BASE

The directory where METplus is installed

echo ${METPLUS_BUILD_BASE}

Example value:

/home/metplus_user/METplus-5.0.1
ls ${METPLUS_BUILD_BASE} -1

Example contents:

build_components
docs
environment.yml
internal
manage_externals
metplus
parm
produtil
pyproject.toml
README.md
requirements.txt
setup.py
ush

$METPLUS_DATA

The directory containing sample input data to use for the tutorial

echo ${METPLUS_DATA}

Example value:

/d1/metplus_user/METplus_Data
ls ${METPLUS_DATA} -1

Example contents:

met_reformat
met_test
model_applications
mccabe Wed, 06/30/2021 - 13:59

METplus Overview

METplus Overview

METplus Overview

The following content will discuss and demonstrate some of the basic concepts of the METplus wrappers. This will include discussion of the repository structure and configuration files, as well as how to run a simple example and some associated settings that users will change the most frequently as they continue to work with the METplus system.

As you progress through later tutorial sessions, additional examples of the METplus wrappers will be provided that focus more on their specific MET tool configurable settings and use case based examples.

Proceed to the next page to start with a overview of the METplus directory layout.

jopatz Wed, 01/25/2023 - 15:24

METplus: Directories and Configuration Files - Overview

METplus: Directories and Configuration Files - Overview

METplus directory structure

A brief description and overview of the METplus/ directory structure can be found in the METplus User's Guide section called METplus Wrappers Directory Structure. The files/directories in ${METPLUS_BUILD_BASE} should match this list.

ls ${METPLUS_BUILD_BASE}

METplus default configuration file

Look inside the directory ${METPLUS_BUILD_BASE}/parm

ls ${METPLUS_BUILD_BASE}/parm

Look at the METplus default configuration file:

ls ${METPLUS_BUILD_BASE}/parm/metplus_config

The METplus default configuration file (defaults.conf) is always read first. Any additional configuration files passed in on the command line are then processed in the order in which they are specified. This allows for each successive conf file the ability to override variables defined in any previously processed conf files. It also allows for defining and setting up conf files from a general (settings used by all use cases, ie. MET install dir) to more specific (Plot type when running track and intensity plotter) structure. The idea is to created a hiearchy of conf files that is easier to maintain, read, and manage. It is important to note, running METplus creates a single configuration file, which can be viewed to understand the result of all the conf file processing.

When METplus is run, the final metplus conf file is generated here:
metplus_runtime.conf:METPLUS_CONF={OUTPUT_BASE}/metplus_final.conf.YYYYMMDDHHmmss

This is a unique final config file for each METplus run. Use this file to see the result of all the conf file processing; this can be very helpful when troubleshooting.

NOTE: The syntax for METplus configuration files MUST include a "[config]" section header with the variable names and values on subsequent lines.

More information about the default configuration variables can be found in the METplus User's Guide section called Default Configuration File.
 

The met_config directory (in ${METPLUS_BUILD_BASE}/parm) contains "wrapped" MET configuration files that are used by calls to the MET applications via the METplus wrappers. The wrappers set environment variables that control settings in the wrapped MET configuration files through these environment variables. See the METplus User's Guide section called How METplus controls MET configuration variables for more information.

METplus Use Cases

The use_cases directory - this is where the use cases you will be running exist. Under the use_cases directory are two directories: met_tool_wrapper and model_applications. The met_tool_wrapper directory contains use cases that run a single METplus wrapper. They are a good starting point to see how the wrapper scripts generate commands that run the MET tools. The model_applications directory contains more complex use cases that often run multiple wrappers and demonstrate real evaluations from users.

MET Tool Wrapper Use Cases

Look at the MET Tool Wrapper Use Cases

ls ${METPLUS_BUILD_BASE}/parm/use_cases/met_tool_wrapper

The met_tool_wrapper use case files are organized into subdirectories by wrapper, e.g. Example or GridStat. They contain METplus configuration files (ending with .conf). If a MET tool that is called by a use case uses a MET configuration file, the file used for the met_tool_wrapper use cases is found in parm/met_config.

  • met_tool_wrapper/Example - directory
  • met_tool_wrapper/Example/Example.conf - use case configuration file
  • met_tool_wrapper/GridStat - directory
  • met_tool_wrapper/GridStat/GridStat.conf - use case configuration file
  • parm/met_config/GridStatConfig_wrapped - MET configuration file used in the GridStat.conf use case

 

Model Application Use Cases

Look at the Model Application use cases

ls ${METPLUS_BUILD_BASE}/parm/use_cases/model_applications

The model_applications use case files are organized in directories by category, e.g. precipitation or data_assimilation. They contain METplus configuration files (ending with .conf). If additional files such as Python Embedding scripts are included with a use case, these files are found in a directory named after the METplus configuration file without the .conf extension.

  • model_applications/data_assimilation - directory
  • model_applications/data_assimilation/StatAnalysis_fcstHAFS_obsPrepBufr_JEDI_IODA_interface.conf - use case configuration file
  • model_applications/data_assimilation/StatAnalysis_fcstHAFS_obsPrepBufr_JEDI_IODA_interface - directory containing supplemental files for the use case
  • model_applications/data_assimilation/StatAnalysis_fcstHAFS_obsPrepBufr_JEDI_IODA_interface/read_ioda_mpr.py - script called by the use case

 

Example Use Case (aka "Hello World" Example - METplus style)

Let's look at the Example use case, Example.conf, under met_tool_wrapper/Example

less ${METPLUS_BUILD_BASE}/parm/use_cases/met_tool_wrapper/Example/Example.conf

In this file are variables, denoted in ALL_CAPS. These can be modified as needed. In the METplus system, unmodified config files remain in the directories under ${MET_BUILD_BASE}, while user-modified files are stored in ${METPLUS_TUTORIAL_DIR}/user_config.

No changes are needed in Example.conf. Close it and continue to the next page.

Unless otherwise indicated, all directories are relative to your ${METPLUS_TUTORIAL_DIR} directory.

admin Mon, 06/24/2019 - 15:59

METplus: User Configuration Settings

METplus: User Configuration Settings

Modify your Tutorial/User conf files

In this section you will modify the configuration files that will be read for each call to METplus.

The paths in this practical session guide assume:

  • You have created a user_config directory in your ${METPLUS_TUTORIAL_DIR} directory
  • You have added the shared METplus ush directory to your PATH (done in the Tutorial Setup script)
  • You are using the shared installation of MET.

If not, then you need to adjust accordingly.

  1. Change to the ${METPLUS_TUTORIAL_DIR} directory.  Try running run_metplus.py. You should see the usage statement output to the screen.
cd ${METPLUS_TUTORIAL_DIR};
run_metplus.py
The METplus python script run_metplus.py can be run from anywhere, but for consistency, we will change to ${METPLUS_TUTORIAL_DIR} so that all the directories including user_config and output are below the working directory.
  1. Now try to pass in the example.conf configuration file found in your parm directory under use_cases/met_tool_wrapper/Examples tutorial.conf
run_metplus.py \
${METPLUS_BUILD_BASE}/parm/use_cases/met_tool_wrapper/Example/Example.conf

You should see output like this:

Running METplus 5.0.1

Starting METplus v5.0.1

12/15 19:53:06.223 metplus (config_metplus.py:102) INFO: Starting METplus configuration setup.
12/15 19:53:06.225 metplus (config_metplus.py:230) INFO: Parsing config file: /var/autofs/mnt/linux-amd64/debian/buster/local/METplus-5.0.1/parm/metplus_config/defaults.conf
12/15 19:53:06.226 metplus (config_metplus.py:230) INFO: Parsing config file: /var/autofs/mnt/linux-amd64/debian/buster/local/METplus-5.0.1/parm/use_cases/met_tool_wrapper/Example/Example.conf
Traceback (most recent call last):
  File "/usr/local/METplus-5.0.1/ush/run_metplus.py", line 128, in <module>
    main()
  File "/usr/local/METplus-5.0.1/ush/run_metplus.py", line 44, in main
    config = pre_run_setup(config_inputs)
  File "/usr/local/METplus-5.0.1/metplus/util/run_util.py", line 23, in pre_run_setup
    config = setup(config_inputs)
  File "/usr/local/METplus-5.0.1/metplus/util/config_metplus.py", line 108, in setup
    config = launch(override_list)
  File "/usr/local/METplus-5.0.1/metplus/util/config_metplus.py", line 256, in launch
    mkdir_p(config.getdir('OUTPUT_BASE'))
  File "/usr/local/METplus-5.0.1/metplus/util/config_metplus.py", line 693, in getdir
    raise ValueError(f"{dir_name} cannot be set to "
ValueError: OUTPUT_BASE cannot be set to or contain '/path/to'

ERROR: run_metplus  failed: OUTPUT_BASE cannot be set to or contain '/path/to'

 

Note it ends with an error message stating that OUTPUT_BASE was not set correctly. You will need to configure the METplus wrappers to be able to run a use case.

The values in defaults.conf are read in first when you run run_metplus.py. The settings in these files can be overridden in the use case conf files and/or a user's custom configuration file.

Some variables in the system conf are set to '/path/to' and must be overridden to run METplus, such as OUTPUT_BASE in defaults.conf.

  1. View the defaults.conf file and notice how OUTPUT_BASE = /path/to . This implies it is REQUIRED to be overridden to a valid path.
less ${METPLUS_BUILD_BASE}/parm/metplus_config/defaults.conf
Note: The default installation of METplus has /path/to values for MET_INSTALL_DIR and INPUT_BASE. The value for MET_INSTALL_DIR is set in the shared METplus configuration when it was installed. This was done because these settings will likely be set to the same values for all users. If METplus was installed on a machine that has sample input data available, the value for INPUT_BASE is often set to that directory as well.
  1. View the tutorial configuration files in your ${METPLUS_TUTORIAL_DIR} directory.
less ${METPLUS_TUTORIAL_DIR}/tutorial.conf

The INPUT_BASE, OUTPUT_BASE, and MET_INSTALL_DIR variables must all be set to run METplus. Since MET_INSTALL_DIR (and possibly INPUT_BASE) should already be set in the default METplus configuration file (completed on install of METplus), only OUTPUT_BASE is required to run. If INPUT_BASE is not set in the tutorial configuration file, it should be set correctly in the defaults configuration file.

Note: A METplus conf file is not a shell script. You CANNOT refer to environment variables as you would in a shell script or command prompt, i.e. ${HOME}. Instead, you must reference the environment variable $HOME as {ENV[HOME]}
You can create additional configuration files to be read by the METplus wrappers to override variables. If a variable is found in multiple configuration files that were passed to METplus, the value used will be the last one read in the sequence of configuration files. See the metplus_final.conf file in the output directory to see what values were actually used for a given METplus run.

We will test out using these configurations on the next page.

admin Mon, 06/24/2019 - 16:00

METplus: How to Run with Example.conf

METplus: How to Run with Example.conf

Running METplus

Running METplus involves invoking the python script run_metplus.py followed by a list of configuration files.

Reminder: The default configuration file (defaults.conf) is always read in and processed first before the configuration files passed in on the command line.

If you have configured METplus correctly and call run_metplus.py without passing in any configuration files, it will generate a usage statement to indicate that other config files are required to perform a useful task. It will generate an error statement if something is amiss.

  1. Review the Example.conf configuration file - which is METplus' version of a "Hello World" example
less ${METPLUS_BUILD_BASE}/parm/use_cases/met_tool_wrapper/Example/Example.conf
  1. Call the run_metplus.py script again, this time passing in the Example.conf configuration file and the tutorial.conf configuration file. You should see logs output to the screen.
run_metplus.py \
${METPLUS_BUILD_BASE}/parm/use_cases/met_tool_wrapper/Example/Example.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf

Note: The environment variable METPLUS_BUILD_BASE determines where to look for paths to use_case files. The METPLUS_TUTORIAL_DIR environment variable determines where to look for user-modified config files.

  1. Check the directory specified by the OUTPUT_BASE configuration variable. You should see that files and sub-directories have been created
ls ${METPLUS_TUTORIAL_DIR}/output
  1. Review the METplus log file to see what was run. Compare the log output to the Example.conf configuration file to see how they correspond to each other. The log file will have today's date in the filename. Since METplus was configured to list today's timestamp in YYYYMMDDHHMMSS format, each run of METplus will generate a separate log file. This is the same behavior as the METplus final configuration file. List all of the log files and view the latest METplus log file:
cat `ls -1 ${METPLUS_TUTORIAL_DIR}/output/logs/metplus.log.*`

You will notice that METplus ran for 5 valid times, processing 4 forecast hours for each valid time. For each run time, it ran twice using two different input templates to find files.

metplus INFO: ****************************************
metplus INFO: * Running METplus
metplus INFO: * at valid time: 201702010000
metplus INFO: ****************************************
metplus.Example INFO: Running ExampleWrapper at valid time 20170201000000
metplus.Example INFO: Input directory is /dir/containing/example/data
metplus.Example INFO: Input template is {init?fmt=%Y%m%d}/file_{init?fmt=%Y%m%d}_{init?fmt=%2H}_F{lead?fmt=%3H}.{custom?fmt=%s}
metplus.Example INFO: Processing custom string: ext
metplus.Example INFO: Processing forecast lead 3 hours initialized at 2017-01-31 21Z and valid at 2017-02-01 00Z
metplus.Example INFO: Looking in input directory for file: 20170131/file_20170131_21_F003.ext
metplus.Example INFO: Processing custom string: nc
metplus.Example INFO: Processing forecast lead 3 hours initialized at 2017-01-31 21Z and valid at 2017-02-01 00Z
metplus.Example INFO: Looking in input directory for file: 20170131/file_20170131_21_F003.nc
metplus.Example INFO: Processing custom string: ext
metplus.Example INFO: Processing forecast lead 6 hours initialized at 2017-01-31 18Z and valid at 2017-02-01 00Z
metplus.Example INFO: Looking in input directory for file: 20170131/file_20170131_18_F006.ext
metplus.Example INFO: Processing custom string: nc
metplus.Example INFO: Processing forecast lead 6 hours initialized at 2017-01-31 18Z and valid at 2017-02-01 00Z
metplus.Example INFO: Looking in input directory for file: 20170131/file_20170131_18_F006.nc
metplus.Example INFO: Processing custom string: ext
metplus.Example INFO: Processing forecast lead 9 hours initialized at 2017-01-31 15Z and valid at 2017-02-01 00Z
metplus.Example INFO: Looking in input directory for file: 20170131/file_20170131_15_F009.ext
metplus.Example INFO: Processing custom string: nc
metplus.Example INFO: Processing forecast lead 9 hours initialized at 2017-01-31 15Z and valid at 2017-02-01 00Z
metplus.Example INFO: Looking in input directory for file: 20170131/file_20170131_15_F009.nc
metplus.Example INFO: Processing custom string: ext
metplus.Example INFO: Processing forecast lead 12 hours initialized at 2017-01-31 12Z and valid at 2017-02-01 00Z
metplus.Example INFO: Looking in input directory for file: 20170131/file_20170131_12_F012.ext
metplus.Example INFO: Processing custom string: nc
metplus.Example INFO: Processing forecast lead 12 hours initialized at 2017-01-31 12Z and valid at 2017-02-01 00Z
metplus.Example INFO: Looking in input directory for file: 20170131/file_20170131_12_F012.nc
metplus INFO: ****************************************
metplus INFO: * Running METplus
metplus INFO: * at valid time: 201702010600
metplus INFO: ****************************************
...
  1. Now run METplus passing in the Example.conf and tutorial.conf files from the previous run AND an explicit override of the OUTPUT_BASE variable.
run_metplus.py \
${METPLUS_BUILD_BASE}/parm/use_cases/met_tool_wrapper/Example/Example.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf \
config.OUTPUT_BASE=${METPLUS_TUTORIAL_DIR}/output/changed
  1. Check the directory that corresponds to the config.OUTPUT_BASE value that you set in the command line override. You should see that files and sub-directories have been created in the new location. Note that you can reference an environment variable when setting values on the command line.
ls ${METPLUS_TUTORIAL_DIR}/output/changed

Remember: Additional conf files and config variable overrides are processed after the default METplus config file (defaults.conf). OUTPUT_BASE was set in tutorial.conf and then overridden with config.OUTPUT_BASE.
Order matters, since each successive conf file and/or explicit variable override will take precedence over any value set for variables defined previously.

Note: The processing order allows for structuring your conf files to contain system/user configurations (settings used for every run) and use case specific configurations (settings only used for a given use case).

admin Mon, 06/24/2019 - 16:04

Modifying Timing Control in Example.conf

Modifying Timing Control in Example.conf

Timing Control in METplus

METplus configuration variables that control timing information are described in the Timing Control section of the System Configuration chapter in the METplus User's Guide. The Example wrapper is a good tool to help understand how these settings control what is run by the METplus wrappers.

  1. Copy the Example.conf configuration file in your user_config directory, renaming it Example_timing.conf
cp ${METPLUS_BUILD_BASE}/parm/use_cases/met_tool_wrapper/Example/Example.conf \
${METPLUS_TUTORIAL_DIR}/user_config/Example_timing.conf
  1. Open the new Example_timing.conf file with an editor.
vi ${METPLUS_TUTORIAL_DIR}/user_config/Example_timing.conf
  1. Make the following changes:

3a. Change VALID_BEG and VALID_END:

VALID_BEG = 2017020100
VALID_END = 2017020200

    to:

VALID_BEG = 2022011809
VALID_END = 2022011809

3b. Change LEAD_SEQ:

LEAD_SEQ = 3H , 6H, 9H, 12H

    to:

LEAD_SEQ = 3H

3c. Change EXAMPLE_CUSTOM_LOOP_LIST:

EXAMPLE_CUSTOM_LOOP_LIST = ext, nc

    to:

EXAMPLE_CUSTOM_LOOP_LIST = ext
Note that the valid begin and end time are now set to the same time (January 18, 2022 at 9Z) and there is only 1 forecast lead time (3 hours).

Also note that "ext" stands for extension and "nc" is the typical extension for a netCDF file. The initial example demonstrates how you can have METplus loop over two types of files but the removal of "nc" results in METplus only looking for files that end with "ext".

  1. Call the run_metplus.py script again, this time passing in the Example_timing.conf configuration file and the tutorial.conf configuration file. You should see logs output to the screen.
run_metplus.py \
${METPLUS_TUTORIAL_DIR}/user_config/Example_timing.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf
  1. Review the screen output and/or log file output.
Notice that only a single time was run and the initialization time was computed based on the valid time and forecast lead.
INFO: Processing forecast lead 3 hours initialized at 2022-01-18 06Z and valid at 2022-01-18 09Z
  1. Open Example_timing.conf again to increase the frequency of output by making the following changes:

6a. Change VALID_END:

VALID_END = 2022011809

    to:

VALID_END = 2022011815
Notice that two valid times should now be run (January 18, 2022 at 9Z and January 18, 2022 at 15Z).
  1. Call the run_metplus.py script again and see how the output has changed.
run_metplus.py \
${METPLUS_TUTORIAL_DIR}/user_config/Example_timing.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf
Notice that now 2 valid times were run and the initialization time was computed based on each valid time and forecast lead.
INFO: Processing forecast lead 3 hours initialized at 2022-01-18 06Z and valid at 2022-01-18 09Z
INFO: Processing forecast lead 3 hours initialized at 2022-01-18 12Z and valid at 2022-01-18 15Z
  1. Change the timing settings to loop by initialization (or retrospective) time. Open Example_timing.conf again and make the following changes:

8a. Change LOOP_BY:

LOOP_BY = VALID

    to:

LOOP_BY = INIT

8b. Change VALID_TIME_FMT to INIT_TIME_FMT:

VALID_TIME_FMT = %Y%m%d%H

    to:

INIT_TIME_FMT = %Y%m%d%H

8c. Change VALID_BEG to INIT_BEG:

VALID_BEG = 2022011809

    to:

INIT_BEG = 2022011809

8d. Change VALID_END to INIT_END:

VALID_END = 2022011815

    to:

INIT_END = 2022011815

8e. Change VALID_INCREMENT to INIT_INCREMENT:

VALID_INCREMENT = 3H

    to:

INIT_INCREMENT = 3H
  1. Call the run_metplus.py script once again and see how the output has changed.
run_metplus.py \
${METPLUS_TUTORIAL_DIR}/user_config/Example_timing.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf
Notice that now 2 initialization times were run and the valid time was computed based on each init time and forecast lead.
INFO: Processing forecast lead 3 hours initialized at 2022-01-18 09Z and valid at 2022-01-18 12Z
INFO: Processing forecast lead 3 hours initialized at 2022-01-18 15Z and valid at 2022-01-18 18Z
  1. Change the value for LOOP_BY from INIT to its nickname RETRO. Open Example_timing.conf again and make the following changes:

10a. Change LOOP_BY:

LOOP_BY = INIT

    to:

LOOP_BY = RETRO
  1. Call the run_metplus.py script another time.
run_metplus.py \
${METPLUS_TUTORIAL_DIR}/user_config/Example_timing.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf
Notice that the output did not change. Note that LOOP_BY can also be set to RETRO instead of INIT if that term is preferred. However, the other timing variables must start with INIT_ and not RETRO_. The same applies for using REALTIME instead of VALID.
lisag Wed, 01/12/2022 - 13:04

Modifying Filename Template in Example.conf

Modifying Filename Template in Example.conf

Filename Template Settings in METplus

METplus configuration variables that control filename templates are described in the Directory and Filename Template Info section of the System Configuration chapter in the METplus User's Guide. The Example wrapper is a good tool to help understand how these settings control what is run by the METplus wrappers.

  1. List the sample available in the ${METPLUS_DATA} directory
ls ${METPLUS_DATA}/met_test/data/sample_fcst/2007033000/

The output should list:

nam.t00z.awip1236.tm00.20070330.grb
Think about what is constant and what may change routinely:

nam.t00z.awip1236.tm00.20070330.grb

This translates into nam.t{init?fmt=%2H}z.awip1236.tm00.{init?fmt=%Y%m%d}.grib,  We will use this in this next exercise.

  1. Copy the Example_timing.conf configuration file in your user_config directory, renaming it Example_filename.conf
cp ${METPLUS_TUTORIAL_DIR}/user_config/Example_timing.conf \
${METPLUS_TUTORIAL_DIR}/user_config/Example_filename.conf
  1. Open the new Example_filename.conf file with an editor.
vi ${METPLUS_TUTORIAL_DIR}/user_config/Example_filename.conf
  1. Make the following changes:

4a. Change EXAMPLE_INPUT_DIR:

EXAMPLE_INPUT_DIR = /dir/containing/example/data

    to:

EXAMPLE_INPUT_DIR = {INPUT_BASE}/met_test/data/sample_fcst

4b. Change EXAMPLE_INPUT_TEMPLATE:

EXAMPLE_INPUT_TEMPLATE = {init?fmt=%Y%m%d}/file_{init?fmt=%Y%m%d}_{init?fmt=%2H}_F{lead?fmt=%3H}.{custom?fmt=%s}

    to:

EXAMPLE_INPUT_TEMPLATE = nam.t{init?fmt=%2H}z.awip1236.tm00.{init?fmt=%Y%m%d}.grib
  1. Call the run_metplus.py script again, this time passing in the Example_timing.conf configuration file and the tutorial.conf configuration file. You should see logs output to the screen.
run_metplus.py \
${METPLUS_TUTORIAL_DIR}/user_config/Example_filename.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf
  1. Review the screen output and/or log file output.
INFO: Looking in input directory for file: nam.t09z.awip1236.tm00.20220118.grib
...
INFO: Looking in input directory for file: nam.t15z.awip1236.tm00.20220118.grib
Note that none of the files listed are the one listed in Step 1 (nam.t00z.awip1236.tm00.20070330.grb). You will need to make additional modifications to Example.conf to direct METplus to look for the correct file.
  1. Open the new Example_filename.conf file with an editor.
vi ${METPLUS_TUTORIAL_DIR}/user_config/Example_filename.conf
  1. Make the following changes:

8a. Change INIT_BEG and INIT_END:

INIT_BEG = 2022011809
INIT_END = 2022011815

    to:

INIT_BEG = 2007033000
INIT_END = 2007033000
  1. Call the run_metplus.py script again, passing in the modified Example_timing.conf configuration file and the tutorial.conf configuration file. You should see logs output to the screen.
run_metplus.py \
${METPLUS_TUTORIAL_DIR}/user_config/Example_filename.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf
  1. Review the screen output and/or log file output.  It appears it is now looking for the right file.
INFO: Looking in input directory for file: nam.t00z.awip1236.tm00.20070330.grib
lisag Wed, 01/12/2022 - 13:14

Modifying Log Timestamp using Example.conf

Modifying Log Timestamp using Example.conf

The METplus configuration variable named LOG_TIMESTAMP_TEMPLATE controls to timestamp that is included in the log file names. The default value of LOG_TIMESTAMP_TEMPLATE is %Y%m%d%H%M%S, which will include the year, month, day, hour, minute, and second when the run_metplus.py command was executed. This will create a new log file each time run_metplus.py is called from the command line.

Starting in METplus v5.0.0, the log timestamp is also included by default in the final configuration file that is generated by the METplus wrappers. This helps with debugging because the final config file and its corresponding log file(s) are more easily identified. The path to the final config file is set by METPLUS_CONF.

The values of these settings can be changed by including them in a METplus configuration file that is passed into run_metplus.py. They can also be set directly in the run_metplus.py command using the syntax config.VARIABLE_NAME=VALUE. The following examples will use the latter.

Changing the Log Timestamp Template

In this example we will change the log timestamp template to only include the year, month, and day of the run. This will create a single log file each day. Each call to run_metplus.py will add its log output to the daily file.

  1. Run the Example_timing use case from the previous exercise and override the log timestamp template to only include year, month, and day.
run_metplus.py \
${METPLUS_TUTORIAL_DIR}/user_config/Example_timing.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf \
config.LOG_TIMESTAMP_TEMPLATE=%Y%m%d
  1. List the contents of the log directory and notice that there is now a log file that matches the format metplus.log.YYYYMMDD
ls -1 ${METPLUS_TUTORIAL_DIR}/output/logs/
  1. Run the Example_filename use case from the previous exercise and again override the log timestamp template to only include YYYYMMDD.
run_metplus.py \
${METPLUS_TUTORIAL_DIR}/user_config/Example_filename.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf \
config.LOG_TIMESTAMP_TEMPLATE=%Y%m%d
  1. Review the log file and notice that it contains the log output from both runs.
less /d1/personal/mccabe/out2/logs/metplus.log.`date +%Y%m%d`
  1. Search for the word "Running" to see each command.
grep Running /d1/personal/mccabe/out2/logs/metplus.log.`date +%Y%m%d`

Adding Run ID in Log Timestamp Template

The METplus configuration variable RUN_ID was added in METplus v5.0.0. This variable contains an 8 character string that is automatically generated by and is unique to each call to run_metplus.py. This variable can be referenced in other METplus configuration variables. This can be useful in a variety of ways. For example, it can be used to distinguish log files for METplus runs that may have started within the same second.

  1. Run the Example_timing use case and override the log timestamp template to include the RUN_ID.
run_metplus.py \
${METPLUS_TUTORIAL_DIR}/user_config/Example_timing.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf \
config.LOG_TIMESTAMP_TEMPLATE=%Y%m%d%H%M%S.{RUN_ID}
  1. List the contents of the log directory and notice that there is now a log file that matches the format metplus.log.YYYYMMDDHHMMSS.XXXXXXXX where XXXXXXXX is a random set of characters.
ls -1 ${METPLUS_TUTORIAL_DIR}/output/logs/
mccabe Tue, 01/24/2023 - 11:15