Prequisites: Software

The following is a full list of software that is required to utilize all the functionality in METplus.
NOTE: We will not by running the cyclone plotter or grid plotting scripts in this tutorial. So the additional libraries mentioned below are not installed.

• Python 2.7.x
• Additional Python libraries: - required by the cyclone plotter and grid plotting scripts.
  • numpy
  • matplotlib - not installed, Is is not required for this tutorial.
  • cartopy - not installed, It is not required for this tutorial.
• MET 8.1
• R - used by tcmpr_plotter_wrapper, Wraps the MET plot_tcmpr.R script
• ncap2 - NCO netCDF Operators
• wgrib2
• ncdump - NetCDF binaries
• convert - Utilitity from ImageMagick software suite

Getting The Software

For this tutorial we'll use the METplus v2.2 release. We will configure METplus to run the met-8.1 software, which has already been installed in a common location and is available through a module file.

Please use the following instructions to setup METplus on your tutorial machine:

• Unpack the release tarball and create a METplus symbolic link (copy and paste the following commands):

Sample Data Input and Location of Output

The Sample Input Data for running this METplus tutorial is provided in a shared location on-disk and does not require copying to your home directory. How to set the path to the sample input data and location of the output when running METplus will be explained later in the tutorial, utilizing METplus configuration files.

Just note that:
When running METplus use cases, all output is placed in your ${HOME}/metplus_tutorial/output directory.
When running MET on the command line, all output will be written to ${HOME}/metplus_tutorial/output/met_output

Your Runtime Environment

For this tutorial, the MET software has been setup as a loadable module file. We need to add the following command to your environment so the MET software is available when you login.

Edit the .usermodlogin file to configure your runtime environment to load the MET software:

Open the following file with the editor of your choice:

Add the following to the LAST LINE of the file:

Save and close that file, and then source it to apply the settings to the current shell. The next time you login, you do not have to source this file again.:

The system PATH environment variable is set to allow the METplus application to be run from any directory.
As a note, there is also an optional JLOGFILE environment variable (Job Log), which indicates where JLOGS will be saved. JLOGS provide an overview of the METplus python modules being run and is intended to be used when running METplus within a workflow management systems on an HPC cluster. If this environment variable is unset, then output will be directed to stdout (your display). JLOGFILE will not be defined during the tutorial, If interested, more information regarding the JLOGFILE can be found in the METplus Users Guide. You will now edit the system PATH.

Edit the .usermodrc file to configure your runtime environment for use with METplus:

open the following file with the editor of your choice:

Add the following to update your system PATH setting:
NOTE: If the PATH variable already exists in your .usermodrc file,
than make certain to ADD the line below after the existing setting.

csh:

bash:

Save and close that file, and then source it to apply the settings to the current shell. The next time you login, you do not have to source this file again.:

METplus directory structure

Brief description and overview of the METplus/ directory structure.

• doc/ - Doxygen files for building the html API documentation.
• doc/METplus_Users_Guide - LyX files and the user guide PDF file.
• internal_tests/ - for engineering tests
• parm/ - where config files live
• README.md - general README
• sorc/ - executables and documentation build system
• ush/ - python scripts

METplus default configuration files

The METplus default configuration files metplus_system.conf, metplus_data.conf, metplus_runtime.conf, and metplus_logging.conf are always read by default and in the order shown. Any additional configuration files passed in on the command line are than processed in the order in which they are specified. This allows for each successive conf file the ability to override variables defined in any previously processed conf files. It also allows for defining and setting up conf files from a general (settings used by all use cases, ie. MET install dir) to more specific (Plot type when running track and intensity plotter) structure. The idea is to created a hiearchy of conf files that is easier to maintain, read, and manage. It is important to note, running METplus creates a single configuration file, which can be viewed to understand the result of all the conf file processing.

TIP: The final metplus conf file is defined here:
metplus_runtime.conf:METPLUS_CONF={OUTPUT_BASE}/metplus_final.conf
Use this file to see the result of all the conf file processing, this can be very helpful when troubleshooting,

NOTE: The syntax for METplus configuration files MUST include a "[section]" header with the variable names and values on subsequent lines.

Unless otherwise indicated, all directories are relative to your ${HOME}/METplus directory.

The parm/metplus_config directory - there are four config files:

1. metplus_system.conf
   • contains "[dir]" and "[exe]" to set directory and executable paths
   • any information specific to host machine
2. metplus_data.conf
   • Sample data or Model input location
   • filename templates and regex for filenames
3. metplus_runtime.conf
   • contains "[config]" section
   • var lists, stat lists, configurations, process list
   • anything else needed at runtime
4. metplus_logging.conf
   • contains "[dir]" section for setting the logcation of log files.
   • contains "[config]" section for setting various logging configuration options.

The parm/use_cases directory - The use cases:
This is where the use cases you will be running exist. The use case filenaming and directory structure is by convention. It is intended to simplify management of common configuration items for a specific use case. Under the use_cases directory is a subdirectory for each use case. Under that is an examples subdirectory and a more general configuration file that contains information that is shared by examples of the use case.

• For example, the track and intensity use case conf directory and file structure.
• track_and_intensity - directory
• track_and_intensity/track_and_intensity.conf - Any common conf file settings shared by track and intensity examples
• track_and_intensity/examples - directory to hold various track and intensity example conf files.
• track_and_intensity/examples/tcmpr_mean_median.conf - specific use case

Within each use case there is a met_config subdirectory that contains MET specific config files for that use case. The parm/use_cases/<use case>/met_config directory - store all your MET config files, e.g.:

• GridStatConfig_anom (for MET grid_stat)
• TCPairsETCConfig (for MET tc_pairs)
• SeriesAnalysisConfig_by_init (for MET series_analysis)
• SeriesAnalysisConfig_by_lead (for MET series_analysis)

Create a METplus custom conf file

In this section you will create a custom configuration file that you will use in each call to METplus.

The paths in this practical assume: You have installed METPlus under your $HOME directory and You are using the shared installation of MET. If not, than you need to adjust your paths accordingly.

Perform the following commands ...

Change to your ${HOME}/METplus/parm directory and copy your custom conf file.

NOTE: By placing your custom configuration file under the parm directory, you do not have to specify the path to mycustom.conf in the command when refering to it. The commands in this tutorial are setup that way, so make sure to indeed create this file under the METplus/parm directory.

NOTE: A METplus conf file is not a shell script.
You CAN NOT refer to environment variables as you would in a shell script, or as in the command above ie. ${HOME}.
Instead, in a conf file, you must define as {ENV[HOME]}

Reminder: Make certain to maintain the KEY = VALUE pairs under their respective current [sections] in the conf file.

Below is the exact content of your mycustom.conf file you just copied to your parm directory. It is instructive to REVIEW the settings to understand what changes were REQUIRED in order to run METplus with the default example use cases from a clean download of the software.

Also Note: [config] is not required if you aren't setting any values in this file for that section, but you can add it in case you want to add options in that section later - which you will be doing.

Overriding conf files

This page is meant to be instructive and to reinforce the concept of default configuration variables and overriding conf variables. This page will override your previous set OUTPUT_BASE configuration variable, for your first use case.

The values in the default metplus_system.conf, metplus_data.conf, metplus_runtime.conf, and metplus_logging.conf configuration files can be overridden in the use case conf files and/or a user's custom configuration file. Recall the general use case layout:

• ${HOME}/METplus/parm/use_cases/<use_case>/<use_case>.conf
• ${HOME}/METplus/parm/use_cases/<use_case>/examples/<some_example>.conf

Some variables in the system conf are set to '/path/to' and must be overridden to run METplus, such as OUTPUT_BASE in metplus_system.conf.

Reminder: When adding variables to be overridden, make sure to place the variable under the appropriate section.
For example, [config], [dir], [exe]. If necessary, refer to the default appropriate parm/metplus_config conf files to determine the [section] that corresponds to the variable you are overriding. The value set will be the last one in the sequence of configuration files. See OUTPUT_BASE/metpus_final.conf to see what values were used for a given METplus run.

Example:
View the ${HOME}/METplus/parm/metplus_config/metplus_system.conf file and notice how OUTPUT_BASE = /path/to . This implies it is REQUIRED to be overridden to a valid path.

If you are NOT using a custom conf file, you will need to set the values in metplus_config. We recommend using a custom conf file to avoid conflicts if using a shared METplus installation. It is also beneficial when upgrading to a new version of METplus, as you will not have to scan the metplus_config directory to find all of the changes you have made.

Running METplus

Running METplus involves invoking the python script, master_metplus.py from any directory followed by a list of configuration files using the -c option for each additional conf file (conf files are specified relative to the METplus/parm directory).

Reminder: The default set of conf files are always read in and processed in the following order;
metplus_system.conf, metplus_data.conf, metplus_runtime.conf, metplus_logging.conf.

Example: Run the default configuration

If you have configured METplus correctly and run master_metplus.py with only your custom conf file, it will generate a usage statement to indicate that other config files are required to perform a useful tasks. It will generate an error statement if something is amiss.

Run master_metplus.py on the command line:

THE EXAMPLES BELOW SHOW THE SYNTAX FOR PASSING IN CONF FILES AND ARE NOT MEANT TO BE RUN.
However, feel free to run the command and become familiar with the errors that are generated when referring to a path and/or conf file that does not exist.

Remember: Additional conf files are processed after the metplus_config files in the order specified on the command line.
Order matters, since each successive conf file will override any variable defined in a previous conf file.

NOTE: The processing order allows for structuring your conf files from general (variables shared-by-all) to specific (variables shared-by-few).

Syntax: Running a use_case configuration

Syntax: Runing a use_case/examples configuration

Syntax: Runing a use_case/examples configuration with your CUSTOM conf.