Prequisites: Software

The following is a full list of software that is required to utilize all the functionality in METplus.
NOTE: We will not by running the cyclone plotter or grid plotting scripts in this tutorial. So the additional libraries mentioned below are not installed.

• Python 2.7.x
• Additional Python libraries: - required by the cyclone plotter and grid plotting scripts.
  • numpy
  • matplotlib - not installed, Is is not required for this tutorial.
  • cartopy - not installed, It is not required for this tutorial.
• MET 8.1 - already installed locally on the tutorial machines.
• R - used by TCMPRPlotter wrapper, Wraps the MET plot_tcmpr.R script
• ncap2 - NCO netCDF Operators
• wgrib2 - only needed for series analysis wrappers (SeriesByLead and SeriesByInit)
• ncdump - NetCDF binaries
• convert - Utilitity from ImageMagick software suite

Prequisites: Environment

Before running these instructions, you will need to ensure that you have a few environment variables set up correctly. If they are not set correctly, these instructions will not work properly.

1. Check that you have METPLUS_TUTORIAL_DIR set correctly:

If you don't see a path in your user directory output to the screen, set this environment variable in your user profile before continuing.

2. Check that you have METPLUS_BUILD_BASE, MET_BUILD_BASE, and METPLUS_DATA set correctly:

If any of these variables are not set, please set them. They will be referenced throughout the tutorial.

3. Check that you have loaded the MET module correctly:

You should see the usage statement for Point-Stat. The version number listed should correspond to the version listed in MET_BUILD_BASE. If it does not, you will need to either reload the met module, or add ${MET_BUILD_BASE}/bin to your PATH.

Getting The Software

For this tutorial we'll use the METplus v2.2 release, which has been installed in a common location. We will configure METplus to run the met-8.1 software, which has also been installed in a common location. We will copy the parameter files from the shared METplus location to your own directory so you can modify them without changing the shared installation settings.

Please use the following instructions to setup METplus on your tutorial machine:

1. Create a directory to store your METplus files (include the version number for traceability):

2. Create a symbolic link so that you can refer to the METplus directory instead of METplus-2.2 (you may need to remove the link if it already exists):

3. List the contents of the metplus_tutorial directory:

You should see something similar to this:

4. Copy the parm directory from the shared METplus location to your home directory:

5. Create directory to store configuration files you will create:

For reference, the following are instructions to download and install METplus on your own machine.

• If you wish, download and unpack the release tarball and create a METplus symbolic link:

DO NOT RUN THESE COMMANDS FOR THE TUTORIAL!

Sample Data Input and Location of Output

The Sample Input Data used to run the examples in this tutorial already exist in a shared location. For reference, here are example instructions to obtain the sample input data.

DO NOT RUN THESE COMMANDS FOR THE TUTORIAL!

This will create a directory named 'grid_to_grid' under '/d1/METplus_Data' that contains all of the sample input data needed to run the grid_to_grid use cases (found in parm/use_cases/grid_to_grid/examples). Other sample data tarballs are available under 'Assets' on the METplus GitHub releases page.

When running METplus wrapper use cases, all output is placed in the ${METPLUS_TUTORIAL_DIR}/output directory.

When running MET on the command line, all output will be written to ${METPLUS_TUTORIAL_DIR}/output/met_output

Your Runtime Environment

For this tutorial, the MET software is available via a module file. This has already been loaded for all users, so the MET software is available when you login.

The PATH environment variable is set to allow the METplus scripts to be run from any directory.

1. Edit the .bash_profile file to configure your runtime environment for use with METplus:

Add the following at the end of the file to set the PATH and METPLUS_PARM_BASE environment variables. Also update LD_LIBRARY_PATH to avoid runtime NetCDF linker errors:

bash:

csh:

Save and close that file, then source it to apply the settings to the current shell. The next time you login, you do not have to source this file again:

If you did everything correctly, running 'which master_metplus.py' should show you the path to the script in the shared location, ${METPLUS_BUILD_BASE}:

METplus directory structure

Brief description and overview of the METplus/ directory structure.

• doc/ - Doxygen files for building the html API documentation.
• doc/METplus_Users_Guide - LyX files and the user guide PDF file.
• internal_tests/ - for engineering tests
• parm/ - where config files live
• README.md - general README
• sorc/ - executables and documentation build system
• ush/ - python scripts

METplus default configuration files

The METplus default configuration files metplus_system.conf, metplus_data.conf, metplus_runtime.conf, and metplus_logging.conf are always read by default and in the order shown. Any additional configuration files passed in on the command line are than processed in the order in which they are specified. This allows for each successive conf file the ability to override variables defined in any previously processed conf files. It also allows for defining and setting up conf files from a general (settings used by all use cases, ie. MET install dir) to more specific (Plot type when running track and intensity plotter) structure. The idea is to created a hiearchy of conf files that is easier to maintain, read, and manage. It is important to note, running METplus creates a single configuration file, which can be viewed to understand the result of all the conf file processing.

The final metplus conf file is defined here:
metplus_runtime.conf:METPLUS_CONF={OUTPUT_BASE}/metplus_final.conf
Use this file to see the result of all the conf file processing, this can be very helpful when troubleshooting,

NOTE: The syntax for METplus configuration files MUST include a "[section]" header with the variable names and values on subsequent lines.

Unless otherwise indicated, all directories are relative to your ${METPLUS_PARM_BASE} directory.

The metplus_config directory - there are four config files:

1. metplus_system.conf
   • contains "[dir]" and "[exe]" to set directory and executable paths
   • any information specific to host machine
2. metplus_data.conf
   • Sample data or Model input location
   • filename templates and regex for filenames
3. metplus_runtime.conf
   • contains "[config]" section
   • var lists, stat lists, configurations, process list
   • anything else needed at runtime
4. metplus_logging.conf
   • contains "[dir]" section for setting the location of log files.
   • contains "[config]" section for setting various logging configuration options.

The use_cases directory - The use cases:
This is where the use cases you will be running exist. The use case filenaming and directory structure is by convention. It is intended to simplify management of common configuration items for a specific use case. Under the use_cases directory is a subdirectory for each use case. Under that is an examples subdirectory and a more general configuration file that contains information that is shared by examples of the use case.

• For example, the track and intensity use case conf directory and file structure.
• track_and_intensity - directory
• track_and_intensity/track_and_intensity.conf - use case configuration file
• track_and_intensity/examples - directory to hold various track and intensity example conf files.
• track_and_intensity/examples/tcmpr_mean_median.conf - specific use case
• track_and_intensity/examples/README - File containing instructions to run the use case examples in this directory.

Within each use case there is a met_config subdirectory that contains MET specific config files for that use case. The use_cases/<use case>/met_config directory - store all your MET config files, e.g.:

• GridStatConfig_anom (for MET grid_stat)
• TCPairsETCConfig (for MET tc_pairs)
• SeriesAnalysisConfig_by_init (for MET series_analysis)
• SeriesAnalysisConfig_by_lead (for MET series_analysis)

Modify your METplus conf files

In this section you will modify the configuration files that will be read for each call to METplus.

The paths in this practical assume:

• You have copied the parm directory from the shared METplus location into your ${METPLUS_TUTORIAL_DIR}/METplus directory
• You have added the shared METplus ush directory to your PATH
• You have set the METPLUS_PARM_BASE environment variable to your copied parm directory
• You are using the shared installation of MET.

If not, then you need to adjust accordingly.

1. Try running master_metplus.py. You should see the usage statement output to the screen.

2. Now try to pass in the example.conf configuration file found in your parm directory under use_cases/wrappers/examples

You should see an error message stating that OUTPUT_BASE was not set correctly. You will need to configure the METplus wrappers to be able to run a use case.

The values in the default metplus_system.conf, metplus_data.conf, metplus_runtime.conf, and metplus_logging.conf configuration files are read in first when you run master_metplus.py. The settings in these files can be overridden in the use case conf files and/or a user's custom configuration file.

Some variables in the system conf are set to '/path/to' and must be overridden to run METplus, such as OUTPUT_BASE in metplus_system.conf.

3. View the metplus_system.conf file and notice how OUTPUT_BASE = /path/to . This implies it is REQUIRED to be overridden to a valid path.

Note: The default installation of METplus has /path/to values for MET_INSTALL_DIR, TMP_DIR, and INPUT_BASE. These values were set in the shared METplus configuration when it was installed. This was done because these settings will likely be set to the same values for all users.

4. Change directory to your ${METPLUS_PARM_BASE} directory and modify the METplus base configuration files using the editor of your choice (vi listed as an example).

If METPLUS_PARM_BASE is set correctly, you should now be in ${METPLUS_TUTORIAL_DIR}/METplus/parm/metplus_config

Note: A METplus conf file is not a shell script.
You CAN NOT refer to environment variables as you would in a shell script or command prompt, i.e. ${HOME}.
Instead, you must reference the environment variable $HOME as {ENV[HOME]}

Reminder: Make certain to maintain the KEY = VALUE pairs under their respective current [sections] in the conf file.

5. Set the [dir] INPUT_BASE variable to the location of the METplus sample data that will be used in the exercises

6. Set the [dir] OUTPUT_BASE variable to the location you will write output data
   Set the [dir] MET_INSTALL_DIR variable to the location where MET is installed (this should already be done when METplus was installed in the shared location)
   Set the [dir] TMP_DIR variable to a location you have write permissions

NOTE: When installing METplus, you will need to set the full path to the non-MET executables in the metplus_system.conf file if they are not found in the user's path. This step was completed when METplus was installed in the shared location.

Creating user conf files

You can create additional configuration files to be read by the METplus wrappers to override variables.

Reminder: When adding variables to be overridden, make sure to place the variable under the appropriate section.
For example, [config], [dir], [exe]. If necessary, refer to the default appropriate parm/metplus_config conf files to determine the [section] that corresponds to the variable you are overriding. The value set will be the last one in the sequence of configuration files. See OUTPUT_BASE/metpus_final.conf to see what values were used for a given METplus run.

1. Create a new configuration file in your user_config directory and override the [dir] OUTPUT_BASE variable to point to a different location.

Copy and paste the following text into the file and save it.

We will test out using these configurations on the next page.

Running METplus

Running METplus involves invoking the python script master_metplus.py followed by a list of configuration files using the -c option for each additional conf file.

Reminder: The default set of conf files are always read in and processed in the following order;
metplus_system.conf, metplus_data.conf, metplus_runtime.conf, metplus_logging.conf.

If you have configured METplus correctly and run master_metplus.py without passing in any configuration files, it will generate a usage statement to indicate that other config files are required to perform a useful task. It will generate an error statement if something is amiss.

1. Review the example.conf configuration file

2. Call the master_metplus.py script, passing in the example.conf configuration file with the -c command line option. You should see logs output to the screen.

Note: If master_metplus.py can't be found, you may need to add it to the path. Optionally, you can specify the full path to the script.

Note: The environment variable METPLUS_PARM_BASE determines where to look for relative paths to configuration files. If the environment variable is not set, the script will look in the parm directory that is associated with the version of master_metplus.py that is being called. Optionally, you can specify the full path to the configuration files.

OPTIONAL:

3. Check the directory specified by the OUTPUT_BASE configuration variable. You should see that files and sub-directories have been created

4. Review the master log file to see what was run. Compare the log output to the example.conf configuration file to see how they correspond to each other. The log file will have today's date in the filename

You will notice that METplus ran for 4 valid times, processing 4 forecast hours for each valid time. For each run, it used the input template to determine which file to look for.

5. Now run METplus passing in the example.conf from the previous run AND your newly created configuration file.

6. Check the directory specified by the OUTPUT_BASE configuration variable that you set in the change_output_base.conf configuration file. You should see that files and sub-directories have been created in the new location.

Remember: Additional conf files are processed after the metplus_config files in the order specified on the command line.
Order matters, since each successive conf file will override any variable defined in a previous conf file.

Note: The processing order allows for structuring your conf files from general (variables shared-by-all) to specific (variables shared-by-few).