METplus directory structure

A brief description and overview of the METplus/ directory structure can be found in the METplus User's Guide section called METplus Wrappers Directory Structure. The files/directories in ${METPLUS_BUILD_BASE} should match this list.

METplus default configuration file

Look inside the directory ${METPLUS_BUILD_BASE}/parm

Look at the METplus default configuration file:

The METplus default configuration file (defaults.conf) is always read first. Any additional configuration files passed in on the command line are then processed in the order in which they are specified. This allows for each successive conf file the ability to override variables defined in any previously processed conf files. It also allows for defining and setting up conf files from a general (settings used by all use cases, ie. MET install dir) to more specific (Plot type when running track and intensity plotter) structure. The idea is to created a hiearchy of conf files that is easier to maintain, read, and manage. It is important to note, running METplus creates a single configuration file, which can be viewed to understand the result of all the conf file processing.

When METplus is run, the final metplus conf file is generated here:
metplus_runtime.conf:METPLUS_CONF={OUTPUT_BASE}/metplus_final.conf.YYYYMMDDHHmmss

This is a unique final config file for each METplus run. Use this file to see the result of all the conf file processing; this can be very helpful when troubleshooting.

NOTE: The syntax for METplus configuration files MUST include a "[config]" section header with the variable names and values on subsequent lines.

More information about the default configuration variables can be found in the METplus User's Guide section called Default Configuration File.
 

The met_config directory (in ${METPLUS_BUILD_BASE}/parm) contains "wrapped" MET configuration files that are used by calls to the MET applications via the METplus wrappers. The wrappers set environment variables that control settings in the wrapped MET configuration files through these environment variables. See the METplus User's Guide section called How METplus controls MET configuration variables for more information.

METplus Use Cases

The use_cases directory - this is where the use cases you will be running exist. Under the use_cases directory are two directories: met_tool_wrapper and model_applications. The met_tool_wrapper directory contains use cases that run a single METplus wrapper. They are a good starting point to see how the wrapper scripts generate commands that run the MET tools. The model_applications directory contains more complex use cases that often run multiple wrappers and demonstrate real evaluations from users.

MET Tool Wrapper Use Cases

Look at the MET Tool Wrapper Use Cases

The met_tool_wrapper use case files are organized into subdirectories by wrapper, e.g. Example or GridStat. They contain METplus configuration files (ending with .conf). If a MET tool that is called by a use case uses a MET configuration file, the file used for the met_tool_wrapper use cases is found in parm/met_config.

• met_tool_wrapper/Example - directory
• met_tool_wrapper/Example/Example.conf - use case configuration file
• met_tool_wrapper/GridStat - directory
• met_tool_wrapper/GridStat/GridStat.conf - use case configuration file
• parm/met_config/GridStatConfig_wrapped - MET configuration file used in the GridStat.conf use case

Model Application Use Cases

Look at the Model Application use cases

The model_applications use case files are organized in directories by category, e.g. precipitation or data_assimilation. They contain METplus configuration files (ending with .conf). If additional files such as Python Embedding scripts are included with a use case, these files are found in a directory named after the METplus configuration file without the .conf extension.

• model_applications/data_assimilation - directory
• model_applications/data_assimilation/StatAnalysis_fcstHAFS_obsPrepBufr_JEDI_IODA_interface.conf - use case configuration file
• model_applications/data_assimilation/StatAnalysis_fcstHAFS_obsPrepBufr_JEDI_IODA_interface - directory containing supplemental files for the use case
• model_applications/data_assimilation/StatAnalysis_fcstHAFS_obsPrepBufr_JEDI_IODA_interface/read_ioda_mpr.py - script called by the use case

Example Use Case (aka "Hello World" Example - METplus style)

Let's look at the Example use case, Example.conf, under met_tool_wrapper/Example

In this file are variables, denoted in ALL_CAPS. These can be modified as needed. In the METplus system, unmodified config files remain in the directories under ${MET_BUILD_BASE}, while user-modified files are stored in ${METPLUS_TUTORIAL_DIR}/user_config.

No changes are needed in Example.conf. Close it and continue to the next page.

Unless otherwise indicated, all directories are relative to your ${METPLUS_TUTORIAL_DIR} directory.

Modify your Tutorial/User conf files

In this section you will modify the configuration files that will be read for each call to METplus.

The paths in this practical session guide assume:

• You have created a user_config directory in your ${METPLUS_TUTORIAL_DIR} directory
• You have added the shared METplus ush directory to your PATH (done in the Tutorial Setup script)
• You are using the shared installation of MET.

If not, then you need to adjust accordingly.

1. Change to the ${METPLUS_TUTORIAL_DIR} directory.  Try running run_metplus.py. You should see the usage statement output to the screen.

2. Now try to pass in the example.conf configuration file found in your parm directory under use_cases/met_tool_wrapper/Examples tutorial.conf

You should see output like this:

Note it ends with an error message stating that OUTPUT_BASE was not set correctly. You will need to configure the METplus wrappers to be able to run a use case.

Some variables in the system conf are set to '/path/to' and must be overridden to run METplus, such as OUTPUT_BASE in defaults.conf.

3. View the defaults.conf file and notice how OUTPUT_BASE = /path/to . This implies it is REQUIRED to be overridden to a valid path.

4. View the tutorial configuration files in your ${METPLUS_TUTORIAL_DIR} directory.

The INPUT_BASE, OUTPUT_BASE, and MET_INSTALL_DIR variables must all be set to run METplus. Since MET_INSTALL_DIR (and possibly INPUT_BASE) should already be set in the default METplus configuration file (completed on install of METplus), only OUTPUT_BASE is required to run. If INPUT_BASE is not set in the tutorial configuration file, it should be set correctly in the defaults configuration file.

We will test out using these configurations on the next page.

Running METplus

Running METplus involves invoking the python script run_metplus.py followed by a list of configuration files.

Reminder: The default configuration file (defaults.conf) is always read in and processed first before the configuration files passed in on the command line.

If you have configured METplus correctly and call run_metplus.py without passing in any configuration files, it will generate a usage statement to indicate that other config files are required to perform a useful task. It will generate an error statement if something is amiss.

1. Review the Example.conf configuration file - which is METplus' version of a "Hello World" example

2. Call the run_metplus.py script again, this time passing in the Example.conf configuration file and the tutorial.conf configuration file. You should see logs output to the screen.

Note: The environment variable METPLUS_BUILD_BASE determines where to look for paths to use_case files. The METPLUS_TUTORIAL_DIR environment variable determines where to look for user-modified config files.

3. Check the directory specified by the OUTPUT_BASE configuration variable. You should see that files and sub-directories have been created

4. Review the METplus log file to see what was run. Compare the log output to the Example.conf configuration file to see how they correspond to each other. The log file will have today's date in the filename. Since METplus was configured to list today's timestamp in YYYYMMDDHHMMSS format, each run of METplus will generate a separate log file. This is the same behavior as the METplus final configuration file. List all of the log files and view the latest METplus log file:

You will notice that METplus ran for 5 valid times, processing 4 forecast hours for each valid time. For each run time, it ran twice using two different input templates to find files.

5. Now run METplus passing in the Example.conf and tutorial.conf files from the previous run AND an explicit override of the OUTPUT_BASE variable.

6. Check the directory that corresponds to the config.OUTPUT_BASE value that you set in the command line override. You should see that files and sub-directories have been created in the new location. Note that you can reference an environment variable when setting values on the command line.

Remember: Additional conf files and config variable overrides are processed after the default METplus config file (defaults.conf). OUTPUT_BASE was set in tutorial.conf and then overridden with config.OUTPUT_BASE.
Order matters, since each successive conf file and/or explicit variable override will take precedence over any value set for variables defined previously.

Note: The processing order allows for structuring your conf files to contain system/user configurations (settings used for every run) and use case specific configurations (settings only used for a given use case).

Timing Control in METplus

METplus configuration variables that control timing information are described in the Timing Control section of the System Configuration chapter in the METplus User's Guide. The Example wrapper is a good tool to help understand how these settings control what is run by the METplus wrappers.

1. Copy the Example.conf configuration file in your user_config directory, renaming it Example_timing.conf

2. Open the new Example_timing.conf file with an editor.

3. Make the following changes:

3a. Change VALID_BEG and VALID_END:

    to:

3b. Change LEAD_SEQ:

    to:

3c. Change EXAMPLE_CUSTOM_LOOP_LIST:

    to:

4. Call the run_metplus.py script again, this time passing in the Example_timing.conf configuration file and the tutorial.conf configuration file. You should see logs output to the screen.

5. Review the screen output and/or log file output.

6. Open Example_timing.conf again to increase the frequency of output by making the following changes:

6a. Change VALID_END:

    to:

7. Call the run_metplus.py script again and see how the output has changed.

8. Change the timing settings to loop by initialization (or retrospective) time. Open Example_timing.conf again and make the following changes:

8a. Change LOOP_BY:

    to:

8b. Change VALID_TIME_FMT to INIT_TIME_FMT:

    to:

8c. Change VALID_BEG to INIT_BEG:

    to:

8d. Change VALID_END to INIT_END:

    to:

8e. Change VALID_INCREMENT to INIT_INCREMENT:

    to:

9. Call the run_metplus.py script once again and see how the output has changed.

10. Change the value for LOOP_BY from INIT to its nickname RETRO. Open Example_timing.conf again and make the following changes:

10a. Change LOOP_BY:

    to:

11. Call the run_metplus.py script another time.

Filename Template Settings in METplus

METplus configuration variables that control filename templates are described in the Directory and Filename Template Info section of the System Configuration chapter in the METplus User's Guide. The Example wrapper is a good tool to help understand how these settings control what is run by the METplus wrappers.

1. List the sample available in the ${METPLUS_DATA} directory

The output should list:

2. Copy the Example_timing.conf configuration file in your user_config directory, renaming it Example_filename.conf

3. Open the new Example_filename.conf file with an editor.

4. Make the following changes:

4a. Change EXAMPLE_INPUT_DIR:

    to:

4b. Change EXAMPLE_INPUT_TEMPLATE:

    to:

5. Call the run_metplus.py script again, this time passing in the Example_timing.conf configuration file and the tutorial.conf configuration file. You should see logs output to the screen.

6. Review the screen output and/or log file output.

7. Open the new Example_filename.conf file with an editor.

8. Make the following changes:

8a. Change INIT_BEG and INIT_END:

    to:

9. Call the run_metplus.py script again, passing in the modified Example_timing.conf configuration file and the tutorial.conf configuration file. You should see logs output to the screen.

10. Review the screen output and/or log file output.  It appears it is now looking for the right file.

The METplus configuration variable named LOG_TIMESTAMP_TEMPLATE controls to timestamp that is included in the log file names. The default value of LOG_TIMESTAMP_TEMPLATE is %Y%m%d%H%M%S, which will include the year, month, day, hour, minute, and second when the run_metplus.py command was executed. This will create a new log file each time run_metplus.py is called from the command line.

Starting in METplus v5.0.0, the log timestamp is also included by default in the final configuration file that is generated by the METplus wrappers. This helps with debugging because the final config file and its corresponding log file(s) are more easily identified. The path to the final config file is set by METPLUS_CONF.

The values of these settings can be changed by including them in a METplus configuration file that is passed into run_metplus.py. They can also be set directly in the run_metplus.py command using the syntax config.VARIABLE_NAME=VALUE. The following examples will use the latter.

Changing the Log Timestamp Template

In this example we will change the log timestamp template to only include the year, month, and day of the run. This will create a single log file each day. Each call to run_metplus.py will add its log output to the daily file.

1. Run the Example_timing use case from the previous exercise and override the log timestamp template to only include year, month, and day.

2. List the contents of the log directory and notice that there is now a log file that matches the format metplus.log.YYYYMMDD

3. Run the Example_filename use case from the previous exercise and again override the log timestamp template to only include YYYYMMDD.

4. Review the log file and notice that it contains the log output from both runs.

5. Search for the word "Running" to see each command.

Adding Run ID in Log Timestamp Template

The METplus configuration variable RUN_ID was added in METplus v5.0.0. This variable contains an 8 character string that is automatically generated by and is unique to each call to run_metplus.py. This variable can be referenced in other METplus configuration variables. This can be useful in a variety of ways. For example, it can be used to distinguish log files for METplus runs that may have started within the same second.

1. Run the Example_timing use case and override the log timestamp template to include the RUN_ID.

2. List the contents of the log directory and notice that there is now a log file that matches the format metplus.log.YYYYMMDDHHMMSS.XXXXXXXX where XXXXXXXX is a random set of characters.