MET Tool: MODE

MODE Tool: General

MODE Functionality

MODE, the Method for Object-Based Diagnostic Evaluation, provides an object-based verification for comparing gridded forecasts to gridded observations. MODE may be used in a generalized way to compare any two fields containing data from which objects may be well defined. It has most commonly been applied to precipitation fields and radar reflectivity. The steps performed in MODE consist of:

Define objects in the forecast and observation fields based on user-defined parameters.
Compute attributes for each of those objects: such as area, centroid, axis angle, and intensity.
For each forecast/observation object pair, compute differences between their attributes: such as area ratio, centroid distance, angle difference, and intensity ratio.
Use fuzzy logic to compute a total interest value for each forecast/observation object pair based on user-defined weights.
Based on the computed interest values, match objects across fields and merge objects within the same field.
Write output statistics summarizing the characteristics of the single objects, the pairs of objects, and the matched/merged objects.

MODE may be configured to use a few different sets of logic with which to perform matching and merging. In this tutorial, we'll use the most simple approach, but users are encouraged to read Chapter 14 of the MET User's Guide for a more thorough description of MODE's capabilities.

MODE Usage

View the usage statement for MODE by simply typing the following:

mode

Usage: mode
	fcst_file	Input gridded forecast file containing the field to be verified
	obs_file	Input gridded observation file containing the verifying field
	config_file	MODEConfig file containing the desired configuration settings
	[-config_merge merge_config_file]	Overrides the default fuzzy engine settings for merging within the fcst/obs fields (optional).
	[-outdir path]	Overrides the default output directory (optional).
	[-log file]	Outputs log messages to the specified file
	[-v level]	Level of logging
	[-compress level]	NetCDF compression level

The forecast and observation fields must be on the same grid. You can use copygb to regrid GRIB1 files, wgrib2 to regrid GRIB2 files, or use the automated regridding functionality within the MET config files.

The MODE tool usage statement also has instructions for verifying multivariables. These are the same as the MODE usage, only passing ASCII lists that contain multiple files instead of a single file.

At a minimum, the input gridded fcst_file, the input gridded obs_file, and the configuration config_file must be passed in on the command line.

cindyhg Tue, 06/25/2019 - 07:58

Configure

MODE Tool: Configure

Start by making an output directory for MODE and changing directories:

mkdir -p ${METPLUS_TUTORIAL_DIR}/output/met_output/mode

cd ${METPLUS_TUTORIAL_DIR}/output/met_output/mode

The behavior of MODE is controlled by the contents of the configuration file passed to it on the command line. The default MODE configuration file may be found in the data/config/MODEConfig_default file.

Prior to modifying the configuration file, users are advised to make copies of existing configuration files:

cp ${METPLUS_DATA}/met_test/scripts/config/MODEConfig_APCP_12 MODEConfig_APCP_12

cp ${METPLUS_DATA}/met_test/scripts/config/MODEConfig_APCP_24 MODEConfig_APCP_24

cp ${METPLUS_DATA}/met_test/scripts/config/MODEConfig_RH MODEConfig_RH

We'll be using these three configuration files during this session.

Open up the MODEConfig_APCP_12 file to view it.

vi MODEConfig_APCP_12

The configuration items for MODE are used to specify how the object-based verification approach is to be performed. In MODE, as in the other MET statistics tools, you can compare any two fields. When necessary, the items in the configuration file are specified separately for the forecast and observation fields. In most cases though, users will be comparing the same forecast and observation fields. The configurable items include parameters for the following:

The forecast and observation fields and vertical levels or accumulation intervals to be compared
Options to mask out a portion of or threshold the raw fields
The forecast and observation object definition parameters
Options to filter out objects that don't meet a size or intensity criteria
Flags to control the logic for matching/merging
Weights to be applied for the fuzzy engine matching/merging algorithm
Interest functions to be used for the fuzzy engine matching/merging algorithm
Total interest threshold for matching/merging
Various plotting options

While the MODE configuration file contains many options, beginning users will typically only need to modify a few of them. You may find a complete description of the configurable items in the mode configuration file section of the MET User's Guide. Please take some time to review them.

At the bottom of MODE_Config_APCP_12, change "version" to "11.0"

////////////////////////////////////////////////////////////////////////////////

output_prefix = "";
version = "V11.0";

////////////////////////////////////////////////////////////////////////////////

Close MODEConfig_APCP_12. Also change the version number in MODEConfig_APCP_24 and MODEConfig_RH.

We'll start here using by running the configuration files we copied over, as-is.

cindyhg Tue, 06/25/2019 - 08:01

Run

MODE Tool: Run

Next, run MODE three times on the command line using those three configuration files with the following commands:

mode \

${METPLUS_DATA}/met_test/out/pcp_combine/sample_fcst_12L_2005080712V_12A.nc \

${METPLUS_DATA}/met_test/out/pcp_combine/sample_obs_2005080712V_12A.nc \

MODEConfig_APCP_12 \

-outdir . \

-v 2

mode \

${METPLUS_DATA}/met_test/data/sample_fcst/2005080700/wrfprs_ruc13_24.tm00_G212 \

${METPLUS_DATA}/met_test/out/pcp_combine/sample_obs_2005080800V_24A.nc \

MODEConfig_APCP_24 \

-outdir . \

-v 2

mode \

${METPLUS_DATA}/met_test/data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212 \

${METPLUS_DATA}/met_test/data/sample_fcst/2005080712/wrfprs_ruc13_00.tm00_G212 \

MODEConfig_RH \

-outdir . \

-v 2

If you receive error output during any of the MODE runs above that looks like the following:

ERROR :
ERROR : check_met_version() -> The version number listed in the config file (V8.1) is not compatible with the current version of the code (V11.0.0).
ERROR :

You will need to follow the previous instructions and make sure the version is set to V11.0 in all of the configuration files prior to re-running the commands.

These commands make use of sample data that's distributed with the MET tarball. They run MODE on 12-hour accumulated precipitation, 24-hour accumulated precipitation, and on a field of relative humidity.

cindyhg Tue, 06/25/2019 - 08:02

Output

MODE Tool: Output

The output of MODE typically consists of 4 files: 2 ASCII statistics files, 1 NetCDF object file, and 1 PostScript summary plot. The output of any of these files may be disabled using the appropriate MODE command line argument. In this example, the output is written to the current mode directory, as we requested on the command line.

The MODE output file naming convention is similar to that of the other MET tools. It contains timing information about the forecast being evaluated (forecast valid, lead, and accumulation times).

If MODE is rerun on the same fields but with a slightly different configuration, the new output will override the old output, unless it is redirected to a different directory using the -outdir command line argument. Users can also edit the output_prefix in the MODE configuration file to customize the output file names.

The 4 MODE output files are described briefly below:

The PostScript file ends in .ps and is described below.
The NetCDF object file ends in _obj.nc and contains the object indices.
The ASCII contingency table statistics file and ends in _cts.txt.
The ASCII object statistics file ends in _obj.txt and contains all of the object and object comparison data.

You can use ghostview (gv) to look at the postscript file output from each of these three forecasts.

gv mode_240000L_20050808_000000V_240000A.ps &

If ghostview is not available, use display to view the files. Click on the image to get a command box, then use File -> Next to move to the next page of the image.

There are multiple pages of output. Take a moment to look them over:

Page 1 summarizes the entire MODE run. Thumbnail images show the input data, resolved objects, and numbers identifying each object for both the forecast and observation fields. The color indicates object matching between the forecast and observation fields. Royal blue indicates an unmatched object. The object definition information is listed at the bottom of the page, and a sorted list of total object interest is listed on the right side.
Page 2 is an expanded view of the forecast thumbnail images.
Page 3 is an expanded view of the observation thumbnail images.
Page 4 has images showing the forecast objects with observation object outlines overlaid, and vice-versa.
Page 5 shows images and statistics for matching object clusters (i.e. one or more forecast objects matching one or more observation objects). These statistics also appear in the ASCII output from MODE.
When double-thresholding or fuzzy engine merging is enabled, additional PostScript pages are added to illustrate those methods.

You may view the output NetCDF file using ncview. Execute the following command to view the NetCDF object output of MODE:

ncview mode_120000L_20050807_120000V_120000A_obj.nc &

Click through the 2D variable names in the ncview window to see plots of the four object fields in the file (NOTE: if a window pops up informing you "the min and max are both...", just Click "OK" and then the field will render). The fcst_obj_id and obs_obj_id contain the indices for the forecast and observation objects defined by MODE. The fcst_clus_id and obs_clus_id contain indices for the matched cluster objects.

What are the benefits of spatial methods over traditional statistics? The weaknesses? What are some examples where an object-based verification would be inappropriate?

cindyhg Tue, 06/25/2019 - 08:04