MODE Tool: Configure

Start by making an output directory for MODE and changing directories:

The behavior of MODE is controlled by the contents of the configuration file passed to it on the command line. The default MODE configuration file may be found in the data/config/MODEConfig_default file. Prior to modifying the configuration file, users are advised to make copies of existing configuration files:

We'll be using these three configuration files during this session. Open up the MODEConfig_APCP_12 file to view it.

The configuration items for MODE are used to specify how the object-based verification approach is to be performed. In MODE, as in the other MET statistics tools, you can compare any two fields. When necessary, the items in the configuration file are specified separately for the forecast and observation fields. In most cases though, users will be comparing the same forecast and observation fields. The configurable items include parameters for the following:

• The forecast and observation fields and vertical levels or accumulation intervals to be compared
• Options to mask out a portion of or threshold the raw fields
• The forecast and observation object definition parameters
• Options to filter out objects that don't meet a size or intensity criteria
• Flags to control the logic for matching/merging
• Weights to be applied for the fuzzy engine matching/merging algorithm
• Interest functions to be used for the fuzzy engine matching/merging algorithm
• Total interest threshold for matching/merging
• Various plotting options

While the MODE configuration file contains many options, beginning users will typically only need to modify a few of them. You may find a complete description of the configurable items in section 14.3.2 of the MET User's Guide. Please take some time to review them.

We'll start here using by running the configuration files we copied over, as-is.

MODE Tool: Run

Next, run MODE three times on the command line using those three configuration files with the following commands:

These commands make use of sample data that's distributed with the MET tarball. They run MODE on 12-hour accumulated precipitation, 24-hour accumulated precipitation, and on a field of relative humidity.

MODE Tool: Output

The output of MODE typically consists of 4 files: 2 ASCII statistics files, 1 NetCDF object file, and 1 PostScript summary plot. The output of any of these files may be disabled using the appropriate MODE command line argument. In this example, the output is written to the current mode directory, as we requested on the command line.

The MODE output file naming convention is similar to that of the other MET tools. It contains timing information about the forecast being evaluated (forecast valid, lead, and accumulation times). If you rerun MODE on the same fields but with a slightly different configuration, the new output will override the old output, unless you redirect it to a different directory using the -outdir command line argument. You can also edit the output_prefix in the MODE configuration file to customize the output file names. The 4 MODE output files are described briefly below:

• The PostScript file ends in .ps and was described on the previous page.
• The NetCDF object file ends in _obj.nc and contains the object indices.
• The ASCII contingency table statistics file and ends in _cts.txt.
• The ASCII object statistics file ends in _obj.txt and contains all of the object and object comparison data.

You can use ghostview (gv) to look at the postscript file output from each of these three forecasts.

There are multiple pages of output. Take a moment to look them over:

1. Page 1 summarizes the entire MODE run. Thumbnail images show the input data, resolved objects, and numbers identifying each object for both the forecast and observation fields. The color indicates object matching between the forecast and observation fields. Royal blue indicates an unmatched object. The object definition information is listed at the bottom of the page, and a sorted list of total object interest is listed on the right side.

2. Page 2 is an expanded view of the forecast thumbnail images.
3. Page 3 is an expanded view of the observation thumbnail images.
4. Page 3 is an expanded view of the observation thumbnail images.
5. Page 4 has images showing the forecast objects with observation object outlines overlaid, and vice-versa.
6. Page 5 shows images and statistics for matching object clusters (i.e. one or more forecast objects matching one or more observation objects). These statistics also appear in the ASCII output from MODE.
7. When double-thresholding or fuzzy engine merging is enabled, additional PostScript pages are added to illustrate those methods.

You may view the output NetCDF file using ncview. Execute the following command to view the NetCDF object output of MODE:

Click through the 2D variable names in the ncview window to see plots of the four object fields in the file (NOTE: if a window pops up informing you "the min and max are both...", just Click "OK" and then the field will render). The fcst_obj_id and obs_obj_id contain the indices for the forecast and observation objects defined by MODE. The fcst_clus_id and obs_clus_idcontain indices for the matched cluster objects.

What are the benefits of spatial methods over traditional statistics? The weaknesses? What are some examples where an object-based verification would be inappropriate?

To accumulate the output of the object based verification, you use the MODE-Analysis Tool. We will use this next.

MTD: Configure

Start by making an output directory for MTD and changing directories:

The behavior of MTD is controlled by the contents of the configuration file passed to it on the command line. The default MTD configuration file may be found in the data/config/MTDConfig_default file. Prior to modifying the configuration file, users are advised to make a copy of the default:

The configuration items for MTD are used to specify how the space-time-object-based verification approach is to be performed. Just as MODE may be used to compare any two fields, the same is true of MTD. When necessary, the items in the configuration file are specified separately for the forecast and observation fields. In most cases though, users will be comparing the same forecast and observation fields. The configurable items include specifications for the following:

• The verification domain.
• The forecast and observation fields and vertical levels or accumulation intervals to be compared.
• The forecast and observation object definition parameters.
• Options to filter out objects that don't meet a minimum volume.
• Matching/merging weights and interest functions.
• Total interest threshold for matching/merging.
• Flags to control output files.

For this tutorial, we'll configure MTD to process the same series of data we ran through the Series-Analysis tool. Just like MODE, MTD compares a single forecast field to a single observation field in each run.

Open up the MTDConfig_tutorial file for editing with the text editor of your choice and edit it as follows:

Set the fcst dictionary as follows:

Set the obs dictionary as follows:

Set:

To retain all objects regardless of their volume.

Save and close this file.

MTD: Run

First, we need to prepare our observations by putting 1-hourly StageII precipitation forecasts into 3-hourly buckets. Create an output directory:

Run the following PCP-Combine commands to prepare the observations:

Rather than listing 8 input forecast and observation files on the command line, we will write them to a file list first. Since the 0-hour forecast does not contain 3-hourly accumulated precip, we will exclude that from the list. We will use the 3-hourly APCP output from PCP-Combine that we prepared above:

Next, run the following MTD command:

Just as with MODE, MTD applies a convolution operation to smooth the data. However, there are two important differences. In MODE, the convolution shape is a circle (radius = conv_radius). In MTD, the convolution shape is a square (width = 2*conv_radius+1) and for time t, the values in that square are averaged for times t-1, t, and t+1. Convolving in space plus time enables MTD to identify more continuous space-time objects.

If your data has high enough time frequency that the features at one timestep overlap those at the next timestep, it may be well-suited for MTD.

MTD: Output

The MTD output typically consists of 6 files: 5 ASCII statistics files and 1 NetCDF object file. MTD does not create any graphical output. In this example, the output is written to the current mtd directory as we requested on the command line.

The MTD output file naming convention begins with mtd_ followed by the last valid time it encountered. The output file names may be modified using the output_prefix option in the configuration file which should be used to prevent the output of one run from over-writing the output of a previous run. The 6 MTD output files are described briefly below:

• The NetCDF object file ends in .nc and contains gridded fields of the raw data, simple object indices, and cluster object indices for each forecast and observed timestep.
• The ASCII file ending with _2D.txt contains many columns similar to the output of MODE. This data summarizes the 2-dimensional object attributes for each individual time slice of the 3D forecast and observation objects.
• The ASCII files ending with _single_simple.txt and _single_cluster.txt contain 3D space-time attributes for simple and cluster objects, respectively.
• The ASCII files ending with _pair_simple.txt and _pair_cluster.txt contain 3D space-time attributes for pairs of simple and cluster objects, respectively.

Use the ncview utility to view the NetCDF object output of MTD:

Select the variable named fcst_raw and click the time index to advance through the timesteps. Now, do the same for the fcst_object_id variable. Notice that the objects are defined in the active areas in the raw fields. Also notice some features merging (i.e. combining) as time passes while other features split (i.e. break apart). While they may be disconnected at a particular timestep, they remain part of the same space-time object.

Next, explore the ASCII output files and pay close attention to the header columns. Notice the generalization of the 2D MODE object attributes to 3 dimensions. Area meausure become volume. MTD measure the object speed. Each object has a begining and ending time.