Session 4: MODE and MTD
Session 4: MODE and MTDMETplus Practical Session 4
During the fourth METplus practical session, you will run the tools indicated below:
During this practical session, please work on the Session 4 exercises. You may navigate through this tutorial by following the arrows in the bottom-right or by using the left-hand menu navigation.
The MET Users Guide can be viewed here:
okular /classroom/wrfhelp/MET/8.0/met-8.0/doc/MET_Users_Guide.pdf &
MET Tool: MODE
MET Tool: MODE cindyhg Tue, 06/25/2019 - 07:58MODE Tool: General
MODE Functionality
MODE, the Method for Object-Based Diagnostic Evaluation, provides an object-based verification for comparing gridded forecasts to gridded observations. MODE may be used in a generalized way to compare any two fields containing data from which objects may be well defined. It has most commonly been applied to precipitation fields and radar reflectivity. The steps performed in MODE consist of:
- Define objects in the forecast and observation fields based on user-defined parameters.
- Compute attributes for each of those objects: such as area, centroid, axis angle, and intensity.
- For each forecast/observation object pair, compute differences between their attributes: such as area ratio, centroid distance, angle difference, and intensity ratio.
- Use fuzzy logic to compute a total interest value for each forecast/observation object pair based on user-defined weights.
- Based on the computed interest values, match objects across fields and merge objects within the same field.
- Write output statistics summarizing the characteristics of the single objects, the pairs of objects, and the matched/merged objects.
MODE may be configured to use a few different sets of logic with which to perform matching and merging. In this tutorial, we'll use the most simple approach, but users are encouraged to read Chapter 14 of the MET User's Guide for a more thorough description of MODE's capabilities.
MODE Usage
View the usage statement for MODE by simply typing the following:
Usage: mode | ||
fcst_file | Input gridded forecast file containing the field to be verified | |
obs_file | Input gridded observation file containing the verifying field | |
config_file | MODEConfig file containing the desired configuration settings | |
[-config_merge merge_config_file] | Overrides the default fuzzy engine settings for merging within the fcst/obs fields (optional). | |
[-outdir path] | Overrides the default output directory (optional). | |
[-log file] | Outputs log messages to the specified file | |
[-v level] | Level of logging | |
[-compress level] | NetCDF compression level |
The forecast and observation fields must be on the same grid. You can use copygb to regrid GRIB1 files, wgrib2 to regrid GRIB2 files, or use the automated regridding functionality within the MET config files.
At a minimum, the input gridded fcst_file, the input gridded obs_file, and the configuration config_file must be passed in on the command line.
MODE Tool: Configure
MODE Tool: Configure cindyhg Tue, 06/25/2019 - 08:01MODE Tool: Configure
Start by making an output directory for MODE and changing directories:
cd ${HOME}/metplus_tutorial/output/met_output/mode
The behavior of MODE is controlled by the contents of the configuration file passed to it on the command line. The default MODE configuration file may be found in the data/config/MODEConfig_default file. Prior to modifying the configuration file, users are advised to make copies of existing configuration files:
cp /classroom/wrfhelp/MET/8.0/met-8.0/scripts/config/MODEConfig_APCP_24 MODEConfig_APCP_24
cp /classroom/wrfhelp/MET/8.0/met-8.0/scripts/config/MODEConfig_RH MODEConfig_RH
We'll be using these three configuration files during this session. Open up the MODEConfig_APCP_12 file to view it.
The configuration items for MODE are used to specify how the object-based verification approach is to be performed. In MODE, as in the other MET statistics tools, you can compare any two fields. When necessary, the items in the configuration file are specified separately for the forecast and observation fields. In most cases though, users will be comparing the same forecast and observation fields. The configurable items include parameters for the following:
- The forecast and observation fields and vertical levels or accumulation intervals to be compared
- Options to mask out a portion of or threshold the raw fields
- The forecast and observation object definition parameters
- Options to filter out objects that don't meet a size or intensity criteria
- Flags to control the logic for matching/merging
- Weights to be applied for the fuzzy engine matching/merging algorithm
- Interest functions to be used for the fuzzy engine matching/merging algorithm
- Total interest threshold for matching/merging
- Various plotting options
While the MODE configuration file contains many options, beginning users will typically only need to modify a few of them. You may find a complete description of the configurable items in section 14.3.2 of the MET User's Guide. Please take some time to review them.
We'll start here using by running the configuration files we copied over, as-is.
MODE Tool: Run
MODE Tool: Run cindyhg Tue, 06/25/2019 - 08:02MODE Tool: Run
Next, run MODE three times on the command line using those three configuration files with the following commands:
/classroom/wrfhelp/MET/8.0/met-8.0/out/pcp_combine/sample_fcst_12L_2005080712V_12A.nc \
/classroom/wrfhelp/MET/8.0/met-8.0/out/pcp_combine/sample_obs_2005080712V_12A.nc \
MODEConfig_APCP_12 \
-outdir . \
-v 2
/classroom/wrfhelp/MET/8.0/met-8.0/data/sample_fcst/2005080700/wrfprs_ruc13_24.tm00_G212 \
/classroom/wrfhelp/MET/8.0/met-8.0/out/pcp_combine/sample_obs_2005080800V_24A.nc \
MODEConfig_APCP_24 \
-outdir . \
-v 2
/classroom/wrfhelp/MET/8.0/met-8.0/data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212 \
/classroom/wrfhelp/MET/8.0/met-8.0/data/sample_fcst/2005080712/wrfprs_ruc13_00.tm00_G212 \
MODEConfig_RH \
-outdir . \
-v 2
These commands make use of sample data that's distributed with the MET tarball. They run MODE on 12-hour accumulated precipitation, 24-hour accumulated precipitation, and on a field of relative humidity.
MODE Tool: Output
MODE Tool: Output cindyhg Tue, 06/25/2019 - 08:04MODE Tool: Output
The output of MODE typically consists of 4 files: 2 ASCII statistics files, 1 NetCDF object file, and 1 PostScript summary plot. The output of any of these files may be disabled using the appropriate MODE command line argument. In this example, the output is written to the current mode directory, as we requested on the command line.
The MODE output file naming convention is similar to that of the other MET tools. It contains timing information about the forecast being evaluated (forecast valid, lead, and accumulation times). If you rerun MODE on the same fields but with a slightly different configuration, the new output will override the old output, unless you redirect it to a different directory using the -outdircommand line argument. You can also edit the output_prefix in the MODE configuration file to customize the output file names. The 4 MODE output files are described briefly below:
- The PostScript file ends in .ps and was described on the previous page.
- The NetCDF object file ends in _obj.nc and contains the object indices.
- The ASCII contingency table statistics file and ends in _cts.txt.
- The ASCII object statistics file ends in _obj.txt and contains all of the object and object comparison data.
You can use okular to look at the postscript file output from each of these three forecasts.
There are multiple pages of output. Take a moment to look them over:
-
Page 1 summarizes the entire MODE run. Thumbnail images show the input data, resolved objects, and numbers identifying each object for both the forecast and observation fields. The color indicates object matching between the forecast and observation fields. Royal blue indicates an unmatched object. The object definition information is listed at the bottom of the page, and a sorted list of total object interest is listed on the right side.
- Page 2 is an expanded view of the forecast thumbnail images.
- Page 3 is an expanded view of the observation thumbnail images.
- Page 3 is an expanded view of the observation thumbnail images.
- Page 4 has images showing the forecast objects with observation object outlines overlaid, and vice-versa.
- Page 5 shows images and statistics for matching object clusters (i.e. one or more forecast objects matching one or more observation objects). These statistics also appear in the ASCII output from MODE.
- When double-thresholding or fuzzy engine merging is enabled, additional PostScript pages are added to illustrate those methods.
You may view the output NetCDF file using ncview. Execute the following command to view the NetCDF object output of MODE:
Click through the 2D variable names in the ncview window to see plots of the four object fields in the file (NOTE: if a window pops up informing you "the min and max are both ???", just Click "OK" and then the field will render). The fcst_obj_id and obs_obj_id contain the indices for the forecast and observation objects defined by MODE. The fcst_clus_id and obs_clus_idcontain indices for the matched cluster objects.
What are the benefits of spatial methods over traditional statistics? The weaknesses? What are some examples where an object-based verification would be inappropriate?
To accumulate the output of the object based verification, you use the MODE-Analysis Tool. We will use this next.
Use Case: MODE
Use Case: MODE cindyhg Tue, 06/25/2019 - 08:07METplus Use Case: MODE
The MODE use case utilizes the MET pcp_combine and mode tools.
Optional: Refer to section 4.1 and 7.3 of the MET Users Guide for a description of the MET tools used in this use case.
Optional: Refer to section 4.4 (A-Z Config Glossary) of the METplus 2.0.4 Users Guide for a reference to METplus variables used in this use case.
Setup
Edit Custom Configuration File
Define a unique directory under output that you will use for this use case.
Edit ${HOME}/METplus/parm/mycustom.conf
Change OUTPUT_BASE to contain a subdirectory specific to the MODE use case.
... other entries ...
OUTPUT_BASE = {ENV[HOME]}/metplus_tutorial/output/mode
Using your custom configuration file that you created in the METplus: CUSTOM Configuration File Settings section and the MODE use case configuration files that are distributed with METplus, you should be able to run the use case using the sample input data set without any other changes.
Review Use Case Configuration File: hrefmean-vs-mrms-qpe.conf
Open the file and look at all of the configuration variables that are defined.
Note that variables in hrefmean-vs-mrms-qpe.conf reference other config variables that have been defined in your mycustom.conf configuration file. For example:
This references INPUT_BASE which is set in your custom config file. METplus config variables can reference other config variables even if they are defined in a config file that is read afterwards.
Run METplus
Run the following command:
METplus is finished running when control returns to your terminal console.
Review the Output Files
You should have output files in the following directories:
- mode_HREF_MEAN_APCP_vs_MRMS_QPE_P06M_NONE_A06_000000L_20170622_000000V_NAA_R1_T1_cts.txt
- mode_HREF_MEAN_APCP_vs_MRMS_QPE_P06M_NONE_A06_000000L_20170622_000000V_NAA_R1_T1_obj.nc
- mode_HREF_MEAN_APCP_vs_MRMS_QPE_P06M_NONE_A06_000000L_20170622_000000V_NAA_R1_T1_obj.txt
- mode_HREF_MEAN_APCP_vs_MRMS_QPE_P06M_NONE_A06_000000L_20170622_000000V_NAA_R1_T1.ps
- mode_HREF_MEAN_APCP_vs_MRMS_QPE_P06M_NONE_A06_000000L_20170622_000000V_NAA_R1_T2_cts.txt
- mode_HREF_MEAN_APCP_vs_MRMS_QPE_P06M_NONE_A06_000000L_20170622_000000V_NAA_R1_T2_obj.nc
- mode_HREF_MEAN_APCP_vs_MRMS_QPE_P06M_NONE_A06_000000L_20170622_000000V_NAA_R1_T2_obj.txt
- mode_HREF_MEAN_APCP_vs_MRMS_QPE_P06M_NONE_A06_000000L_20170622_000000V_NAA_R1_T2.ps
Take a look at some of the files to see what was generated.
Review the Log Files
Log files for this run are found in $HOME/metplus_tutorial/output/mode/logs. The filename contains a timestamp of the current day.
NOTE: If you ran METplus on a different day than today, the log file will correspond to the day you ran. Remove the date command and replace it with the date you ran if that is the case.
Review the Final Configuration File
The final configuration file is $HOME/metplus_tutorial/output/mode/metplus_final.conf. This contains all of the configuration variables used in the run.
MET Tool: MTD
MET Tool: MTD cindyhg Tue, 06/25/2019 - 08:13MODE-Time-Domain: General
MODE-Time-Domain Functionality
The MODE-Time-Domain (MTD) tool was added in MET version 6.0. It applies an object-based verification technique in comparing a gridded forecast to a gridded analysis. It defines 3-dimensional space/time objects, tracking 2-dimensional objects through time. It writes summary object information to ASCII statistics files and writes object fields to NetCDF format. The MTD tool can be used to quantify the duration of events and timinig errors.
MODE-Time-Domain Usage
View the usage statement for MODE-Time-Domain by simply typing the following:
The forecast and observation fields must be on the same grid. You can use copygb to regrid GRIB1 files, wgrib2 to regrid GRIB2 files, or use the automated regridding functionality within the MET config files.
At a minimum, the -fcst and -obs options must be used to specify the data to be processed. Alternatively, the -single option specifies that MTD should be run on a single dataset. The -config option specifies the name of the configuration file.
MODE-Time-Domain: Configure
MODE-Time-Domain: Configure cindyhg Tue, 06/25/2019 - 08:15MODE-Time-Domain: Configure
Start by making an output directory for MTD and changing directories:
cd ${HOME}/metplus_tutorial/output/met_output/mtd
The behavior of MTD is controlled by the contents of the configuration file passed to it on the command line. The default MTD configuration file may be found in the data/config/MTDConfig_default file. Prior to modifying the configuration file, users are advised to make a copy of the default:
The configuration items for MTD are used to specify how the space-time-object-based verification approach is to be performed. Just as MODE may be used to compare any two fields, the same is true of MTD. When necessary, the items in the configuration file are specified separately for the forecast and observation fields. In most cases though, users will be comparing the same forecast and observation fields. The configurable items include specifications for the following:
- The verification domain.
- The forecast and observation fields and vertical levels or accumulation intervals to be compared.
- The forecast and observation object definition parameters.
- Options to filter out objects that don't meet a minimum volume.
- Matching/merging weights and interest functions.
- Total interest threshold for matching/merging.
- Flags to control output files.
For this tutorial, we'll configure MTD to process the same series of data we ran through the Series-Analysis tool. Just like MODE, MTD compares a single forecast field to a single observation field in each run.
Open up the MTDConfig_tutorial file for editing with the text editor of your choice and edit it as follows:
Set the fcst dictionary as follows:
field = {
name = "APCP";
level = "A03";
}
conv_thresh = >=2.54;
}
Set the obs dictionary as follows:
field = {
name = "APCP_03";
level = "(*,*)";
}
conv_thresh = >=2.54;
}
Set:
To retain all objects regardless of their volume.
Save and close this file.
MODE-Time-Domain: Run
MODE-Time-Domain: Run cindyhg Tue, 06/25/2019 - 08:16MODE-Time-Domain: Run
First, we need to prepare our observations by putting 1-hourly StageII precipitation forecasts into 3-hourly buckets. Run the following PCP-Combine commands to prepare the observations:
pcp_combine -sum 00000000_000000 01 20050807_030000 03 \
sample_obs/ST2ml_3h/sample_obs_2005080703V_03A.nc \
-pcpdir /classroom/wrfhelp/MET/8.0/met-8.0/data/sample_obs/ST2ml
pcp_combine -sum 00000000_000000 01 20050807_060000 03 \
sample_obs/ST2ml_3h/sample_obs_2005080706V_03A.nc \
-pcpdir /classroom/wrfhelp/MET/8.0/met-8.0/data/sample_obs/ST2ml
pcp_combine -sum 00000000_000000 01 20050807_090000 03 \
sample_obs/ST2ml_3h/sample_obs_2005080709V_03A.nc \
-pcpdir /classroom/wrfhelp/MET/8.0/met-8.0/data/sample_obs/ST2ml
pcp_combine -sum 00000000_000000 01 20050807_120000 03 \
sample_obs/ST2ml_3h/sample_obs_2005080712V_03A.nc \
-pcpdir /classroom/wrfhelp/MET/8.0/met-8.0/data/sample_obs/ST2ml
pcp_combine -sum 00000000_000000 01 20050807_150000 03 \
sample_obs/ST2ml_3h/sample_obs_2005080715V_03A.nc \
-pcpdir /classroom/wrfhelp/MET/8.0/met-8.0/data/sample_obs/ST2ml
pcp_combine -sum 00000000_000000 01 20050807_180000 03 \
sample_obs/ST2ml_3h/sample_obs_2005080718V_03A.nc \
-pcpdir /classroom/wrfhelp/MET/8.0/met-8.0/data/sample_obs/ST2ml
pcp_combine -sum 00000000_000000 01 20050807_210000 03 \
sample_obs/ST2ml_3h/sample_obs_2005080721V_03A.nc \
-pcpdir /classroom/wrfhelp/MET/8.0/met-8.0/data/sample_obs/ST2ml
pcp_combine -sum 00000000_000000 01 20050808_000000 03 \
sample_obs/ST2ml_3h/sample_obs_2005080800V_03A.nc \
-pcpdir /classroom/wrfhelp/MET/8.0/met-8.0/data/sample_obs/ST2ml
Rather than listing 8 input forecast and observation files on the command line, we will write them to a file list first. Since the 0-hour forecast does not contain 3-hourly accumulated precip, we will exclude that from the list. We will use the 3-hourly APCP output from PCP-Combine that we prepared above:
ls -1 sample_obs/ST2ml_3h/sample_obs* > obs_file_list
Next, run the following MTD command:
-fcst fcst_file_list \
-obs obs_file_list \
-config MTDConfig_tutorial \
-outdir . \
-v 2
Just as with MODE, MTD applies a convolution operation to smooth the data. However, there are two important differences. In MODE, the convolution shape is a circle (radius = conv_radius). In MTD, the convolution shape is a square (width = 2*conv_radius+1) and for time t, the values in that square are averaged for times t-1, t, and t+1. Convolving in space plus time enables MTD to identify more continuous space-time objects.
If your data has high enough time frequency that the features at one timestep overlap those at the next timestep, it may be well-suited for MTD.
MODE-Time-Domain: Output
MODE-Time-Domain: Output cindyhg Tue, 06/25/2019 - 08:17MODE-Time-Domain: Output
The MTD output typically consists of 6 files: 5 ASCII statistics files and 1 NetCDF object file. MTD does not create any graphical output. In this example, the output is written to the current mtd directory as we requested on the command line.
The MTD output file naming convention begins with mtd_ followed by the last valid time it encountered. The output file names may be modified using the output_prefix option in the configuration file which should be used to prevent the output of one run from over-writing the output of a previous run. The 6 MTD output files are described briefly below:
- The NetCDF object file ends in .nc and contains gridded fields of the raw data, simple object indices, and cluster object indices for each forecast and observed timestep.
- The ASCII file ending with _2D.txt contains many columns similar to the output of MODE. This data summarizes the 2-dimensional object attributes for each individual time slice of the 3D forecast and observation objects.
- The ASCII files ending with _single_simple.txt and _single_cluster.txt contain 3D space-time attributes for simple and cluster objects, respectively.
- The ASCII files ending with _pair_simple.txt and _pair_cluster.txt contain 3D space-time attributes for pairs of simple and cluster objects, respectively.
Use the ncview utility to view the NetCDF object output of MTD:
Select the variable named fcst_raw and click the time index to advance through the timesteps. Now, do the same for the fcst_object_id variable. Notice that the objects are defined in the active areas in the raw fields. Also notice some features merging (i.e. combining) as time passes while other features split (i.e. break apart). While they may be disconnected at a particular timestep, they remain part of the same space-time object.
Next, explore the ASCII output files and pay close attention to the header columns. Notice the generalization of the 2D MODE object attributes to 3 dimensions. Area meausure become volume. MTD measure the object speed. Each object has a begining and ending time.
Use Case: MTD
Use Case: MTD cindyhg Tue, 06/25/2019 - 08:20METplus Use Case: MTD
Reference Material
The MTD (Mode Time Domain) use case utilizes the MET mtd tools.
Optional: Refer to section 4.1 and 7.3 of the MET Users Guide for a description of the MET tools used in this use case.
Optional: Refer to section 4.4 (A-Z Config Glossary) of the METplus 2.0.4 Users Guide for a reference to METplus variables used in this use case.
Setup
Edit Custom Configuration File
Define a unique directory under output that you will use for this use case.
Change OUTPUT_BASE to contain a subdirectory specific to the MTD use case.
... other entries ...
OUTPUT_BASE = {ENV[HOME]}/metplus_tutorial/output/mtd
Using your custom configuration file that you created in the METplus: CUSTOM Configuration File Settings section and the MODE use case configuration files that are distributed with METplus, you should be able to run the use case using the sample input data set without any other changes.
Review Use Case Configuration File: phpt-vs-qpe.conf
Open the file and look at all of the configuration variables that are defined.
Note that variables in phpt-vs-qpe.conf reference other config variables that have been defined in your mycustom.conf configuration file. For example:
This references INPUT_BASE which is set in your custom config file. METplus config variables can reference other config variables even if they are defined in a config file that is read afterwards.
Run METplus
Run the following command:
METplus is finished running when control returns to your terminal console.
Review the Output Files
You should have output files in the following directories:
- mtd_PROB_PHPT_APCP_vs_QPE_P01M_NONE_A01_20170510_040000V_2d.txt
- mtd_PROB_PHPT_APCP_vs_QPE_P01M_NONE_A01_20170510_040000V_3d_pair_cluster.txt
- mtd_PROB_PHPT_APCP_vs_QPE_P01M_NONE_A01_20170510_040000V_3d_pair_simple.txt
- mtd_PROB_PHPT_APCP_vs_QPE_P01M_NONE_A01_20170510_040000V_3d_single_cluster.txt
- mtd_PROB_PHPT_APCP_vs_QPE_P01M_NONE_A01_20170510_040000V_3d_single_simple.txt
- mtd_PROB_PHPT_APCP_vs_QPE_P01M_NONE_A01_20170510_040000V_obj.nc
Take a look at some of the files to see what was generated
Open the output NetCDF file with ncview to look at the data:
Click on the buttons in the Var: section (i.e. fcst_raw) to view the fields.
Open an output text file to view the contents:
Review the Log Files
Log files for this run are found in $HOME/metplus_tutorial/output/mtd/logs. The filename contains a timestamp of the current day.
NOTE: If you ran METplus on a different day than today, the log file will correspond to the day you ran. Remove the date command and replace it with the date you ran if that is the case.
Review the Final Configuration File
The final configuration file is $HOME/metplus_tutorial/output/mtd/metplus_final.conf. This contains all of the configuration variables used in the run.
Additional Exercises
Additional Exercises cindyhg Tue, 06/25/2019 - 08:25End of Practical Session 4
Congratulations! You have completed Session 4!
If you have extra time, you may want to try these additional METplus exercises.
EXERCISE 4.1: mtd_skip - Change Forecast Lead List to See METplus Skip Files
Instructions: Modify the METplus configuration files to change the forecast leads that are processed by MTD. Following these instructions will give you more insight on how METplus configures MTD.
Change LEAD_SEQ to process forecast leads 1-4
less $HOME/metplus_tutorial/output/exercises/unzip/stage/mtd_lists/201705100300_obs_P01M_NONE.txt
ls /classroom/wrfhelp/METplus_Data/mode/QPE_Data/20170510
EXERCISE 4.2: unzip - Change Forecast Lead List to See METplus Unzip
Instructions: Modify the METplus configuration files to change the forecast leads that are processed by MTD. Following these instructions will give you more insight on how METplus configures MTD.
Now look at the file list that were generated by METplus for MTD forecast files
If you look in the data directories for this run, you will see that the 6 hour forecast file is gzipped.