MET Version 7.0

Release Date:

The MET Version 7.0 was released on March 5, 2018.

Release Notes

What's New?

  • Major Enhancements: add vector statistics for winds, finer control of configuration options, message type groups, and WMO mean summary statistics

Enhancements to Existing Tools:

  • Distribute the latest set of bugfixes, details not listed here.
  • Vector statistics for winds
    • Add vector wind speeds to the existing VL1L2 line type (F_SPEED_BAR and O_SPEED_BAR).
    • Add a new VCNT line type containing statistics for wind vectors.
    • Enhance STAT-Analysis to parse the updated VL1L2 line type, parse the new VCNT line type, and derive VCNT statistics from input VL1L2 lines.
    • Enhance STAT-Analysis to read U/V matched pair (MPR) lines and derive VL1L2 or VCNT output lines.
  • Config file options
    • Refactor config file logic for Point-Stat, Grid-Stat, and Ensemble-Stat.
    • Whenever possible, parse config file options separately for each verification task rather than parsing from the top level.
    • See default config files for the list of options which may be specified separately for each "obs.field" entry.
    • Parse the "regrid" option separately for each field so that regridding logic can be customized for each field (e.g. use BUDGET for precipitation and BILIN for temperature).
    • Add "message_type_group" config file option to define message types that should be processed together as a group. Enhance PB2NC, Point-Stat, and Ensemble-Stat to parse and process groups of message types.
  • PB2NC
    • Add specialized processing for AIRNOW message types to use the TPHR value as the accumulation interval and the QCIND value as the quality control value.
  • STAT-Analysis
    • Enhance the existing summary job type.
    • Print debug messages and warnings when summary includes multiple values for each header column.
    • Add three new output columns for WMO-approved means of daily scores (WMO_TYPE, WMO_MEAN, and WMO_WEIGHTED_MEAN).
    • Process columns of data from multiple input line types when the following format is used "-column LINE_TYPE:COLUMN".
    • Add -derive job command option to automatically compute statistics on the fly from input partial sums (SL1L2/SAL1L2 -> CNT, VL1L2 ->VCNT, and CTC -> CTS).
  • Point-Stat
    • Append the shape to the INTERP_MTHD column but omit it for NEAREST, BILIN, and BUDGET methods.
  • Grid-Stat
    • Add "nbrhd.field" config file option to control the computation of fractional coverage fields.
  • Ensemble-Stat
    • Add "nc_var_str" config file option to customize NetCDF variable names.
    • Add "ensemble_flag.latlon" config file option.
  • MTD
    • Rename output files using a more explicit naming convention:
      • Rename *_3d_ss.txt to *_3d_simple_single.txt.
      • Rename *_3d_sc.txt to *_3d_simple_cluster.txt.
      • Rename *_3d_ps.txt to *_3d_pair_single.txt.
      • Rename *_3d_pc.txt to *_3d_pair_cluser.txt.
    • Update output file conventions to match the logic of MODE:
      • Rename the "OBJ_ID" column to "OBJECT_ID".
      • Rename the "CLUSTER_ID" column to "OBJECT_CAT".
      • Change object naming conventions for "OBJECT_ID" and "OBJECT_CAT":
        • FROM: F_#, O_#, F_#_O_#, CF_#, CO_#, and CF_#_CF_#
        • TO: F###, O###, F###_O###, CF###, CO###, and CF###_CO###
        • Where ### is a 3-digit object number with leading 0's.
        • Indicate unmatched objects with a 3-digit "000" object number.
      • Only write pair information to *_pair_cluster.txt for matches.


Known Issues and Fixes

All Recommended Updates (Last Updated 09/24/2018)

Retrieve all of the recommended updates for METv7.0 in a tarfile:

  • Download the tarfile met-7.0_patches_20180924.tar.gz
  • Copy it into the top-level met-7.0 directory.
  • Uncompress and untar the file. Please note that the new version of the files in the tarball will overwrite the original version o f those files. Any changes you may have made will be lost.
  • Rebuild MET, being sure to run make clean and configure first.

Summary of Recommended Updates

  • 09/24/2018: Fix equality checking for complex thresholds.
  • 09/07/2018: Fix Grid-Stat slow runtimes for neighborhood methods.
  • 07/24/2018: Fix Grid-Stat memory allocation issue causing slow runtimes.
  • 06/13/2018: Fix user-reported compilation errors.
  • 05/18/2018: Fix logic when applying multiple censoring thresholds.
  • 05/18/2018: Fix logic when sizing the ascii output for Point-Stat and Grid-Stat.
  • 05/03/2018: Fix tc_pairs interp12 logic for ATCF ID's ending with "32".
  • 04/26/2018: Fix processing of GRIB2 MXUPHL.
  • 04/03/2018: Fix ORANK header columns.
  • 03/27/2018: Fix STAT-Analysis conversion from VL1L2 to VCNT.
  • 03/27/2018: Fix PB2NC for large input files.
  • 03/20/2018: Fix uninitialized value bug in grid_stat.

Details

Fix equality checking for complex thresholds.
Posted 09/24/2018

Problem: Any thresholds which include '&&' or '||' are called complex thresholds in MET. If you have used complex thresholds in your MET config files, they will show up in the .stat output files, often the FCST_THRESH and/or OBS_THRESH output columns. If you then run the STAT-Analysis tool using the "-fcst_thresh" or "-obs_thresh" job command options to filter those columns, the filtering will have no apparent effect. The logic for comparing the equality of two complex thresholds is insufficient. It always results in true.
Solution: The fix is enhancing the logic for checking the equality of thresholds. For complex thresholds, first convert them to strings and check for equality of those strings. For simple thresholds, use the existing logic, which is slightly faster.
Update: met-7.0/src/basic/vx_config/threshold.cc

Fix Grid-Stat slow runtimes for neighborhood methods.
Posted 09/07/2018

Problem: met-7.0 added support for applying circular neighborhoods, and the computation of the fractional coverage fields was redone to support them. However, the updated code recomputes the fractional coverage value from scratch for each grid point. This approach is roughly 5 times slower than the implementation in previous releases.
Solution: The fix is enhancing the underlying data structures and reimplementing the computation of the fractional coverage fields using the algorithm from previous releases. Rather than recomputing the fractional coverage value from scratch for each grid point, keep track of running sums. For each step, subtract the points from the trailing edge and add the points from the leading edge. This fix produces identical results but runs about as quickly as earlier releases.
Update: met-7.0/src/basic/vx_util/GridTemplate.h
Update: met-7.0/src/basic/vx_util/GridTemplate.cc
Update: met-7.0/src/basic/vx_util/CircularTemplate.cc
Update: met-7.0/src/basic/vx_util/RectangularTemplate.cc
Update: met-7.0/src/basic/vx_util/data_plane_util.cc

Fix Grid-Stat memory allocation issue causing slow runtimes.
Posted 07/24/2018

Problem: Grid-Stat populates a PairDataPoint object to prepare data for the computation of statistics. That object incrementally builds up the memory it requires which is exceedingly slow for a large number of points.
Solution: Add a one line fix to pre-allocate the memory that object requires. This fix improves the runtime for a sample test case from 18 minutes down to about 1 minute.
Update: met-7.0/src/libcode/vx_statistics/pair_data_point.cc

Fix user-reported compilation errors.
Posted 06/13/2018

Problem: A user reported a compilation error through MET-Help from recent GNU compilers in calls to the make_temp_file_name() function.
Solution: Very minor changes to three file to fix compilation errors.
Update: met-7.0/src/libcode/vx_statistics/compute_ci.cc
Update: met-7.0/src/tools/other/pb2nc/pb2nc.cc
Update: met-7.0/src/tools/core/stat_analysis/stat_analysis.cc

Fix logic when applying multiple censoring thresholds.
Posted 05/18/2018

Problem: When multiple censoring thresholds are defined, the output from the first censor is used as the input to the next. Iteratively applying the censoring logic in this way may lead to unexpected results, depending on the thresholds and replacement values chosen.
Solution: Loop through the censoring logic for each data value and stop after finding the first matching censor threshold. Apply this change to the processing of both gridded data and point observations.
Update: met-7.0/src/basic/vx_util/data_plane.h
Update: met-7.0/src/basic/vx_util/data_plane.cc
Update: met-7.0/src/libcode/vx_statistics/pair_base.cc

Fix logic when sizing the ascii output for Point-Stat and Grid-Stat.
Posted 05/18/2018

Problem: The Point-Stat and Grid-Stat tools mistakenly consider the number of probabilistic thresholds when determining the number of columns needed for the MCTC output line type. This leads to the AsciiTable class creating many extra unneeded output columns. The results are not incorrect, but the tools may run much slower when verifying probabilities.
Solution: Revert back to the logic used in met-6.1.
Update: met-7.0/src/tools/core/point_stat/point_stat_conf_info.h
Update: met-7.0/src/tools/core/point_stat/point_stat_conf_info.cc
Update: met-7.0/src/tools/core/grid_stat/grid_stat_conf_info.h
Update: met-7.0/src/tools/core/grid_stat/grid_stat_conf_info.cc

Fix tc_pairs interp12 logic for ATCF ID's ending with "32".
Posted 05/03/2018

Problem: The interp12 logic doesn't work correctly for ATCF ID's ending with "32". For example, ATCF ID "E032" gets converted to "E0I" when it should be "E02I".
Solution: One line fix correcting the renaming logic.
Update: met-7.0/src/tools/tc_utils/tc_pairs/tc_pairs.cc

Fix processing of GRIB2 MXUPHL.
Posted 04/26/2018

Problem: MET is unable to distinguish between multiple records of MXUPHL from a GRIB2 file.
Solution: Update the vx_data2_grib2 library to process this data.
Update: met-7.0/src/libcode/vx_data2d_grib2/data2d_grib2.cc

Fix STAT-Analysis conversion from VL1L2 to VCNT for MSVE and RMSVE columns.
Posted 04/03/2018

Problem: When STAT-Analysis reads a single VL1L2 input line and writes a VCNT line, the resulting MSVE and RMSVE statistics are zero.
Solution: Compute the MSVE and RMSVE statistics using the partial sums rather than the U/V matched pairs.
Update: met-7.0/src/libcode/vx_statistics/met_stats.cc

Fix STAT-Analysis conversion from VL1L2 to VCNT.
Posted 03/27/2018

Fix ORANK header columns.
Posted 04/03/2018

Problem: The last two header columns of the ORANK output line type are incorrectly listed as "ENS_SPREAD ENS_MEAN" when they should be "CLIMO ENS_SPREAD".
Solution: Correct the header columns.
Update: met-6.1/src/libcode/vx_stat_out/stat_columns.h

Fix STAT-Analysis conversion from VL1L2 to VCNT.
Posted 03/27/2018

Problem: When STAT-Analysis reads a single VL1L2 input line and writes a VCNT line, the resulting statistics are all zero. The job fails to compute vector statistics for a single input VL1L2 line.
Solution: After parsing each input VL1L2 line call a member function to compute the vector statistics.
Update: met-7.0/src/tools/core/stat_analysis/parse_stat_line.cc

Fix PB2NC for large input files.
Posted 03/27/2018

Problem: Processing large input file with PB2NC results in a NetCDF error at runtime ... netCDF::exceptions::NcBadId
Solution: Update PB2NC to correctly process these large files.
Update: met-7.0/src/tools/other/pb2nc/pb2nc.cc

Fix uninitialized value bug in grid_stat.
Posted 03/20/2018

Problem: met-7.0 introduced a bug in grid_stat for reading beyond the end of an array which may lead to a segmentation fault. The runtime error may occur sporadically, depending on how variables are initialized. The bug arose when making the defition of the output_flag configurable for each verification task.
Solution: Correct the code to reference the top-level output_flag rather than one for the i-th verification task.
Update: met-7.0/src/tools/core/grid_stat/grid_stat.cc