Release Options
Component
Release Date:
The MET Version 8.0 was released on September 27, 2018.
Release Notes
What's New?
- Major Enhancements: python embedding, observation error logic, shapefiles for masking regions, point observation format overhaul, and rotated lat/lon projections
- New Tools: Added three new utilities to dump shapefile information (gis_dump_dbf, gis_dump_shp, and gis_dump_shx)
- Please see slides describing recent changes: met-8.0_updates.pdf
Changes to Output Formats:
- Added new Ensemble Continuous Statistics line type (ECNT) for Ensemble-Stat and Point-Stat HiRA.
- Removed CRPS, IGN, CRPSS, and SPREAD columns from the RHIST line type and move them to the new ECNT line type.
- Added ENS_MEAN_OERR, SPREAD_OERR, SPREAD_PLUS_OERR columns to the ORANK line type.
- Added ASPECT_DIFF and CURVATURE_RATIO columns to the MODE output.
Enhancements to Existing Tools:
- Distributed the latest set of bugfixes (details not listed here).
- Removed many compilation warnings.
- Python Embedding
- Enhanced the MET tools to read gridded data by calling a Python script which loads data and metadata into a NumPy object.
- Note that users must configure MET using "--enable-python" to enable this functionality.
- Example "scripts/python" and "data/python" directories are now included.
- See Appendix F of the MET User's Guide for more information.
- Common Libraries
- Added support for Rotated Lat/Lon projections.
- Updated the MRMS GRIB2 table.
- Point Pre-Processing Tools
- Made a major change to the NetCDF point observation format written by PB2NC, ASCII2NC, MADIS2NC, and LIDAR2NC. Data are now split out into multiple 1-dimensional arrays. Created "table" variables to store unique strings and, for each observation, to store the indices for those tables.
- Updated the NetCDF point observation format by replacing the UNLIMITED dimension with a fixed dimension, resulting in smaller file sizes.
- Multiple Tools
- For the statistics tools, changed the default config file "output_flag" settings from BOTH to NONE.
- For climatologies, added the "match_month" flag to the "climo_mean" and "climo_stdev" config file dictionaries. Setting "match_month" to FALSE now enable the use of persistence to define the climatology.
- Added support for area-weighted (AW_MEAN) regridding logic.
- Enhanced the ASCII2NC, MADIS2NC, PB2NC, STAT-Analysis, TC-Pairs, and TC-STAT to support the "mask.poly" config file option or "-mask_poly" command line option to process masking regions consistently across all the MET tools. They can now process the output of the Gen-Vx-Mask tool.
- Enabled "vld_thresh = 0" in Ensemble-Stat and Series-Analysis config files.
- Switched from using double precision to booleans for storing mask fields in Point-Stat, Grid-Stat, Ensemble-Stat, Wavelet-Stat, and Series-Analysis. When processing many masking regions, this will run faster and consume less memory.
- Enhanced the logic for finding matching UGRD/VGRD verification tasks in Point-Stat and Grid-Stat to make it more robust.
- Gen-Vx-Mask
- Added support for the using shapefiles to define masking regions with the "-type shape" option.
- Regrid-Data-Plane
- Added the capability to read GOES-16 AOD data using the new "-coord" and "-qc" command line options.
- ASCII2NC
- Added support for the AERONET version 3 point observation format.
- Added support for longer observation variable names.
- PB2NC
- Enhanced the output to include the PrepBufr values for report type, input report type, and instrument type for use in the observation error table lookup.
- Fixed the computation of valid time for AIRNOW observations.
- Added special handling for ANOWPM message types.
- Rounded timestamps to the nearest second instead of truncating.
- Enhanced the "-nmsg" command line option to be specified as the exact number of messages to be processed (e.g. -nmsg 25) or as a percentage of the messages (e.g. -nmsg 25%).
- Time Summaries
- Time summaries are now computed across multiple input files for ASCII2NC and PB2NC.
- Enhanced the "time_summary" config file option by adding a "raw_data" flag to control whether the raw input values are written or only the time summary values.
- Generalized the "time_summary.width" config file option to an integer for centered time windows (existing) or a dictionary with begin and end time offsets for uncentered windows (new).
- Grid-Stat
- Fixed the very slow computation of neighborhood methods fractional coverage fields introduced in met-7.0 when supporting circular neighborhoods. (This was also posted as a bugfix for met-7.0.)
- Fixed a memory allocation issue, leading to faster run times.
- Point-Stat
- The ECNT line type is now computed when applying the HiRA methodology, in which nearby grid points are treated as ensemble member values.
- Point-Stat and Ensemble-Stat
- Added the "mask.llpnt" config file option to appply thresholds to filter point observations by applying thresholds to their lat/lon values directly.
- Ensemble-Stat
- Added support for observation error perturbations. (See "data/config/README" for details on these settings.)
- Added a default lookup table in "data/table_files/obs_error_table.txt" but also support the MET_OBS_ERROR_TABLE environment variable for user -specified table.
- Added the "obs_error" dictionary to the config file to specify how observation error perturbations should be applied.
- Added the capability to write statistics for perturbed and unperturbed data to the new ECNT line type.
- Removed the columns from the RHIST line type that were moved to ECNT.
- Renamed the "-ssvar_mean" command line option as "-ens_mean", but the "-ssvar_mean" option is still quietly supported.
- MODE
- Replaced the "area_thresh" and "inten_perc_thresh" config file options with the more generic "filter_attr_name" and "filter_attr_thresh" options.
- Added "aspect_diff" and "curvature_ratio" as new inputs to the fuzzy logic engine and added corresponding columns to the output file.
- Series-Analysis
- Added debug messages to inform the user about missing data.
- STAT-Analysis
- Added the capability to aggregate the ECNT line type and derive it from input ORANK lines.
- Updated the computation of MSVE statistic in the VCNT line type to be done using partial sums rather than matched pair values. This enables it to be computed from VL1L2 lines in STAT-Analysis.
- STAT-Analysis and TC-STAT
- Added the "-column_union" job command option to summarize values across multiple columns.
- Scripting
- Added a sample Rscript (scripts/Rscripts/plot_ensemble.R) for plotting ensemble line types.
- Updated the pntnc2ascii.R Rscript to handle the obs_var variable.
Known Issues and Fixes
Fix MODE memory corruption bug
Posted: 2019-04-04
Problem: In met-8.0, MODE was enhanced to remove the hard-coded upper limit on the number of object sets by dynamically reallocating memory as needed. This introduced a bug in writing past the end of that dynamically allocated array.
Solution: Add code to reallocate as needed to manage the memory correctly.
Fix GRIB2 support for southern hemisphere Polar Stereographic and Lambert Conformal grids
Posted: 2019-04-04
Problem: When parsing Polar Stereographic and Lambert Conformal grid definitions from GRIB2 files, MET did not apply the correct logic to determine the hemisphere, leading to the following runtime error:
ERROR : MetGrib2DataFile::read_grib2_grid() -> unexpected polar stereo projection center (128)
ERROR : MetGrib2DataFile::read_grib2_grid() -> unexpected polar stereo projection center (128)
Solution: Fix MET's GRIB2 library to parse the hemisphere correctly.
Update:
Fix point_stat error checking for vector statistics
Posted: 2019-03-06
Problem: When computing vector statistics (VL1L2 or VAL1L2 output), point_stat checks to make sure that each UGRD observation is followed by a VGRD observation. However an input point observation file containing more than 33 unique observation variable types results in a runtime error similar to the following:
ERROR : process_obs_file() -> for observation index 503, when computing VL1L2 and/or VAL1L2 vector winds each UGRD observation must be followed by a VGRD observation with the same header and at the same level.
This bug appeared in met-6.1 after switching from identifying point observations by a GRIB code to a string.
ERROR : process_obs_file() -> for observation index 503, when computing VL1L2 and/or VAL1L2 vector winds each UGRD observation must be followed by a VGRD observation with the same header and at the same level.
This bug appeared in met-6.1 after switching from identifying point observations by a GRIB code to a string.
Solution: Update this logic to handle the use of GRIB codes (old format) or the use of the strings (new format).
Update:
All Recommended Updates (Last Updated 04/04/2019)
Retrieve all of the recommended updates for METv8.0 in a tarfile:
- Download the tarfile met-8.0_patches_20190404.tar.gz
- Copy it into the top-level met-8.0 directory.
- Uncompress and untar the file. Please note that the new version of the files in the tarball will overwrite the original version o f those files. Any changes you may have made will be lost.
- Rebuild MET, being sure to run make clean and configure first.
Summary of Recommended Updates
- 04/04/2019: Fix MODE memory corruption bug.
- 04/04/2019: Fix GRIB2 support for southern hemisphere Polar Stereographic and Lambert Conformal grids.
- 03/06/2019: Fix point_stat error checking for vector statistics.
- 01/30/2019: Fix pcp_combine handling of bad data for the -add command.
- 01/04/2019: Fix Fourier decomposition for anomaly statistics in Grid-Stat.
- 12/14/2018: Make the default MTD config file more consistent.
- 12/14/2018: Fix mode_quilt_plot.R script to handle missing data.
- 12/14/2018: Fix include file for GNU 8.2.0.
- 11/08/2018: Fix interpolation bug in Point-Stat HiRA ECNT output.
- 11/06/2018: Addess PB2NC slowness.
- 11/01/2018: Fix GRIB1 precipitation accumulation intervals larger than 100 hours.
- 10/19/2018: Fix pcp_combine segfault for python scripts.
- 10/19/2018: Fix segfaults for python scripts.
- 10/19/2018: Enhance support for NetCDF CF-Convention.
Details
Fix MODE memory corruption bug.
Posted 04/04/2019
Problem: In met-8.0, MODE was enhanced to remove the hard-coded upper limit on the number of object sets by dynamically reallocating memory as needed. This introduced a bug in writing past the end of that dynamically allocated array.
Solution: Add code to reallocate as needed to manage the memory correctly.
Update: met-8.0/src/libcode/vx_shapedata/set.h
Update: met-8.0/src/libcode/vx_shapedata/engine.cc
Fix GRIB2 support for southern hemisphere Polar Stereographic and Lambert Conformal grids.
Posted 04/04/2019
Problem: When parsing Polar Stereographic and Lambert Conformal grid definitions from GRIB2 files, MET did not apply the correct logic to determine the hemisphere, leading to the following runtime error:
ERROR : MetGrib2DataFile::read_grib2_grid() -> unexpected polar stereo projection center (128)
Solution: Fix MET's GRIB2 library to parse the hemisphere correctly.
Update: met-8.0/src/libcode/vx_data2d_grib2/data2d_grib2.cc
Fix point_stat error checking for vector statistics.
Posted 03/06/2019
Problem: When computing vector statistics (VL1L2 or VAL1L2 output), point_stat checks to make sure that each UGRD observation is followed by a VGRD observation. However an input point observation file containing more than 33 unique observation variable types results in a runtime error similar to the following:
ERROR : process_obs_file() -> for observation index 503, when computing VL1L2 and/or VAL1L2 vector winds each UGRD observation must be followed by a VGRD observation with the same header and at the same level.
This bug appeared in met-6.1 after switching from identifying point observations by a GRIB code to a string.
Solution: Update this logic to handle the use of GRIB codes (old format) or the use of the strings (new format).
Update: met-8.0/src/tools/core/point_stat/point_stat.cc
Fix pcp_combine handling of bad data for the -add command.
Posted 01/30/2019
Problem: In pcp_combine, if any of the input values for a grid point are bad data, then the output value should be bad data as well. The -add command checks for bad input values but fails to write the bad data value to the output. Solution: Update the -add command to write the bad data value to the output whenever bad input data is found.
Update: met-8.0/src/tools/core/pcp_combine/pcp_combine.cc
Fix Fourier decomposition for anomaly statistics in Grid-Stat.
Posted 01/04/2019
Problem: The Fourier decomposition logic in Grid-Stat fails to decompose the climatological mean field. This yields erroneous output in the Grid-Stat SAL1L2 and VAL1L2 output line types and is not consistent with the methodology applied at NOAA EMC.
Solution: Update Grid-Stat to also decompose the climatological mean field when computing scalar anomaly statistics. Also, write the decomposed climatological mean field to the output NetCDF matched pairs file, if configured to do so.
Update: met-8.0/src/tools/core/grid_stat/grid_stat.cc
Make the default MTD config file more consistent.
Posted 12/14/2018
Problem: In MTD, the start_time_delta, end_time_delta, and time_centroid_delta object pair attributes are all computed as forecast minus observation. These time deltas can be positive or negative. The interest function for the time_centroid_delta is defined for both positive and negative values, while the interest functions for start_time_delta and end_time_delta are defined only for positive values.
Solution: Update the default MTD configuration file to make it more consistent by defining the interest functions for the start_time_delta and end_time_delta for both postive and negative values. Users are welcome to modify this default configuration however they see fit.
Update: met-8.0/data/config/MTDConfig_default
Fix mode_quilt_plot.R script to handle missing data.
Posted 12/14/2018
Problem: The mode_quilt_plot.R script fails when the first MODE output file it reads contains no objects.
Solution: Update the logic in the script to make it more robust.
Update: met-8.0/scripts/Rscripts/mode_quilt_plot.R
Fix include file for GNU 8.2.0.
Posted 12/14/2018
Problem: Compiling MET using GNU version 8.2.0 requires an additional include file.
Solution: Include limits.h in config_util.cc.
Update: met-8.0/src/basic/vx_config/config_util.cc
Fix interpolation bug in Point-Stat HiRA ECNT output.
Posted 11/08/2018
Problem: When selecting grid points to be used in HiRA ensemble computations, Point-Stat may choose the wrong set of forecast grid points. When the HiRA interpolation width is odd, Point-Stat centers the interpolation area on the lower-left grid point, rather than choosing the nearest one.
Solution: Fix the interpolation logic. For odd interpolation widths, center the interpolation area on the nearest grid point. For even interpolation widths, start with the four surrounding grid points.
Update: met-8.0/src/basic/vx_util/interp_util.h
Update: met-8.0/src/basic/vx_util/interp_util.cc
Address PB2NC slowness.
Posted 11/06/2018
Problem: PB2NC runs rather slowly and could be made more efficient.
Solution: PB2NC calls BUFRLIB to read BUFR/PREPBufr messages. It can call BUFRLIB using a small, medium, or large memory buffer. By default, met-8.0 uses the large memory buffer. This fix uses the level_range setting from the PB2NC configuration file to select the smallest buffer possible for the requested levels. In addition, it refines the logic of how/when to process the message headers and data, enhancing runtime efficiency.
Update: met-8.0/src/tools/other/pb2nc/pb2nc.cc
Update: met-8.0/src/tools/other/pb2nc/readpb.f
Update: met-8.0/src/tools/other/pb2nc/readpb.prm
Fix GRIB1 precipitation accumulation intervals larger than 100 hours.
Posted 11/01/2018
Problem: The GRIB1 library code in MET applies some logic to accumulated precipitation but is only checking for 2-digit (i.e. HH for hours) or 6-digit (i.e. HHMMSS for hours, minutes, and seconds) accumulations. This logic does not work for 3-digit accumulations, such as 120 hours (or 5 days) of accumulation. These accumulations may appear in the post-processed output of WRF-ARW which, by default, stores runtime accumulations.
Solution: Rather than checking for 2 or 6 digits, use a regular expression to check for any number of digits.
Update: met-8.0/src/libcode/vx_data2d_grib/data2d_grib_utils.cc
Fix pcp_combine segfault for python scripts.
Posted 10/19/2018
Problem: Passing data to pcp_combine using a python script results in a segfault unless the "-name" command line option is used. pcp_combine accesses null strings prior to them being set by the python script.
Solution: Change the order of the logic to not access those strings until after they have been set.
Update: met-8.0/src/tools/core/pcp_combine/pcp_combine.cc
Fix segfaults for python scripts.
Posted 10/19/2018
Problem: MET fails to check for NULL pointers when processing python scripts, potentially leading to a segfault.
Solution: Update library code to check for NULL pointers prior to accessing their values.
Update: met-8.0/src/libcode/vx_data2d_python/dataplane_from_numpy_array.cc
Enhance support for NetCDF CF-Convention.
Posted 10/19/2018
Problem: MET fails to parse timing information from CF-Compliant NetCDF files which do not define the "time" variable.
Solution: Add support for time coordinates attribute and standard name for time variables.
Update: met-8.0/src/libcode/vx_data2d_nccf/nccf_file.h
Update: met-8.0/src/libcode/vx_data2d_nccf/nccf_file.cc
Addresses
mailing:
PO Box 3000
Boulder, CO 80307-3000
Shipping:
3090 Center Green Dr.
Boulder, CO 80301
Follow Us
Copyright © 2024. All rights reserved.