Verification and validation activities are critical to the success of modeling and prediction efforts ongoing at organizations around the world. Having reproducible results via a consistent framework is equally important for model developers and users alike. The Model Evaluation Tools (MET) was developed over a decade ago by the Developmental Testbed Center (DTC) and expanded to the METplus framework with a view towards providing a consistent platform delivering reproducible results.
The METplus system is an umbrella verification, validation, and diagnostic tool at the core of the DTC testing and evaluation capability (Brown et al. 2020). It is also supported by the community of thousands of users from both U.S. and international organizations. These tools are designed to be highly flexible to allow for quick adaptation to meet evaluation and diagnostic needs. A suite of python wrappers has been implemented to facilitate a fast set-up and implementation of the system, and to enhance the pre-existing plotting capabilities.
Over the past few years, METplus has been driven by the needs of the Unified Forecast System (UFS) community, the U.S. Air Force, and the National Center for Atmospheric Research (NCAR) laboratories and collaborators. Many organizations across the community have joined with the DTC core partners in contributing to METplus development, fostering a more robust and dynamic framework for the entire Earth-system modeling community to use. During the past year, several leading organizations in verification and statistics research have joined the METplus contributor’s community.
The Naval Research Lab (NRL) started transitioning their capability to METplus during 2019 and began directly collaborating with METplus developers in late July 2020. They intend to contribute methods for the data assimilation and ensemble communities. Dr. Elizabeth Satterfield, a research scientist and the METplus transition coordinator at NRL, stated “By leveraging community-based tools, we can build a unified verification framework which is consistent across the suite of Navy atmospheric models and also be consistent with our operational partners. Such a framework allows us to make use of more modern verification metrics, including feature or process-based metrics, that can assist with identification and diagnosis of specific sources of model error. In addition, this framework will allow NRL to better tailor our verification products to inform data assimilation and model development, as well as the needs of the end user. Finally, a consistent model verification framework aids in collaboration with other U.S. partners (e.g. NCEP, JEDI) who are employing the same community tools.”
Similarly, after months of discussion and project development activities, the Met Office in the United Kingdom has also begun transitioning their verification and diagnostics capability to MET. The project is called NG-Ver, for Next Generation Verification. Adapting MET to work with unstructured grids is a key requirement and good reason for collaborating. Beyond the operational implementation, the collaboration will eventually focus on integrating innovative methods for verification and diagnostics while generalizing METplus support of file formats. Dr. Marion Mittermaier, the manager of the model diagnostics and novel verification methods section at the Met Office, shared in a brief to her staff that stated “METplus offers a wide variety of highly configurable open source verification tools, which the Met Office can contribute to in the fullness of time. Given the open source nature of MET/METplus, it also enables the wider UM partnership to contribute more readily to getting common verification tools established across the partnership for the evaluation of model releases, especially regional configurations.” She also added “I am particularly excited to see the collaboration formalized at last and look forward to working with DTC scientists and developers. We have many common areas of interest and having a joint framework for leveraging new tools will accelerate the availability of these to the user community.”
Finally, the DTC has been coordinating with NCAR’s unified Earth-system modeling initiative, called System for Integrated Modeling of the Atmosphere (SIMA), for integrating METplus into the SIMA framework as a verification, validation, and diagnostics tool. The National Science Foundation (NSF) community is the target for the framework, and the activities directly support the NCAR/NOAA Memorandum of Agreement, signed in 2019, to work collaboratively on developing a common framework for Earth-system modeling. METplus was recently demonstrated to provide verification and diagnostics for weather-scale prediction (~1 deg) from the SIMA Community Atmosphere Model (CAM). Andrew Gettelman, the SIMA climate lead, said "METplus can provide new ways for us to look at cross scale models and do weather verification even on climate scale model output to help us improve processes important for extreme weather events."
Ultimately, these additions to the METplus community have substantially boosted the DTC’s ability to provide testing and evaluation capability to make Research-to-Operations more efficient and provide evidence-based decisions. The DTC looks forward to continuing to expand METplus capability through these collaborations.
Brown et al, 2020: The Model Evaluation Tools (MET): More than a decade of community-supported forecast verification. Bull. Amer. Meteor. Soc. Early Online Release. DOI: https://doi.org/10.1175/BAMS-D-19-0093.1
