During the first few years of the DTC, the United States Air Force (USAF) put forth a vision for the DTC to develop a suite for verification tools that would provide reproducible results such that statistics and metrics could be shared across institutions. The USAF envisioned a suite of tools that would include both standard verification approaches, as well as advanced diagnostic methodologies. In 2008, the DTC released the software package, Model Evaluation Tools (MET) to the community with the intent to provide state-of-the-science verification and diagnostic tools to the community. Over the past 15 years, an enhanced version of MET, called METplus, has become a software system collaboratively developed by the Developmental Testbed Center (DTC) and community partners.
Several operational entities are nearing completion of their research-to-operations process (R2O) to include METplus in their operational verification systems. NOAA’s Environmental Modeling Center (EMC) will be the first to make the transition to a system built using METplus. The first version of the EMC Verification System (EVS v1), which employs METplus to generate real-time model verification statistics, will become operational in fiscal year (FY) 2024. EMC, DTC, and the National Centers for Environmental Prediction (NCEP) Central Operations (NCO) collaborated from FY2022 to the present to install METplus on the Weather and Climate Operational Supercomputer System (WCOSS2), working through software library issues and strict installation standards to ensure METplus meets rigorous WCOSS2 security and installation standards.
Through highly collaborative meetings and frequent communication, EMC and DTC developed a suite of best practices to streamline the METplus installation process with NCO on WCOSS2, drastically reducing the time from the METplus software release to operational capability (e.g., months to days). Ultimately, METplus allows EVS v1 to publish plots for most real-time EMC environmental forecast models to a web page, enabling anyone with internet access the ability to determine how EMC models are currently performing.
EVS version 2 is planned to be an extension of EVS v1 with additional metrics and capability. Development of the EVS v2 system will begin shortly after version 1 becomes operational. The current development plan includes options for the system to run outside of WCOSS2, perhaps in a cloud environment or on other machines.
The METplus governance partners are also developing operational systems powered by METplus. Besides a representative from NOAA labs and centers, the governance partners include NSF NCAR, USAF, United Kingdom Met Office (UKMO), Naval Research Laboratory (NRL), and Australia’s Bureau of Meteorology (BoM). A detailed description of how METplus is being incorporated into the USAF evaluation system is described in the “Director’s Corner” of this newsletter. The USAF plans for METplus to become their cybersecure verification software stack that provides the standard for upgrading their models.
The Met Office made the decision to adopt a MET-based system for its operational verification replacement back in 2019. A first complete instantiation of the current operational system capability will go into parallel testing ahead of operational implementation in mid-2024. The long-term plan is for Met Office scientists and software engineers to become integral members of the open-source development community beyond the DTC. To date, code contributions have been small but significant (e.g. the first inclusion of OpenMP) as have the contributions as subject-matter expertise and beta testers for seven coordinated METplus releases.
Both NRL and Australia's BoM are in the beginning phases of testing METplus for their operational systems. NRL has worked both collaboratively and via funded projects with the NCAR node of DTC to develop support for their ensemble, atmospheric composition, and data assimilation systems. The BoM has been providing in-kind testing and code-management capability to all aspects of METplus to reach their near-term goal of a quick transition of METplus to operational use. In summary, it has taken 15 years, but the vision of a framework of consistent verification tools for the community is becoming more attainable as each institution adopts METplus for both operational use and system development.
