Artificial Intelligence (AI)-based weather prediction (AIWP) systems are rapidly evolving and showing promise for filling critical forecast needs. With new technologies comes the need to conduct thorough evaluations and extend our evaluation metrics to address the emerging challenges and opportunities associated with AI-based weather forecasts. To address this need, the DTC is extending the functionality of the DTC Verification Workflow to provide the capability to conduct benchmarking of AIWP models against current operational NWP models. One key aspect of this work is expanding the capabilities of the enhanced Model Evaluation Tools (METplus) to support the evaluation of AIWP output, both in terms of ingesting AIWP output and adding metrics needed to thoroughly assess the performance of AIWP systems. Establishing and enhancing collaborations with key AI/ML community members is central to making sure this capability captures current best practices and understanding. This benchmarking capability will be applied to provide DTC partners with fast and reliable assessment of AIWP system performance, as well as provide feedback to the AIWP developers.
Sponsors: NOAA
Collaborators: NOAA GSL, CSU/CIRA, EPIC, WP-MIP (including NOAA, ECCC, UK Met Office, BoM)
