Traditional radar product integrity suffers from gaps in coverage over large areas. While some geographical areas are fortunate to have ground- and/or satellite-based radar coverage, most areas around the globe do not have complete coverage from reliable radar networks. The use of radar-based products is advantageous for understanding current weather conditions and how weather systems may evolve over time. Convection and precipitating weather systems often impact critical missions for the United States Air Force (USAF); therefore, access to products that go beyond traditional radar output would be beneficial for mission planning and execution. To address this, the Massachusetts Institute of Technology Lincoln Laboratory (MITLL) has created the Global Synthetic Weather Radar (GSWR) product, which provides near-global coverage of radar-based outputs using advanced machine-learning techniques. To aid the USAF in making evidenced-based decisions when using GSWR products, the Developmental Testbed Center (DTC) performed rigorous veracity testing to compare GSWR products against several ground-based radar outputs to ensure the products were developed properly. The evaluation was performed using the enhanced Model Evaluation Tools (METplus) and was largely conducted on the cloud via Amazon Web Services (AWS). Both traditional evaluation approaches, as well as more advanced spatial approaches, including the application of MET’s Method for Object-based Diagnostic Evaluation (MODE), were employed.
The DTC’s evaluation was based on GSWR analysis data spanning from 15 May - 31 August 2020. These products included composite reflectivity, echo top, and vertically integrated liquid. GSWR products were compared against several observation and analysis datasets, including Multi-Radar/Multi-Sensor (MRMS) products, Next Generation Weather Radar (NEXRAD) products, Operational Programme for the Exchange of Weather Radar Information (OPERA) products.
For all three variables evaluated, GSWR analyses showed the highest skill at the lowest thresholds evaluated and the lowest skill at the highest thresholds evaluated; VIL typically showed lower skill than composite reflectivity and echo tops. Object-based evaluation provided value-added information beyond traditional measures. Key results from the object-based evaluation established that GSWR often produced fewer, larger composite reflectivity and echo top MODE objects when compared to the “truth” datasets and GSWR MODE objects typically exhibited weaker intensities and a south-west displacement. The results from this robust veracity testing activity were shared with the USAF and MITLL with the goal of providing information to make informed operational decisions as well as aid in the future development of GSWR products.
