The Developmental Testbed Center (DTC), in collaboration with the National Oceanic and Atmospheric Administration (NOAA) and the Unified Forecast System's Verification and Validation Cross-Cutting Team (UFS-V&V), hosted a three-day workshop to identify key verification and validation metrics for UFS applications. The workshop was held remotely 22-24 February 2021. Registration for the event totaled 315 participants from across the research and operational community. 

The goal of this workshop was to identify and prioritize key metrics to use during the evaluation of  UFS research products, guiding their transition from research-to-operations (R2O).  Because all UFS evaluation decisions affect a diverse set of users, workshop organizers invited members of the government, academic, and private sectors to participate. This outreach resulted in the participation of scientists not only from NOAA, but also from the National Center for Atmospheric Research (NCAR), National Aeronautics and Space Administration (NASA), US National Ice Center (USNIC), seventeen universities, seven commercial entities, and seven international forecast offices and universities.  Ten NOAA research labs were represented, as well as all of the National Weather Service’s National Centers for Environmental Prediction (NCEP), five Regional Headquarters, and ten Weather Forecast offices (WFOs). Rounding out the government organizations included Department of Defense (DOD) and Department of Energy (DOE) entities along with several state government Departments of Environmental Protection.

In preparation for the workshop, a series of three pre-workshop surveys were distributed to interested parties between October 2020 and February 2021. Questions pertaining to fields and levels, temporal and spatial metadata, sources of truth (i.e. observations, analyses, reference models, climatologies), and preferred statistics were included in the surveys. The results were then used to prepare the list of candidate metrics, including their meta-data, for curation by the workshop breakout groups.

Keynote speakers, including Drs. Ricky Rood, Dorothy Koch, and Hendrik Tolman, along with the Workshop Co-Chairs, kicked off the workshop. The R2O process, in which metrics are used to advance the innovations towards progressively higher readiness levels, proceed through stages and gates, described here, as the tools are assessed and vetted for operations. Presentations included 1) a discussion of R2O stages and gates; 2) the results of the pre-workshop surveys; and 3) how the workshop would proceed. Online instantaneous surveys were used throughout the workshop to gather quantitative input from the participants. During the first two days, the breakout groups refined the results of the pre-workshop surveys. At the end of the second day, the participants were invited to fill out 13 online surveys (listed below) to prioritize the metrics for the full R2O stages and gates . On the last day, breakout groups discussed numerous ways to assign metrics to the R2O gates.

Three prominent themes emerged from the workshop.  The first suggested that metrics used in the near term need to be tied to observation availability and should evolve as new observations become available.  A second emphasized that metrics should be relevant to the user and easy to interpret.  Lastly, the results of the ranking polls tended to place the sensible weather and upper-air fields as top priorities, leaving fields from the components of a fully coupled system (i.e. marine, cryosphere, land) nearer the bottom. Given this outcome, a tiger team of experts will be convened to help the UFS V&V team complete the consolidation and synthesis of the results to ensure component fields are also included across all gates.

These summary activities are wrapping up. The UFS V&V group is working with the chairs of other UFS working groups and application teams to finalize the metrics. The organizers intend to schedule a wrap-up webinar in mid-June to update the community on the final metrics.  For more information and updates on the synthesis work, please visit the DTC UFS Evaluation Metrics Workshop website. Additionally, the Verification Post-Processing and Product Generation Branch at EMC has begun developing evaluation plans for the final R2O gate, transition to operations. Look for updates at the EMC Users Verification website.

The Workshop Organizing Committee included Tara Jensen (NCAR and DTC), Jason Levit (NOAA/EMC), Geoff Manikin (NOAA/EMC), Jason Otkin (UWisc CIMSS), Mike Baldwin (Purdue University), Dave Turner (NOAA/GSL), Deepthi Achuthavarier (NOAA/OSTI), Jack Settelmaier (NOAA/SRHQ), Burkely Gallo (NOAA/SPC), Linden Wolf (NOAA/OSTI), Sarah Lu (SUNY-Albany), Cristiana Stan (GMU), Yan Xue (OSTI), and Matt Janiga (NRL).

My involvement in model physics development and coordination for operational use, in both Europe and the U.S., has afforded me an interesting and rewarding perspective on the unique philosophies and approaches taken along the path of research to operations on both sides of the Atlantic. In many ways, the challenges faced by large organizations or collaborative consortiums when developing/selecting physics for operations are very similar for Europe and the U.S. For instance, when scientists have invested a significant portion of their careers in their physics development, agreeing on best practices often becomes emotional and personal, and the process of selecting physics for operations can even become territorial. On the other hand, thanks to disparate policies regarding model open-source code, license agreements, and memorandums of understanding (MoU) in European regional model development, the approaches to physics parameterization research to operations (R2O) also diverge significantly between Europe and the US.   

The U.S. has been immensely successful convening academia in numerical weather prediction research in general, and in physics development in particular, thanks to the WRF community. To a large extent, this is also realized in the development of physics parameterizations for the UFS through the introduction of the Common Community Physics Package (CCPP). In Europe, on the other hand, the closed source and various MoUs among participating nations have caused serious challenges involving academia in developing relatively recent versions of model code - in particular, in the regional model community. The European Centre for Medium-Range Weather Forecasts (ECMWF) has been more successful than the European limited-area model community in this regard with the release of the open Integrated Forecast System, IFS; although, this scaled-down academic version still lags behind operational releases. However, the European approach whereby one suite is developed, improved, and continuously refined in close collaboration with the developers of the different components - again drawing the parallel to ECMWF - is compelling, and also notably successful. It works because the research scientists who are developing the cloud microphysics scheme sit next door to the scientists working on convection and Planetary Boundary Layer (PBL) schemes, and they all work in the same building as the operational branch. A similar approach is also seen in the regional model community in Europe, as model physics suites are developed and refined in a collaborative manner within different consortia (e.g. HIRLAM, ALADIN, COSMO), under strict MoUs. In the US, the challenge with open source and community involvement is instead the coordination and organization piece. The know-how needed to combine different physics schemes that interact in complex ways, contributed from all over the world, into a functioning suite suitable for operations pose an array of challenges. In a large organization such as NOAA this coordination does not take place by simply walking down the hall to see your favorite convection-parameterization developer, because schemes are contributed by a wide variety of universities, research labs, and operational centers. Can we combine the best of both worlds? 

Our approach to the UFS R2O project has been to work under a new paradigm in regards to physics development for operations at NOAA. EMC and the OAR labs, as well as DTC, work closely together on a common suite aimed for operations. Some frustration associated with operational implementation is the feeling of always having to “put out fires,” chasing systematic errors, instead of having the liberty of thinking long term to include advanced process descriptions. In this project we are aiming to find a balance between the two, in a two-stream approach addressing both short-term needs and long-term goals. The long-term goal is a seamless (across space and timescales) physics parameterization suite that aims to unify, to the extent possible, processes that represent vertical mixing and cloud formation. The key here is to approach the challenge as a unified representation of physical processes with a well thought out conceptualization of the final suite, and avoid patching together existing schemes as an afterthought. However, we recognize the limitations in scope of the R2O project as such, as funding is only available on a year-to-year basis, and thus hope that this project can be a long-term sustained activity, so that our goals in developing a next-generation physics suite can be realized. Another challenge we are facing is the extremely rapid development of the rest of the coupled model system that constitutes the UFS, such as a new land-surface scheme, coupled ocean, ice and waves, chemistry, stochastic physics, and fully coupled data assimilation. For our project to be successful, it is extremely important that communication and trust is established among the involved parties, and almost a year in, I strongly believe that we are on the right track during this exciting time for NWP in the US.

Evelyn Grell’s story begins in a suburb of Philadelphia, PA where she was raised.  Her interest in weather was inspired from spending time with grandparents on the Jersey shore.  Her grandfather was an avid sailor and even built his own sailboat. Naturally, monitoring the weather and especially the wind direction was essential before setting sail. 

Evelyn never seriously considered meteorology as a career until she was obliged to take a meteorology course in college for her undergraduate degree in Earth and Environmental Sciences. Understanding the ever-changing weather through the lens of physical laws had great appeal.  Riding the momentum of this new passion, she earned a master’s degree in meteorology at The Pennsylvania State University, where she was introduced to numerical weather prediction. This was during the 1980s when an active partnership was in place between Penn State and NCAR in NWP work, thanks to the pioneering work of Richard Anthes, Thomas Warner, Nelson Seaman and others, in developing the Penn State/NCAR model, which evolved into MM3, MM4, and finally MM5.  That model was a predecessor to the Weather Research and Forecasting model (WRF).  Thanks to this connection, she landed a position right out of school, working on a collaborative project between NCAR and NOAA, applying modeling to investigate the potential impact of the new (at the time) wind-profiler network on weather prediction.  

After nine years at NOAA, her meteorological career took a long hiatus when she and her German husband Georg decided to try living in Germany.  Georg promptly found a job there and Evelyn was able to spend time at home with their two young sons.  That was certainly not without challenges, as she knew very little German at first, so she endured some feelings of isolation throughout this period.  However, they were fortunate to live in a spectacularly beautiful area at the foot of the Alps, and eventually made lifelong friends there.  After four years, they decided to return to the Boulder area where Georg accepted an opportunity at NOAA/CIRES.  Evelyn worked in the local elementary school for a couple of years until a meteorology opportunity knocked. 

Currently, Evelyn’s work involves testing and evaluation of various model physics parameterizations in the single-column and regional versions of the UFS model and WRF. Typical tasks include running the model, implementing some changes, then running it again, and analyzing the results to determine the impact of the change.  

Evelyn has divided her time between two different DTC projects.  The HWRF Physics Advancement project focused on evaluating physics changes in the Hurricane WRF model, specifically the convection parameterization. That project was both interesting and challenging because it offered insight into elements of a complicated cycled model system that she had never dealt with before.

She currently belongs to the Physics Across Scales/Planetary Boundary Layer Team, which is assessing the performance of two operational physics suites across spatiotemporal scales in an effort to evaluate whether these suites can be applied across the broad range of scales needed for our operational models.  This team is also examining potential causes of observed biases in the model boundary layer. 

Collaborating on these DTC projects has offered her the opportunity to interact with some exceptionally bright people, and to learn new skills and new approaches for improving operational weather prediction.

Evelyn enjoys hiking, reading, puzzles, cross-country skiing, and she and her husband are looking forward to traveling the world when the pandemic is no longer a concern.

The Hydrometeorological Testbed at NOAA/NWS/NCEP Weather Prediction Center (WPC) is a naturalistic decision-making environment, a physical space, a collaboration space, and an insight-generating laboratory. We explore observations and models (Numerical Weather Prediction, Machine Learning and statistical models) in order to evaluate, validate and verify weather-forecasting procedures, tools, and techniques. 

We recently wrapped up our season long Virtual Winter Weather Experiment (WWE) 2020-2021 during which we evaluated eight experimental Unified Forecast System (UFS) convection-allowing models (CAMs) and one machine learned snow to liquid equivalent technique (in the western US only) for snowfall forecasting using an immersive forecasting activity.  We asked participants to view model information and draw their own forecasts, rank models in a pre and post evaluation survey (both subjectively and objectively), and discuss how such guidance might influence their forecasts or forecast process. Participants enjoyed the immersive forecasting activity and appreciated the opportunity to explore these experimental data sets in a pseudo-operational way. We learned that predictability for the most common events was hit or miss, large-scale predictability, at time scales of 60-84 h, was still uncertain, and CAMs could not correct for this very well. However, the information contained in such forecasts was still useful and could be brought to bear in the forecast process, and thus could be meaningful in Impact Decision Support Services (IDSS). We continued to explore the predictability challenges by designing case studies focused on Days 3 and 2 in our retrospective, intensive forecasting sessions. This aspect of the forecast process was also considered valuable because we can begin to explore notions of forecast consistency between model cycles and interactions with the forecast strategies, processes, and procedures in future experiments. For more information on the WWE, contact Dr. Kirstin Harnos (kirstin.harnos at noaa.gov).

A large part of our success comes from the participation of a large number of NWS Weather Forecast Offices, regional centers, Environmental Modeling Center, Physical Sciences Laboratory, and our academic partners as shown below:

Our  immersive forecasting activities will continue into the warm season for our Virtual Flash Flood and Intensive Rainfall Experiment (FFaIR). We will continue to utilize CAMs provided by the Center for the Analysis and Prediction of Storms, Environmental Modeling Center, and the Global Systems Laboratory for the purpose of detecting and forecasting heavy and significant precipitation that may lead to flash flooding. We will do so through an operational product lens (i.e., Excessive Rainfall Outlook) and a hybrid forecast product for 6-hour rainfall, which bridges the traditional Quantitative Precipitation Forecast guidance from WPC with the Mesoscale Precipitation Discussion product. We will forecast rainfall accumulations, rainfall rates, durations, and flooding in the Day 1 period synthesizing many operational and experimental deterministic and ensemble CAM systems. We will continue to extract meaningful information from these systems under a variety of real-time forecasting scenarios during the peak of the warm season. For more information on FFaIR, contact Dr. Sarah Trojniak (sarah.trojniak at noaa.gov).

We are looking forward to expanding the breadth of our knowledge as we seek collaboration with the social-science community. The information we produce informs the public we serve, from the methods we employ to solve physical science problems, to how we equip and prepare forecasters. Only through many different perspectives can we hope to capture a wide-angle view of forecast challenges to improve the predictions of precipitation that empower all of us to save lives and protect property.

We encourage researchers, forecasters, and emergency-support function personnel to reach out so we can work together and appreciate each other's challenges to better apply our various sciences, techniques, and approaches to empower life-saving and protective action against hazards, local and national. 

1. NOAA/NWS/NCEP Weather Prediction Center
2. Cooperative Institute for Research in Environmental Sciences (CIRES), University of Colorado Boulder

My projects with the DTC Visitor Program have been high points in my 30+ year career as a computational scientist interested in atmospheric sciences.  In 1986 as a Staff Sergeant in the United States Air Force, I discovered my deep interest and aptitude in computer science, and uncovered a new fascination with science, especially the geophysical kind. It inspired me to chart a course towards finally finishing my BS degree and explore graduate school with the notion of applying supercomputing to atmospheric science. From the beginning, I had this view of NCAR as a highly respected institution, but wasn’t sure I ever really envisioned myself spending time there.

Fourteen years later, my first DTC visit was 2010-2011; my research focused on the enhancement of prototype HRRR Alaska NWP forecasts we had been performing at the University of Alaska’s Arctic Region Supercomputing Center (ARSC). We had been running the HRRR-AK multiple times per day, and identified a need for tools that would allow us to build custom products to compare forecasts with observations using the Model Evaluation Tools (MET) and Gridpoint Statistical Interpolation (GSI) software for data assimilation. For a deeper dive on this project, take a look at the report. 

The second DTC project grew its roots during an enlightening meeting I had with Geoff DiMego, then Chief, Mesoscale Modeling Branch at NOAA, in 2009.  During this time, I learned of the evolving NOAA Environmental Modeling System / Nonhydrostatic Multiscale Model on the B-grid (NEMS-NMMB) development and deployment in the NCEP operational environments, and became curious  about how we might apply this research to our own Alaska weather-modeling efforts. I teamed up with Dr. Dèlia Arnold, a scientific consultant with Vienna, Austria’s Zentralanstalt für Meteorologie und Geodynamik (ZAMG). The primary focus for our project was to explore the deployment of NEMS/NMMB to a non-NCEP environment for potential future community use.  

The great challenges were adapting the NEMS/NMMB — which had been explored primarily on NOAA systems using Intel compilers — to a broader collection of computing platforms and regional model domains. Through a concerted  effort, we were able to port the system to Gnu-based Linux environments ranging from typical workstations to Cray supercomputers, and NCAR’s Yellowstone. We also explored the creation of regional simulations over Alaska and Catalonia and launched a temporary NEMS/NMMB Alaska region real-time forecast system — using many of our existing HRRR-AK workflow programs to drive this — on the Cray XK6m at the University of Alaska Fairbanks Arctic Region Supercomputing Center. If you’re curious about how this project moved forward, read the story here.  

In 2020, I commenced my current DTC project, which in many ways, has roots dating back to the turn of the century when I started collaborating with “The Grid” community.  Of particular interest for me back then was the potential of The Grid to make complex and computationally intensive models available to Joe or Jane Scientist, while avoiding  the complexities of command lines, operating systems, etc. The vision was that users would go to a web page, specify parameters, etc. through an intuitive GUI, and launch the job, not necessarily knowing or caring how and where the model was actually being executed.  

The vision is ambitious, and the specific outcomes for the DTC project include the development of low-level command-line tools that will allow users to create custom and complex NWP workflows by using the DTC NWP Docker containers as loosely-coupled independent software services deployed in the Amazon Cloud.

So, I find myself approaching the end of a mostly enjoyable and fascinating career, having had the wonderful opportunity to serve DTC and the wider NWP community to make some inroads toward realizing this vision.  The project is ongoing, and a recent status update was presented at the 2021 UCAR Software Engineering Assembly’s Improving Scientific Software Conference.

After more than thirty years in the academic world, Don currently spends his time living in Interior Alaska as owner/manager of the single-member LLC, Boreal Scientific Computing, pursuing research and development activities.

The “Model Uncertainty Model Intercomparison Project” (MUMIP) is an international effort to better understand model-physics uncertainty, and how to represent it in stochastic physical parameterizations. After all, physical parameterizations provide an approximate solution to physical processes occurring in a grid-box and are, as such, a source of forecast model uncertainty due to a large variety of factors, e.g. unresolved subgrid-scale variability treated as a grid-box mean, unknown parameter values, physical processes which have been excluded, structural errors, incomplete calculations of processes or inherent process uncertainty.

MUMIP is a joint project of the WMO Working Group on Numerical Experimentation (WGNE) and the World Weather Research Program (WWRP) Working Group on Probability, Dynamics and Ensemble Forecasting (PDEF), working groups dedicated to the development of Earth system models for use in weather, climate, water and environmental prediction on all time scales, and diagnosing and resolving model shortcomings and uncertainties. Scientists from a number of national and international centers, including University of Oxford, University of Reading, the UK MetOffice, the European Centre for Medium-Range Weather Forecasts (ECMWF), Météo-France, Deutscher Wetterdienst (DWD, German Weather Service), NOAA’s Physical Sciences Laboratory (PSL), NCAR’s  Mesoscale and Microscale Meteorology Laboratory (MMM), and now the Developmental Testbed Center (DTC), will be contributing to this project.

While stochastic physics schemes are often tuned using ad-hoc methods, objective methods derived from physical constraints can be used to better inform the development and improvement of schemes, which is the focus of the MUMIP project. The uncertainty in parameterizations may be addressed by stochastic methods, which aim to select a random state consistent with the resolved state. The objective methods in MUMIP then inform the development of deterministic and stochastic schemes, i.e. comparing state variables and tendencies in a convection-permitting high-resolution model simulation against a lower-resolution parameterized-convection model simulation. This is done by “coarse-graining” a high-resolution simulation (i.e. computing spatio-temporal averages) onto a grid of a lower-resolution simulation.  The premise here is when the parameterizations work perfectly, the statistics of the state variables in the coarse-grained higher-resolution simulation should match those of parameterized lower-resolution simulations. In reality, however, discrepancies are often discovered when performing this type of comparison, where such discrepancies can then be used to improve the physical parameterizations. Additionally, the high-resolution distribution offers useful information about the subgrid-scale uncertainty that helps to objectively inform stochastic parameterizations.

MUMIP participants will run an array of approximately 40,000 Single Column Model (SCM) simulations forced by coarse-grained high-resolution model output (Figure 1), initially from the DWD ICON (3-km) model. In what is planned to be a 3-year project, the DTC will use the Common Community Physics Package (CCPP) SCM, as well as a coarse-grained 3-km NOAA Unified Forecast System (UFS) simulation. In order to use this forcing data, the SCMs will ingest forcing fields using the DEPHY format, a standard agreed upon at the International Workshop for SCM/LES comparisons organized and hosted by Météo-France in June 2020. The use of the DEPHY format, which has already been implemented in the CCPP SCM, is key for complementary initiatives towards improvement and tuning of model physics.  In April 2021, DTC staff participated in the workshop on the Improvement and Calibration of Clouds in Models organized by Météo-France, where the focus was to discuss and share the latest improvements in parameterizations. In summary, there are a number of ongoing multi-institutional initiatives related to SCMs and their role in hierarchical model development, and the DTC is taking advantage of these collaborations to pursue improvements in model physics.

Figure: High-resolution model output (small grids) is coarse grained and mapped to grid (large boxes) to provide column forcing to drive an array of CCPP SCMs.

The latest Unified Forecasting System (UFS) application to be released is the Short-Range Weather (SRW) Application, which targets predictions of atmospheric behavior on a limited spatial domain and on time scales from less than an hour out to several days.

The SRW Application version 1.0 includes a prognostic atmospheric model, pre- and post-processing, and a community workflow for running the system end-to-end. Specific configurations of the release (e.g. model resolutions, domain location, and physics options) are documented and supported through an online forum.  

The release is available through the GitHub repository.

Future work and releases will include:

• A more extensive set of supported developmental physics suites.
• A larger number of predefined domains/resolutions and a fully supported capability to create a user-defined domain.
• Inclusion of data assimilation capability.
• A verification package integrated into the workflow.
• Stochastic perturbations techniques.