With wildfires increasingly impacting society and ecosystems spanning local to global scales, NOAA has expanded its strategic investments and plans to address wildfire-related hazards. In 2023, NOAA established a new Fire Weather Testbed (FWT), a joint effort between three of NOAA’s Line Offices: NOAA Research (Oceanic and Atmospheric Research), NOAA Satellites (National Environmental Satellite, Data, and Information Service), and the National Weather Service. Housed within NOAA’s Global Systems Laboratory in Boulder, Colorado, the FWT will comprise physical and virtual facilities for conducting evaluations and experiments to facilitate the transfer of new technologies and applications into operational platforms as quickly as possible. 

The FWT will convene a broad range of fire-related communities across agencies and jurisdictions, including decision makers, researchers, and operational fire weather forecasters to improve and tailor tools, applications, products, and information. A primary focus centers on understanding the information needs of stakeholders involved with wildland fire before, during, and after the flames to create a fire-ready nation: one that is fire-adapted and fire-resilient.

One of the unique goals of the FWT will be a thorough “User Needs Assessment” of the fire-weather community to better understand the needs and gaps impacting the front-line firefighters and decision makers across the many timelines (instantaneous to seasonal) pertinent to fire management. The FWT is also hiring multiple Social and Behavioral Scientists to ensure that relevant social science is incorporated throughout the evaluation process, from designing and facilitating evaluations to ensuring user feedback is incorporated into new tools and technologies. Social science will guide the development of new tools so that they are effective in helping operational decision-makers understand often complex weather forecasts to make well-informed decisions amidst uncertainties typical in wildland fire situations.  

One of the many significant challenges to improving fire-weather forecasts is that wildland fires often begin in complex terrain with very sparse data observations. This combination of mountainous geography, varying vegetation types, and lack of observations makes it difficult to observe, much less accurately model and skillfully forecast, weather near wildfires. To successfully model weather in these complex environments, we need high-quality data and verification. The FWT will work closely with other NOAA Testbeds and Proving Grounds to facilitate the evaluation and transition of state-of-the-art wildfire weather-related products and tools..

Fire weather forecasting includes many aspects requiring verification. Upper left: Fire behavior, spread, and emissions are dependent on local and regional weather including humidity, temperature, wind, radiation, vegetation (fuel), and topography. Upper right: Skillful forecasts of upslope winds (flow from left to right), transport winds (flow from right to left), and mixing height forecasts are critical to estimate dispersion and transport from wildland fires. Lower left: New ensemble forecasts, such as the Warn-On-Forecast Smoke system developed by the National Severe Storms Laboratory, can provide probabilistic guidance for values-at-risk, such as the location of a parade (X) that has a 50% chance of experiencing problematic ground-level smoke concentrations. Lower right: Verification of forecasts using remotely sensed data (inset shows the Suomi NPP Aerosol Index) and in situ observations (main) will improve our ability to predict and communicate impactful fire weather-related conditions that ultimately protect life and property.

The FWT and the Developmental Testbed Center (DTC) will collaborate to ensure that the most advanced verification methods and techniques and technologies are being tested and implemented in fire weather-based forecasting within the Unified Forecast System (UFS).  For example, the DTC is currently working on adding verification capabilities to the enhanced Model Evaluation Tools (METplus) software system, as well as testing methods within the Short-Range Weather App. The DTC will build on antecedent air quality evaluation efforts to include the ability to assess the Rapid Refresh Forecast System (RRFS)-Smoke forecasts of PM2.5, PM10, and aerosol optical depth forecasts within METplus.  Once verification of these fields is finalized, METplus use-cases, or examples, will be added to the SRW App for future use by the general modeling community, including the FWT. Tools such as the Method for Object-based Diagnostics Evaluation (MODE) will be used by the Short Range, Medium Range, Subseasonal to Seasonal (S2S), and the Seasonal Forecast System (SFS) to evaluate precursor fields such as precipitation, and atmospheric moisture.  It will also help evaluate fire parameters such as spread and impact of short term atmospheric phenomena.

The DTC provided two training series on how to configure METplus for fire weather in November 2023.  The recordings and presentations can be found on the DTC website.

The U.S. Air Force’s partnership with the DTC dates back to 2003 and one of the main areas of collaboration has been the advancement of scientific and technical research into a community-backed set of tools for model verification known as Model Evaluation Tools (MET) and its subsequent evolution into the METplus suite of tools. The USAF has provided financial as well as in-kind participation in governance to the DTC and in return benefits from the expansion of this tool set across many operational numerical weather centers from the UK Met Office and its partners, NOAA/NWS, and others.

The USAF operates both global and regional atmospheric models, including deterministic and probabilistic solutions, leveraging external collaborations and state-of-the-science solutions. The products that are produced must be verified and the results communicated effectively to Air Force leadership and to the downstream warfighters generating actionable insights daily. Over the last few years, the 557th Weather Wing (the Air Force’s operational modeling center) has moved its modeling operations to a new high performance computing environment at Oak Ridge National Lab and is now leveraging the METplus framework on that system. Previous instantiations of operational model verification involved a series of scheduled calls to older versions of MET via many scripts resulting in a cluttered repository and hard to trace orchestration. In the new environment, a Development Security Operations approach using a Continuous Integration and Continuous Deployment (CI/CD) model has been employed for all software development, including these new Model Verification Tools (MVTK).

The Cylc workflow engine used by the 557 WW, UK Met Office and their partners is an orchestration tool for modeling which can initiate jobs based on events or at specific times. Using Cylc as the backbone for MVTK allows for routine triggering of the necessary observation and modeling pre-processing METplus jobs upon file receipt. Subsequently, the main statistics modules contained in MET are run and the resultant output aggregate files are sent to a shared file repository. Figures are generated using home-grown python scripts and are scheduled in Cylc at desired intervals, including near real-time and monthly/seasonal aggregation. In future operations, containerized versions of METplus will become the standard for upgrading our model verification software stack across multiple computing platforms and a streamlined end-to-end approach leveraging METplus analysis tools and METviewer. The USAF is looking forward to continuing to work with DTC to benefit from enhancements to METplus.

Our previous partnerships with DTC can be viewed as three pronged: support to 557 WW users and their requests for capabilities (described above), cybersecurity, and veracity testing. Creating a cybersecure METplus that can be run in the Air Force computing environments has proven to be a big lift, but we hope that the hard work of DTC personnel will benefit all users of METplus with a more secure codebase. While the 557 WW continuously verifies operationally running models, our acquisition corps, planners, and leadership often want to know what benefit new innovations are bringing to the Air Force Weather Enterprise – this is where veracity testing by DTC plays a role. In the past we have exercised options to evaluate new methods for forecasting dust and flooding. We are very excited to include all three activities from our previous support to DTC in our next contract, covering fiscal years 2024-26.

Dustin Swales works for NOAA-GSL where his primary role is leading the development of the CCPP Framework, the physics coupling infrastructure used by NOAA UFS applications. The Framework is used by a wide range of users beyond NOAA, and so a significant piece of this role requires working closely with collaborators at NCAR to develop this Framework across modeling applications. Tangential to this, he works on CCPP Physics development, the physical parameterizations enabled by the Framework. This involves working closely with physics developers, both internal and external to NOAA, to incorporate their advancements into NOAA operations.

He hails from Pittston, Pennsylvania, near Scranton where he grew up with a slightly older brother, who now lives in Colorado with his family as well. From an early age, he enjoyed spending time outdoors exploring. Fortunately his family home offered ample outdoor recreational opportunities nearby, whether this meant exploring old-growth forests in the Pinchot State Forest, searching (often unsuccessfully) for wild trout, or on a mission to harvest enough wild berries to bake muffins.

Dustin attended Penn State, and randomly picked Meteorology as a major without giving it too much consideration. It turned out to be a very good fit! He’d always enjoyed math and the physical sciences, so studying the atmospheric sciences satisfied his innate curiosities. He received his B.S in 2007 and M.S in 2009. As a graduate student, he built a shallow water model to explore near-frontal surface wind patterns observed by space-borne radar. At some point during that period, he became inspired to learn about computational methods and scientific programming. The first job he landed was at NOAA/NESDIS in Camp Springs MD, where he worked on algorithm development for ozone-observing instruments for a NOAA mission.

As it is for many of our staff, Dustin and his wife were always drawn to Colorado, and after three years of discussion, they finally made the move. His wife Amanda took a job teaching in Denver Public Schools and still works at the same school today. Dustin took a position at NOAA/PSL (CIRES). While working there, he split his time between two groups, enabling him to work on a variety of often disparate projects. With one group, he studied atmospheric rivers (ARs) and the pathways by which moisture is transported through the complex terrain of the Intermountain West, a region for which the stakeholders had a great deal of interest, given ARs’ hydrological impact on water resources. Concurrently, his work with the other group focused primarily on physical parameterization development, mostly relating to radiation and clouds. After 10 years at PSL/CIRES, he moved to NCAR/RAL to work on development for the CCPP Physics and CCPP Single Column Mode as part of the DTC. In 2023, he moved to NOAA/GSL, still with the DTC, to lead the development effort of the CCPP Framework.

A typical day for Dustin starts by waking up early with his 7-month old son, Dean. Currently he divides his time working on a few DTC projects along with other NOAA/GSL projects. On the DTC side, his main focus is unifying the development efforts for the CCPP Framework occurring at both NOAA and NCAR. Outside of the Framework, he also works on CCPP Physics development, lately with a focus toward making the library of parameterizations more interoperable. Outside of the DTC, he works on other CCPP-related projects, such as using Machine Learning algorithms in the CCPP and enabling the use of GPUs in the CCPP.

Where’s the reward for him? He feels many of his projects lie in the research-to-operations realm or related to operational development and support, and it’s deeply rewarding to support this mission. Building better software (hopefully) makes it easier for scientists to innovate, which in turn makes better forecasts.

During the first few years of the DTC, the United States Air Force (USAF) put forth a vision for the DTC to develop a suite for verification tools that would provide reproducible results such that statistics and metrics could be shared across institutions.  The USAF envisioned a suite of tools that would include both standard verification approaches, as well as advanced diagnostic methodologies.  In 2008, the DTC released the software package, Model Evaluation Tools (MET) to the community with the intent to provide state-of-the-science verification and diagnostic tools to the community. Over the past 15 years, an enhanced version of MET, called METplus, has become a software system collaboratively developed by the Developmental Testbed Center (DTC) and community partners.

Several operational entities are nearing completion of their research-to-operations process (R2O) to include METplus in their operational verification systems.  NOAA’s Environmental Modeling Center (EMC) will be the first to make the transition to a system built using METplus. The first version of the EMC Verification System (EVS v1), which employs METplus to generate real-time model verification statistics, will become operational in fiscal year (FY) 2024. EMC, DTC, and the National Centers for Environmental Prediction (NCEP) Central Operations (NCO) collaborated  from FY2022 to the present to install METplus on the Weather and Climate Operational Supercomputer System (WCOSS2), working through software library issues and strict installation standards to ensure METplus meets rigorous WCOSS2 security and installation standards.

Through highly collaborative meetings and frequent communication, EMC and DTC developed a suite of best practices to streamline the METplus installation process with NCO on WCOSS2, drastically reducing the time from the METplus software release to operational capability (e.g., months to days). Ultimately, METplus allows EVS v1 to publish plots for most real-time EMC environmental forecast models to a web page, enabling anyone with internet access the ability to determine how EMC models are currently performing.

EVS version 2 is planned to be an extension of EVS v1 with additional metrics and capability. Development of the EVS v2 system will begin shortly after version 1 becomes operational. The current development plan includes options for the system to run outside of WCOSS2, perhaps in a cloud environment or on other machines.

The METplus governance partners are also developing operational systems powered by METplus. Besides a representative from NOAA labs and centers, the governance partners include NSF NCAR, USAF, United Kingdom Met Office (UKMO), Naval Research Laboratory (NRL), and Australia’s Bureau of Meteorology (BoM).  A detailed description of how METplus is being incorporated into the USAF evaluation system is described in the “Director’s Corner” of this newsletter. The USAF plans for METplus to become their cybersecure verification software stack that provides the standard for upgrading their models.

The Met Office made the decision to adopt a MET-based system for its operational verification replacement back in 2019. A first complete instantiation of the current operational system capability will go into parallel testing ahead of operational implementation in mid-2024. The long-term plan is for Met Office scientists and software engineers to become integral members of the open-source development community beyond the DTC. To date, code contributions have been small but significant (e.g. the first inclusion of OpenMP) as have the contributions as subject-matter expertise and beta testers for seven coordinated METplus releases.

Both NRL and Australia's BoM are in the beginning phases of testing METplus for their operational systems. NRL has worked both collaboratively and via funded projects with the NCAR node of DTC to develop support for their ensemble, atmospheric composition, and data assimilation systems. The BoM has been providing in-kind testing and code-management capability to all aspects of METplus to reach their near-term goal of a quick transition of METplus to operational use. In summary, it has taken 15 years, but the vision of a framework of consistent verification tools for the community is becoming more attainable as each institution adopts METplus for both operational use and system development.

Cristiana Stan of George Mason University originally proposed a very exciting and new type of DTC visitor project that involved holding a “hack-a-thon” where teams of two graduate students would be tasked with selecting and developing a METplus Use Case for one of the subseasonal to seasonal (S2S) metrics identified during the 2021 DTC UFS Evaluation Metrics Workshop.  The winning team would then be given the opportunity to visit the DTC for up to three months to collaborate with the METplus team on continuing to integrate additional S2S diagnostics into METplus. Unfortunately, there was insufficient interest from students to make the hack-a-thon work.  With an eye towards maintaining the original goal of increasing student engagement and experience with METplus, the DTC worked with professor Stan to modify the scope of her  project. At the time, Cristiana was already working to create a METplus Use Case for a metric  developed by her research group to monitor the El Niño Southern Oscillation that would be implemented at NOAA’s Climate Prediction Center as part of a Test Bed project. The scope of the DTC visitor project was to expand the existing use case and make it applicable to forecast data. 

As a result, both Cristiana Stan and her graduate student, Loren Doyle, visited the DTC in 2023. Loren visited during the Spring and Summer and Cristiana during the Summer.  They  met with Danial Adriaansen, John Halley Gotway, Christiana Kalb, and John Opatz to discuss implementation of METplus with two metrics designed to evaluate the relationship between the Madden Julian Oscillation and the El Niño Southern Oscillation.  A previous use case creating these metrics using METplus had already been added to the METplus repository, but input data needed to shift to UFS data, which created new obstacles. Additionally, Cristiana and Loren were able to review the ongoing work to improve METplus’ usability through a focus group and provide their own experience with METplus and opportunities for improvement with the unique perspective of research development.

While the visit included some hybrid meetings, the in-person meetings benefited from the diagrams drawn with the dry erase markers on the ‘traditional’ white boards. For example, the subseasonal to seasonal (S2S) forecast systems use different strategies for initializing the forecasts. The initial conditions vary from a particular date to a particular day of the week and/or from a few times a week to every day of the week. METplus diagnostics must be flexible to accommodate datasets with different structures as well as sizes and being able to visualize these structures was helpful for adapting existing capabilities in METplus and informing developers of the future developments.

The Common Community Physics Package (CCPP) Visioning Workshop, which took place virtually on August 15-17 of 2023, convened discussions focused on the current status and future direction of the CCPP. The workshop was organized by a multi-institutional committee composed of representatives from DTC (Grant Firl, Lulin Xue, Dustin Swales, Ligia Bernardet), NCAR’s Mesoscale and Microscale Meteorology (Laura Fowler) and Climate and Global Dynamics (Courtney Peverley) Laboratories, University of Oklahoma Center for Analysis and Prediction of Storms (Ming Xue), and NOAA’s Environmental Modeling Center (Fanglin Yang). Participants from four NCAR labs and six NOAA labs and centers comprised approximately two-thirds of the participants. The remaining participants were from the U.S. Department of Energy Pacific Northwest National Laboratory, the Joint Center for Satellite Data Assimilation, the United States Naval Research Laboratory, the Brazilian National Institute for Space Research, the Stevens Institute of Technology, the University of Maryland, the Central University of Rajasthan, and the Norwegian Meteorological Institute.

Workshop participants recognized that the CCPP is now a mature product, used operationally at NOAA as part of the Hurricane Analysis and Forecast System v1, slated for all future Unified Forecast System (UFS) application implementations, and in different stages of integration within models under the purview of the U.S. Navy Research Laboratory and the National Center for Atmospheric Research. The growth of the CCPP community, the ongoing progress in Earth System Model science, and the advancements in computational technology demand continuous CCPP development.

High-priority items identified by participants that are needed to facilitate exchange and enable collaborative development are: unification of the CCPP Framework development being conducted at NOAA and at NCAR, generalization of existing parameterizations for higher interoperability, and development of a vision for how the multiple sets of CCPP-compliant physics will be managed and served to the community.

Additional recommendations drawn from the workshop can be classified in two categories. The first one is the improvement of existing practices. Highlights include changes in the directory structure of the CCPP Physics repository for ease of use, issuing additional tags to record important code snapshots (such as those associated with UFS prototypes for upcoming operational implementations), establishing a formal governance for the repository of CCPP Standard Names, and documenting the limitations of schemes and suites regarding the scales and processes they were developed for and tested on.

The second category is development needed to be prepared for the future. Highlights include continuing engagement with the aerosol and chemistry community to devise optimal solutions for the interface between atmospheric chemistry and host models, looking ahead toward three-dimensional physics, developing a module for common atmospheric physics functions (such as saturation vapor pressure) to ensure greater consistency amongst schemes within a suite, and ability for all schemes to return tendencies to enable studies in physics-dynamics coupling. On the computational front, the use of fine-grain platforms with graphical processing unit (GPU) compute architectures has become a priority. 

In summary, the cross-institutional engagement with the CCPP and the openness of the community to collaborate on physics development were evident in this workshop. Next steps include the submission of a meeting summary to the Bulletin of the American Meteorological Society and the prioritization of development topics.

Upcoming Events

Attending AGU or AMS and interested in learning more about what the DTC staff are up to?  Check out these presentations!

2023 AGU Annual Meeting

2024 AMS Annual Meeting 

Internship Opportunities

The NCAR Earth System Science Internship (NESSI), hosted by NCAR Education, Engagement & Early-Career Development (EdEC), offers students interested in the Earth system sciences an opportunity to conduct research with NCAR scientists from across all labs on a wide range of topics including but not limited to atmospheric science, computational science, engineering, and solar & space physics. The program is designed to support students and promote Earth system science through research, mentoring, and community building. 

NESSI is accepting applications for the 2024 Summer internships through January 6, 2024. Find more details on the FAQ page.

Significant Opportunities in Atmospheric Research and Science (SOARS) is an undergraduate to graduate program built around a summer research internship, mentoring by top scientists and engineers, and a supportive learning community. In addition to the summer internship, SOARS includes year-round support, funding to attend conferences and last-dollar tuition scholarships. Successful protégés are eligible to participate in the program for up to four years. SOARS is now accepting applications for its 2024 program through February 1, 2024. Apply here.