The DTC community mourns the passing of Tara Jensen this past September. Tara joined the DTC in December of 2008, eleven months after the initial release of the Model Evaluation Tools (MET), a verification software package developed and supported by the DTC. Over the next six years, Tara contributed to a variety of DTC projects, ranging from collaborations with two other NOAA Testbeds (i.e., the Hydrometeorology and Hazardous Weather Testbeds) to internal projects focused on mesoscale and ensemble model evaluations. Central to all of these activities was the application of MET and demonstration of the utility of its advanced verification methods, including the Method for Object-based Diagnostic Evaluation (MODE). From the beginning, Tara demonstrated her passion for significantly impacting the outcome of each project and developing into a verification expert.

In 2014, Tara became the lead for the DTC’s Verification Task, which was charged with the continued development and support of MET. Initial development of MET was supported by the United States Air Force (USAF), with some additional support from NSF NCAR and NOAA, but at that time MET was primarily used internally by the DTC. With encouragement from Bill Lapenta, who was the Director of the National Centers for Environmental Prediction (NCEP), Tara promoted the adoption of MET by the NOAA Environmental Modeling Center (EMC) and then more broadly within NOAA. Tara’s enthusiasm and dedication quickly led to greater visibility of MET and wider interest in this community software package. Under Tara’s leadership, MET became the statistical engine of the larger enhanced Model Evaluation Tools (METplus) system. With MET at its core, METplus also contains low-level Python-based automation scripts, Python-based analysis and plotting tools, interactive visualization software, scientifically-relevant use cases, and extensive documentation. METplus is the cornerstone of DTC testing and evaluation projects. In addition, DTC partners have adopted METplus to power their operational verification systems, including NOAA with their EMC Verification System (EVS) and within the USAF’s 557th Weather Wing.

Engagement with key players within NCEP was essential to attaining NOAA’s buy-in for METplus. Members of EMC’s verification team attest that it was Tara’s vision for the entire weather enterprise, along with support from Tara and the METplus team, that helped EMC pivot from home-grown verification codes to METplus’ “masterfully written code”. Now that the EMC Verification System (EVS), which has METplus at its core, has been implemented in operations, METplus is playing a critical role in the model evaluation and upgrade process at NOAA. Her EMC colleagues commented on how they will miss her joyous and optimistic spirit and will always remember her kindness, leadership, commitment and passion for work.

Tara’s contributions to our community have been acknowledged through well-deserved awards.  Under Tara’s leadership, the MET and METplus teams won UCAR’s Scientific and/or Technical Advancement in 2010 and 2022, respectively. In 2024, Tara received the American Meteorological Society (AMS) Scientific and Technological Activities Commision (STAC) Outstanding Service Award from the AMS Probability and Statistics Committee for her vast contributions to statistical verification tools in operational meteorology and exhibiting exceptional dedication, expertise, and mentorship. Tara also received the Performance Recognition for Outstanding DTC (PROUD) Achievements Award for her visionary leadership and her relentless dedication to the success of METplus in August 2024.

Tara’s impacts on our community are far and wide and she will be dearly missed!

In May 2024, the DTC lost a major contributor and developer. Randy Bullock was a brilliant mathematician and software developer who was instrumental in the creation of tools used by countless DTC visitors and staff, as well as the worldwide community of METplus users. The DTC would like to note our sadness in losing Randy as a colleague and simultaneously celebrate Randy’s contributions, which were extensive. 

One of Randy’s most important accomplishments was development of the MODE (Method for Object-based Diagnostic Evaluation) approach for forecast evaluation, a major component of METplus. This was one of the first spatial forecast-verification methods developed to evaluate gridded forecasts. MODE (and some other spatial methods) were designed to prevent the “double penalty” problem when evaluating predictions of spatially coherent phenomena such as precipitation and clouds. The double penalty arises with traditional verification methods, in which a forecast at a grid point can be counted as both a miss and a false alarm if the forecast feature is displaced from the observed feature. Instead of considering direct grid-to-grid comparisons, MODE provides a mechanism for evaluating and comparing physical characteristics (e.g., size, intensity, location) of the forecast and observed phenomena (e.g., of precipitation) and measuring the displacement of the objects and comparisons of their characteristics (e.g., location, intensity, size). MODE-TD (MODE Time Domain), an extension of MODE, evaluates  three-dimensional features (e.g., x- and y- locations, across time). MODE was also recently enhanced to define and compare objects using multiple input fields. These innovative capabilities are used to compare spatial features in gridded NWP model data and observations, and have been widely applied, including to other weather phenomena.

Randy’s technical expertise was legendary and essential to DTC activities beyond the world of MODE. He was the local expert who supported different model grids and map projections in the vx_grid library, and he greatly enjoyed working out the math behind grid conversion functions. After retiring, he helped the METplus team begin the transition from home-grown projection code to the more widely used PROJ projection library. Randy's vx_regrid library is an extension of his expertise with grids that enabled MET to automatically regrid data on the fly. From MET's first release in 2007 to his retirement, Randy either wrote that code, mentored other engineers to learn it, or helped the team fix his code when we broke it!

Altogether Randy contributed over 470,000 real lines of code spread over nearly 1,869 files within the MET repository.  He authored and supported his own configuration file language (in the vx_config library) that is still used, and was a local expert in data formats, from GRIB1 to GIS shape files, and beyond.  He was generous with his knowledge, had incredible patience, and was able to guide colleagues to resolutions for many messy problems. In recent years, he spearheaded Python embedding in MET, enabling the integration of data from Python into MET. Many more of his accomplishments (e.g., development of extensive postscript libraries) also had major impacts on the DTC and well beyond.

Randy Bullock’s software development, generosity, and work will undoubtedly continue to benefit students, researchers, and operations as long as the METplus system continues to exist. He is greatly missed.

Randy was a founding member of the METplus team and the primary developer of the Method for Object-based Diagnostic Evaluation (MODE) tool that has become so incredibly popular within the verification community. Randy retired in 2020 just as COVID was settling in and causing chaos in our world. Once the doors to NCAR opened again, he came back to the office as a casual employee to work through problems when the team needed his support and expertise. We lost Randy and all of the talent he brought to our mission on May 27, 2024.

Randy grew up in Wadena, Minnesota and graduated from Jamestown High School in 1976. He attended North Dakota State University where he earned a BS in Mathematics. He went on to teach calculus at the University of Wisconsin while earning his Master’s Degree. He relocated to Boulder, Colorado to  pursue a PhD in Mathematics. He joined NSF NCAR in 1991 and worked as a software engineer for 33 years. 

Randy made vast and numerous contributions to the scientific community, and his perseverance and tenacity for problem solving were legendary. As a brilliant mathematician and software developer, he was instrumental in the creation and support of tools (e.g., MODE; Method for Object-based Diagnostic Evaluation) used by countless DTC visitors and staff, as well as the worldwide community of METplus users.  Randy was known for his ability to explain things in an easily digestible way and humorous slides in his METplus tutorial presentations.

Randy led an extraordinary life outside of his job. One of his hobbies was astronomy. He wrote a beautiful program that enabled the user to view any solar eclipse from any spatial perspective (e.g., from the moon or any other point in space). He was a gifted guitarist as well. Randy would joke that his favorite "sport" was reading. Reading was a passion that began at a very young age and this love for reading blossomed into a library of books that graced his home and his office.

Randy is remembered as so very much more than his accomplishments or hobbies, though. He had a quirky, mischievous sense of humor that manifested in practical jokes and clever conversations. He took advantage of a coworker’s fear of snakes by luring them to his office with an offer of Halloween candy from the skull on his desk (another Randy staple). A huge fake snake placed at the doorway scared the wits out of this coworker, sending a broad smile across Randy’s face. Another office feature was his Underdog lunchbox on the window shelf. The entire office exuded his mirthful personality.

He was characterized by so many personality traits from stubborn, to generous, to warm, to intractable. Staff and students raved about his ability to present complex information with humor and his unique explanations that enabled anyone to grasp and appreciate the material. He was widely appreciated as a patient teacher and mentor, always finding the time to help staff solve bugs and problems. For Randy, creation was its own reward. He had created his own documentation formatting software and used it for all of his own documentation. Staff raved it was one of the best documentation softwares they’d ever known.

Perhaps the best way to remember Randy was that he closed many one-on-one conversations by asking “Anything else, as long as I’m here?” Yes Randy, we do wish you were still here.

Over the sixteen years the Method for Object-Based Diagnostic Evaluation (MODE) tool has been available within the Model Evaluation Tools (MET), operational forecasting centers have explored its utility for informing the model development process and evaluating operational products in meaningful ways. Through this exploration, MODE has provided insightful information for operational implementation decisions and meaningful data about the performance of operational products.

MODE has been applied to many different phenomena; here we describe applications at NCEP’s Weather Prediction Center (WPC). Building on the relationship between DTC and WPC, established through collaborations related to NOAA’s Hydrometeorology Testbed (HMT), WPC and DTC staff worked closely together to incorporate advanced verification techniques available in the enhanced Model Evaluation Tools (METplus) into their verification workflow.  Through this deep collaboration, WPC began publicly displaying real-time verification using MODE in the early 2010s.  The initial application to quantitative precipitation forecasts (QPF) has matured over the years to include snowfall and more advanced diagnostics.  WPC provides publically available real-time displays of its object-based QPF and snowfall verification, and MODE continues to be a key tool for WPC’s HMT Flash Flood and Intense Rainfall Experiment (FFaIR) and Winter Weather Experiment (WWE). After the capability to track objects through time was added, WPC used the MODE Time Domain tool to construct a number of products that are used for real-time model analysis to communicate the uncertainty in the predictions. Focusing on heavy precipitation events, these tools analyze objects on both hourly and sub-hourly timescales to evaluate the positioning, size, and magnitude of snowband objects in real time.  WPC also uses MODE internally for unique verification. These applications include determining displacement distance of predictions by the WPC, National Blended Model (NBM), and Hurricane Analysis and Forecast System (HAFS) QPF for landfalling tropical cyclones (Albright and Nelson 2024), and verification of the Mesoscale Precipitation Discussion and Excessive Rainfall Outlook, among other applications currently under development. WPC is also introducing the next generation of scientists to useful applications of MODE through hosting Lapenta interns (Christina Comer, Austin Jerke, and Victoria Scheidt between 2020-2022).

From the FFaIR verification page:

From the WWE 2024 MODE page:

When run on different computing platforms, complex numerical weather prediction systems will not necessarily produce forecasts that match bit-for-bit due to differences in precision.  Nonlinearity and parameterizations with threshold behaviors (e.g., convective parameterizations) tend to cause these differences to widen with forecast lead time, making it difficult to validate the implementation of an operational forecast system on a new computing platform. While assisting the United States Air Force (USAF) with an implementation validation test, DTC staff developed a novel approach for applying MODE to this type of validation exercise. MODE was used to identify coherent objects that exceeded a threshold for differences between a baseline and new implementation. The properties of these objects were then analyzed to provide insight into potential sources for the associated differences.

The impacts of MODE on operational decisions extends beyond the US. The UK Met Office used MODE to assess the position, timing, and intensity of jet cores, surface highs and lows, and changes in the behavior of these forecast synoptic features as part of their pre-implementation assessment of a new dynamical core.  This assessment found the combination of a new dynamic core and increased resolution produced a more energetic and active atmospheric model with deeper lows and stronger highs and jet cores (Mittermaier et al 2016). In addition, the Bureau of Meteorology has found MODE useful for evaluating the quality of their Heatwave forecast maps, in particular for severe and extreme areas. This work revealed an over-forecast bias in terms of both spatial spread and intensity (personal communication, Nickolas Loveday).

MODE Forecast objects with analysis outlines for a single NN768EG forecast (t + 48 hr) valid at 0000 UTC 9 Dec 2012. Colors indicate distinct objects. (a) Low pressures less than 990 hPa, (b) high pressure greater than 1036 hPa, and (c) 250-hPa jet cores stronger than 60 m s−1 (from Mittermaier et al. 2016)

The Method for Object-based Diagnostic Evaluation, commonly referred to as MODE, was originally developed to address the skill of forecasts of localized episodic phenomena such as rainfall (Davis et al 2006). The inclusion of this technique in the initial release of the Model Evaluation Tools (MET), the verification package developed and supported by the DTC, in 2008 has led to the exploration of an ever broadening application space for this useful tool. In 2009, the DTC collaborated with the NOAA Hydrometeorology Testbed (HMT) to explore the application of MODE to Atmospheric Rivers by applying MODE to observed and forecasted fields of integrated water vapor (IWV). Collaborations with HMT, as well as the NOAA Hazardous Weather Testbed, also explored the application of MODE to ensemble forecasts.  From there, MODE has been used to examine the spatial and temporal characteristics of cloud cover forecasts from high-resolution NWP models with a novel approach used by Mittermaier and Bullock (2013) where they actually tracked cloud breaks instead of the clouds themselves.  Applications related to clouds and precipitation have expanded over the years to include assessments of the Global Synthetic Weather Radar product and tropical cyclone quantitative precipitation forecasts (Newman et al 2023, Newman et al 2024) by the DTC, using infrared brightness temperatures to identify clouds (Griffin et al 2021), and an application to hail identification (Adams-Selin et al 2023).

While MODE was originally developed to address the needs for evaluating high-resolution NWP forecasts, its application has been expanded to include climate models. In particular, characterizing the location and predictability of the ITCZ in the Community Earth System Model Large Ensemble (CESM-LE), as well as temperature and precipitation anomalies associated with ENSO. In addition to expansions related to applications to the atmosphere, MODE has been applied to sea ice forecasts and more recently to the chlorophyll-a bloom season (Mittermaier et al 2021) and marine heatwaves (Cohen et al 2024). While most of the studies mentioned above involved collaborations with MET team members, its application to marine heatwaves was discovered through a recent workshop presentation. Jacob Cohen indicated he chose MODE for his research because of its flexibility, its widespread use over many applications, and its helpful documentation and user support.

Advancing the capabilities of MODE has continued since its inception to include the capability to track objects through time, referred to as MODE Time Domain or MODE-TD, and the capability to define objects based on multiple fields, referred to as Multivariate MODE.  MODE-TD is being used to assess sea ice location and timing and fire spread, while Multivariate MODE is being applied to identify drylines, regions that meet red flag criteria based on relative humidity and winds and blizzard-like objects based on precipitation type, 10-m winds and visibility.

The applications described above should not be considered an exhaustive list given this information was gathered by soliciting input from DTC staff.  The breadth of information gathered based on our internal knowledge suggests that a thorough literature review would likely produce a broader list of applications!

The Method for Object-based Diagnostic Evaluation (MODE), which was originally developed to work with a single forecast and observation input, has been extended to accommodate multiple inputs, known as Multivariate MODE (MvMODE). This extension was first conceptualized to analyze complex objects such as dry lines. MvMODE has been adapted to identify and assess blizzard-like features, aimed at enhancing feature-driven evaluations of high-impact hydrometeorological events. In addition, recent work has improved the flexibility of MvMODE and streamlined its functionality to enhance user accessibility. METplus v5.1.0 documentation includes a use case for identifying blizzard-like objects, using the inputs of precipitation type, visibility, and 10 m winds, as well as details on new configuration parameters added during the enhancements. Output from MvMODE resembles that of single-variable MODE, including ASCII files of contingency table counts and object attributes, a NetCDF file of the objects, and a summary PostScript file. The recent improvements make MODE a more powerful tool for evaluating complex weather phenomena, providing researchers and forecasters with richer insights into spatial accuracy of feature-based events.

More information on MvMODE can be found in the METplus User’s guide. Information on additional configuration options can be found in the config file section of the user’s guide.