LAR Website header

4. Providing Innovative Information Services

NCAR Strategic Priority: Conducting Computer Science, Computational Science, Applied Mathematics, Statistics, and Numerical Methods R&D

Highlight: Verification Research and Development

Forecast verification and evaluation activities typically are based on relatively simple metrics regarding the meteorological performance of forecasts and forecasting systems. Metrics such as the Probability of Detection, Root Mean Squared Error, and Equitable Threat Score provide information that is useful for monitoring changes in performance of single aspects of forecast performance with time. However, they generally do not provide information that can be used to improve the forecasts, or that can be used by end users (including forecasters) for decision making. Moreover, it is possible for forecasts that are quite useful – including high resolution forecasts – to have very poor scores when evaluated by using these standard metrics. In response to these limitations, the RAL Verification Group develops improved verification approaches and tools that provide more meaningful and relevant information about forecast performance. The focus of this effort is on diagnostic, statistically valid approaches, including object-based evaluation of precipitation and convective forecasts and other approaches (e.g., distribution-based) that can provide more useful information – for forecast developers as well as forecast users – about forecast performance.

Development and dissemination of new forecast verification approaches requires research and application in several areas, including statistical methods, exploratory data analysis, statistical inference, pattern recognition, and evaluation of user needs.

FY07 Accomplishments:

Example of an application of the MODE, a tool included in MET, as an advanced verification technique provided to the NWP community. By comparing the locations of objects identified using the MODE technique between the forecast (left) and analysis (right) fields, one may identify errors that are difficult or impossible to detect using traditional verification metrics.

The Method for Object-based Diagnostic Evaluation (MODE), developed by RAL and MMM scientists and software engineers, provides one approach for diagnostic evaluation of spatial forecasts that directly measures the performance of the forecasts in terms of specific attributes – spatial displacement, intensity, storm size, and so on – and attributes may be designed to represent the use of the forecast for specific applications (Figure 1).

During the past year MODE was implemented as a tool in the Model Evaluation Tools (MET) developed by the Developmental Testbed Center (DTC ) and has been disseminated to the numerical weather prediction (NWP) community. New methods were also developed to summarize output of the MODE tool which could be useful for decision makers as well as for forecast developers for whom the information may suggest an error in timing in the model, which could lead to model improvements.

The team has also organized and coordinated an intercomparison project (ICP) for spatial forecast verification methods, involving scientists from around the world who are developing new methods for evaluation of spatial forecasts. The project will include applications of all of the methods to the same real and idealized datasets, and comparisons of the capabilities of the various methods, with a goal of determining which methods should be applied to achieve different goals, and to identify the kinds of information that each method can provide.  The ICP is also expected to lead to discussions within the verification and NWP communities regarding development of a protocol for judging when new verification methods are ready to be applied in operational settings.

We continued strong efforts in advocacy through participation in, and leadership of the WMO’s Joint Working Group on Verification (JWGV), numerous conferences and workshops, statistical support for forecast evaluation studies undertaken by the RAL Developmental Testbed Center (DTC), and applications of MODE by various scientists at NCAR and in the wider atmospheric science community. The RAL verification group also organized and hosted a verification workshop on state-of-the-art verification methods in February 2007, which included international verification  and NWP experts.

FY2008 Plans:

Attributes of the MODE approach will be more thoroughly investigated, including extensive examination of the impacts of variations in spatial scale, as represented by the parameters used to define objects. New diagnostic methods will be developed to summarize object attribute comparisons, to provide greater understanding of model performance. The approach for incorporating the time dimension in MODE analyses will be further investigated and enhanced.

The verification method intercomparison project (ICP) will continue, and a white paper summarizing the various methods will be submitted to a journal. We also plan to hold a workshop among the participants during the spring to begin discussions of the results of the ICP evaluations.

MODE will be applied to additional datasets and types of forecasts. An initial effort will be made to examine ensemble forecasts of precipitation from an object perspective. MODE will also be applied to convective and precipitation forecasts as part of NCAR’s program on Short Term Explicit Prediction.

The concept of user-focused verification will be further developed and presented to the forecasting and verification communities.

Long-Term Goals:

The long-term goals of the verification research program are to (a) develop a stable version the MODE approach that can be applied in evaluations of a variety of weather, air quality, climate, and other forecast variables, including precipitation, convection, and other variables that can be represented spatially; (b) enhance the MODE approach to take into account the time dimension and other capabilities desired for applications of the technique; (c) develop user-relevant verification approaches in the context of the needs of specific end users; (d) develop new user-relevant verification approaches for evaluation of probabilistic and ensemble forecasts; (e) develop and disseminate new methods for making statistical inferences about verification measures (i.e., methods to take into account the uncertainty in verification measures); and (f) continue to facilitate activities of the international verification community, and further advance the application of improved verification measures in operational settings.

NextGeneration Network Enabled Weather


Since its inception nine years ago, the Aviation Digital Data Service (ADDS) has emphasized user-friendly and intuitive weather graphics to provide users with enhanced weather situational awareness.  Within the ADDS system, there exists a fundamental infrastructure for serving weather data in a network-centric manner.  The ADDS Flight Path Tool (FPT), for example, enables human users to visualize specific portions from immense volumes of data using a highly interactive software application. The software renders the graphics from the digital data after it is passed over the Internet. In contrast, most Internet weather resources create graphics internally then distribute the final graphical product to users.

While graphical presentations of weather data are useful to human users, there is also a need for machine-to-machine data dissemination to provide data to decision support tools and systems that manage air traffic. The challenge is to provide four-dimensional weather data using standard formats for the request and delivery.

To address this problem, NCAR-RAL has teamed with MIT-Lincoln Labs and NOAA-Global Systems Division to explore standards-based, net-centric data access. The goal of this research is to create a virtual weather database spanning more than one physical location, organization, and data system. To date, these three organizations are involved in various distribution and data access mechanisms but all using their own, internally-developed formats. Starting in late 2007, each laboratory intends to research open standards and technologies to share their data sources.

FY2007 Accomplishments:

The majority of this year’s effort has been to instantiate a gridded weather products server using a Web Coverage Service (WCS), a specification of the Open Geospatial Consortium (OGC).  The OGC is an international standards body with over 300 active member organizations and is responsible for developing and maintaining standards related to geospatial data and access mechanisms. Using the WCS specification, NCAR-RAL now distributes a variety of three-dimensional gridded weather products including analyses and forecasts of icing, turbulence, winds, and temperatures.

Additionally, the data available through the WCS services were exposed in a catalog as an OGC Catalogue Service (CS-W).  This allows the automated discovery of data as it comes in and provides information about the data that may be accessed through the WCS services.  This includes information such as the quality of the data, the organization that originally created or gathered it, the data format in which it is available, and its geographic extent.

FY2008 Plans:

Future work will include non-gridded products (such as METARs, PIREPs, AIR/SIGMETs, and TAFs) and related standards to distribute them.  These new products as well as the gridded products will be accessible via a single NNEW catalog that allows the products of each organization to be treated transparently as a virtual database.  The means to integrate JMBL data sources into the virtual database will be explored, as JMBL will most likely be a part of international weather data distribution.

Modeling Weather Extremes

modeling weather extremesTrends in annual CAPE (J/kg) * shear (m/s) 95th percentile values at individual grid points as estimated using linear models applied to CAPE and shear values estimated using global model reanalysis data over a 42-year period. CAPE is the convective available potential energy and Shear is the magnitude of the vector difference between the surface and 6-km estimated wind. The grid resolution is approximately 1.875 degrees longitude by 1.915 degrees latitude, yielding 17,856 grid points, on a 192 x 94 grid. Temporally, values are available every 6 hours over 42 years (1958 through 1999). Only areas with significant trends are shown; the False Discovery Rate was set to 0.05.

In the third and fourth Assessment Reports of the Intergovernmental Panel on Climate Change, discussions of the impacts of climate change on severe thunderstorms have been limited to comments regarding the difficulty of using storm report databases to determine if changes have taken place historically.  Because convective storms occur on very fine spatial scales, it is not possible to directly resolve such phenomena from coarse-scale global datasets.  However, large-scale indicators can be employed to study trends in environments that are conducive to such severe weather. Initial work in this area has focused on identification of useful measures of large-scale environments that are relevant for severe thunderstorm formation based on NCAR global model reanalysis data. These data have been used to investigate trends in the large scale environmental characteristics, as well as spatial and extreme value distribution attributes. The approaches developed for the reanalysis data will be extended to Global Climate Model (GCM) projections of future climate, to determine the expected characteristics of severe weather environments associated with future climate change scenarios.

This work has led to identification of several statistical challenges as well as new areas for research. Statistical challenges include methods for modeling extreme values in a spatial context; addressing the issue of multiple comparisons inherent in working with gridded data; and making inferences about changes in distribution parameters.

FY2007 Accomplishments

The global dataset of convective parameters that has been created using the NCAR model reanalyses was further analyzed during FY07. The reanalysis dataset has been analyzed to include two important severe weather indicators, Convective Available Potential Energy (CAPE) and vertical shear.  Analyses of trends in CAPE, Shear, and functions of these variables, such as CAPE x Shear, were applied to the global set of gridpoints included in this dataset. A false discovery rate (FDR) procedure was employed to account for the effects of multiple hypothesis testing and to make the analysis spatially robust (Figure 1).  Two manuscripts based on these analyses are in preparation, and several presentations have been made.

The focus of this work has begun to shift to consider the evaluation of convective extremes in projections of a changed climate. Output of the NCAR Community Climate System Model (CCSM3), AB1 scenario, are being used to represent the current climate and will be used to compare to results associated with the reanalysis data. These analyses are being done in collaboration with H. Brooks at the NOAA National Severe Storms Laboratory and P. Marsh at the University of Oklahoma.

FY2008 Plans

The techniques developed with the reanalysis data will be applied to output of the CCSM3 to determine whether the characteristics of CAPE and Shear based on GCM output for an unchanged climate are consistent with the characteristics of these parameters in the reanalysis data. Subsequently (assuming consistency is found), the parameters will be analyzed using CCSM3 output for changed climate scenarios to study how the frequency and intensity of environments conducive to severe weather activity can be expected to change under a future climate scenario. The aim of this work is to determine the current distributions of environments conducive to severe weather, and study how these environments are changing.  Initial steps include determining whether the climate models produce the correct spatial patterns found in the reanalysis data, and in particular, if they correctly place the extrema both spatially and temporally.  Once the consistency of these characteristics is verified, it will be possible to address the differences between current and future climates.  Finally, we will investigate approaches for making inferences about changes in the extreme value distribution parameters for the severe weather indicators. Development of these approaches will allow regional assessments of the characteristics of these indices of the potential for severe weather environments.