Overarching Goal
To develop and utilize advanced forecast verification approaches that provide meaningful information to scientific developers and operational end users, and to conduct incisive quality assessment tests to products that are transferred to operational settings.
Motivation
As detailed in previous sections, the ability to quantify the performance capabilities of forecast products – including forecast skill, accuracy, and reliability – is essential to all aspects of the Research Applications Laboratory’s work. The approaches required are far from being well described in textbook studies, and in fact represent an active area of research in which RAL scientists play a leading role. Our sponsors and our principal investigators demand that we maintain a strong program in this area.
Research in this area is very timely. The meteorological community is witnessing a relentless pursuit toward smaller grid spacing in NWP models, and yet these models appear to provide little or no additional deterministic skill beyond their coarser-resolution counterparts. This at least partly results from the fact that forecast performance has traditionally been summarized using a few scalar measures of quality. Such metrics typically involve point-wise root-mean-squared errors, either from forecasts interpolated to observation locations or from gridded forecasts and observations. By using such simple measures of forecast quality, it is difficult to identify the situations where the high-resolution forecast performance differs relative to the coarse-resolution forecasts, since all incorrect forecasts are treated in the same way. Also, these metrics do not provide adequate information about model deficiencies so that users can consider NWP forecasts in ways that maximize their utility or compensate for the shortcomings of the model. In fact, this problem is not unique to high-resolution forecasts. Standard verification metrics do not tell us how or why a forecast is bad or good, and so they cannot be used to help intelligently choose between competing forecasting systems.
Strategic Approach
RAL will continue to work toward development of forecast verification and evaluation approaches that are diagnostic and provide meaningful information to forecast developers and users. The initial emphasis of this work will be on precipitation forecasts and forecasts of convection. However, we will extend this effort to include other types of spatial forecasts for which users and forecast developers need meaningful information about the quality of the forecasts; this information is needed to allow diagnosis and development of needed forecast improvements and to allow optimal use of the forecasts. In addition, we will begin to develop approaches that are appropriate for application to ensemble and probabilistic forecasts, including forecasts of probability distribution functions; these types of forecast information will form the basis of many forecasting systems in the future. Additional work will focus on techniques for appropriately comparing forecasting systems (e.g., development and application of statistical hypothesis tests and confidence interval approaches) and the evaluation of forecasts on multiple spatial scales.
Verification information provides an essential link between the forecasts and the users or decision support systems. The RAL Verification Program will also interact with the Societal Impacts Program to begin to develop forecast verification approaches that are more specifically user-focused.
RAL will continue to collaborate with verification experts at universities, government laboratories, and research institutions, such as the University of Oklahoma and the National Severe Storms Laboratory, and will develop collaborations with other communities (e.g., medicine, economics) with needs for similar approaches. We will continue to engage with (and provide leadership to) the international atmospheric science community on topics related to verification, through activities of the WMO Joint Working Group on Verification. In addition, RAL will provide expertise on verification methods and approaches at international verification workshops and tutorials and will support the planning of verification activities for international programs such as the Beijing 2008 Olympics Forecast Demonstration Project and THORPEX.
RAL strongly believes that atmospheric science advancements emanating from the research community can significantly improve the safety and efficiency of public and private sector organizations if the improvements are developed and tailored for specific end-user groups. Appropriate evaluation of these advancements will allow optimal utilization of the improvements. We will focus our attention on sponsors that provide these research, development and technology transfer opportunities.