Forecast Evaluation and Applied Statistics

Providing meaningful information to researchers and operational end users of weather and climate models

The roots of applied statistics in RAL are in forecast verification, which is the process of determining the quality of forecasts. Statistical verification of forecasts is a critical component of their optimization. Improvements can be made by evaluating forecast products throughout the development process as deficiencies in the algorithms are discovered. Verification also benefits forecasters and end users by supplying them with objective data about the quality or accuracy of the forecasts, which can feed into decision processes (Brown, 1996).

Model Evaluation Tools (MET) is designed to be a highly-configurable, state-of-the-art suite of verification tools.
Model Evaluation Tools (MET) is designed to be a highly-configurable, state-of-the-art suite of verification tools.

The JNTP verification and statistics team provide improved verification approaches and tools that provide more meaningful and relevant information about forecast performance. [n1] The focus of this effort is on diagnostic, statistically valid approaches, including feature–based evaluation of precipitation and convective forecasts, as well as hypothesis testing that can account for various forms of dependence as well as location/timing errors. In addition, JNTP[n2]  develops forecast evaluation tools and training on verification methods that are available for use by members of the operational, model development, and research communities.

Verifying predictions on different spatial and temporal scales presents different challenges.  Our group works with model developers and end-users to address issues such as: 1) displacement errors for storm-scale and meso-scale simulations; 2) forecast consistency of any scale NWP simulations; 3) limits of predictability of subseasonal-to-seasonal predictions; 4) time-agnostic pattern prediction for decadal climate prediction; 5) skill at predicting extreme (i.e. rarely occurring) events; and 6) skill at predicting events over minutes to days[n1] .  Our expertise spans evaluating a wide variety of predictions, including: gridded weather and climate fields (both deterministic and probabilistic), tropical cyclone tracks and intensity, renewable energy prediction, and developing plans for systematic testing.

The wide variety of projects and activities undertaken by our group include: extreme value statistical analysis applied to weather and climate data, development and testing of spatial forecast verification techniques, support of systematic testing and evaluation (T&E) activities within RAL, and the development of a state-of-the-art suite of software tools for performing forecast verification. These tools are free to download and include the Model Evaluation Tools (MET) and several packages for R-Statistics (e.g. distillery, extRemes, ismev, smoothie, SpatialVx, and verification).

Active research:

  • Spatial methods
  • Forecast consistency
  • Extremes
  • Verification of remotely sensed fields
  • Process-oriented/diagnostic verification
  • Representation of observation uncertainty

Search through all publications in NCAR's OpenSky Library.

Selected Publications

Refereed Papers
Non-Refereed Papers
Technical Notes
Books/Book Chapters

Refereed Papers

2011

Gilleland, E., 2011: Spatial Forecast Verification: Baddeley's Delta Metric Applied to the ICP Test Cases, Accepted to Wea. Forecasting

Gilleland, E. and Katz, R.W., 2011: New software to analyze how extremes change over time. Eos, 11 January, 92 (2), 13--14. (pdf)

2010

Gilleland, E., D.A. Ahijevych, B.G. Brown and E.E. Ebert, 2010: Verifying Forecasts Spatially. Bull. Amer. Meteor. Soc., October, 1365--1373.

Heaton, M.J., M. Katzfuss, S. Ramachandar, K. Pedings, E. Gilleland, E. Mannshardt-Shamseldin, and R.L. Smith, 2010: Spatio-Temporal Models for Large-scale Indicators of Extreme Weather. Accepted to Environmetrics

Gilleland, E., J. Lindström, and F. Lindgren, 2010. Analyzing the image warp forecast verification method on precipitation fields from the ICP.Wea. Forecasting25, (4), 1249--1262.

2009

Abeysirigunawardena, D.S., E. Gilleland, D. Bronaugh, 2009. Extreme wind regime responses to climate variability and change in the inner-south-coast of British Columbia Canada. Atmosphere-Ocean, 47(1):41--61.

Ahijevych, D., E. Gilleland, B.G. Brown, and E.E. Ebert, 2009. Application of spatial verification methods to idealized and NWP gridded precipitation forecasts. Wea. Forecasting24 (6), 1485--1497.

Gilleland, E., D. Ahijevych, B.G. Brown, B. Casati, and E.E. Ebert, 2009. Intercomparison of Spatial Forecast Verification Methods. Wea. Forecasting24, 1416--1430, DOI: 10.1175/2009WAF2222269.1.

2008

Gilleland E, TCM Lee, J Halley GotwayRG Bullock, and BG Brown, 2008. Computationally efficient spatial forecast verification using Baddeley's Δ image metric. Mon. Wea. Rev. 136(5):1747--1757.

Zheng X, and RW Katz, 2008. Simulation of spatial dependence in daily rainfall using multisite generators. Water Resources Research44 (in press).

Zheng X, and RW Katz, 2008. Mixture model of generalized chain-dependent processes and its application to simulation of interannual variability of daily rainfall. J. Hydrology349:191--199.

2007

Apipattanavis S, G Podesta, B Rajagopalan, and RW Katz, 2007. A semiparametric multivariate and multisite weather generator. Water Resources Research43(11): W11401, doi:10.1029/2006WR005714.

Furrer EM, and RW Katz, 2007. Generalized linear modeling approach to stochastic weather generators. Climate Research34:129--144.

Habib E, CG Malakpet, A Tokay, and PA Kucera, 2007. Sensitivity of streamflow simulations to temporal variabilitiy and estimation of Z-R relationships. J. Hydrologic Engineering (in review).

Harper BR, RW Katz, and RC Harriss, 2007. Statistical methods for quantifying the effect of the El Nino-Southern Oscillation on wind power in the North Great Plains of the United States. Wind Engineering,31:123--137.

Lamptey BL, RE Pandya, TT Warner, R Boger, RT Bruintjes, PA Kucera, A Laing, MW Moncrieff, MK Ramamurthy, and TC Spangler, 2007. An Africa initiative sponsored by the University Corporation for Atmospheric Research. Bull. Amer. Meteor. Soc. (in review).

2006

Davis CABG Brown, and RG Bullock, 2006a. Object-based verification of precipitation forecasts, Part I: Methodology and application to mesoscale rain areas. Mon. Wea. Rev. 134:1772--1784.

Davis CABG Brown, and RG Bullock, 2006b. Object-based verification of precipitation forecasts, Part II: Application to convective rain systems. Mon. Wea. Rev. 134:1785--1795.

Gilleland E and TL Fowler, 2006. Network design for verification of ceiling and visibility forecasts, Environmetrics 17(6):575--589.

Katz RW and M Ehrendorfer, 2006. Bayesian approach to decision making using ensemble weather forecasts. Weather and Forecasting,21:220--231.

2005

Gochis D, PA Kucera, and co-authors, 2005. Meeting summary of UCAR/NCAR Junior Faculty Forum on Future Scientific Directions: The water cycle across scales working group. Bull. Amer. Meteor. Soc.86:1743--1746.

Gilleland E and D Nychka, 2005. Statistical Models for Monitoring and Regulating Ground-level Ozone, Environmetrics 16: 535--546.

Katz RW, GS Brush, and MB Parlange, 2005. Statistics of extremes: Modeling ecological disturbances. Ecology86:1124--1134.

Stephenson A and E Gilleland, 2005. Software for the Analysis of Extreme Events: The Current State and Future Directions, Extremes8:87--109.

2004

Kucera PA, CB Young, and WF Krajewski, 2004. Geographic Information System based studies of radar beam blockage: A case study of Guam. J. Atmos. Oceanic Technol. 24:995--1006.

Non-refereed Papers

Ahijevych DAE GillelandBG Brown, EE Ebert, L Holland, and C Davis, 2008. Intercomparison of spatial verification methods. 88th Annual American Meteorological Society (AMS) meeting, New Orleans, Louisiana. 9.1 Probability/Statistics conference.

Gilleland E and RW Katz. "Analyzing seasonal to interannual extreme weather and climate variability with the extremes toolkit (extRemes)", 18th Conference on Climate Variability and Change, 86th American Meteorological Society (AMS) Annual Meeting, 29 January - 2 February, 2006, Atlanta, Georgia. P2.15

Technical Notes

Gilleland, E., L. Chen, M. DePersio, G. Do, K. Eilertson, Y. Jin, E.L. Kang, F. Lindgren, J. Lindström, R.L. Smith, and C. Xia, 2011. Spatial Forecast Verification: Image Warping. NCAR Technical Note, TN-482+STR, 30pp. (to appear)

Gilleland E, 2008. Confidence intervals for forecast verification. Submitted to NCAR Technical Notes (pdf)

Books and Book Chapters

Gilleland E, D Nychka, and U Schneider, 2006. Spatial models for the distribution of extremes, Hierarchical modelling for the Environmental Sciences: statistical methods and applications, Edited by JS Clark and A Gelfand. Oxford University Press, New York pp.170--183. ISBN 0-19-8569671

Forecast Evaluation and Applied Statistics