Forecast Verification Reading Group

  • About
  • Schedule
  • Suggested Papers
  • Past Papers/Presentations
  • Join/Leave the Group

The purpose of this reading group is to discuss papers relating to forecast verification. Currently, we will be discussing papers related to the issue of climate model evaluation. However, other topics that may come up either in conjunction with this overall topic, or in addition to this topic include:

Ensemble Verification Methods

Confidence Intervals and Hypothesis Testing -- Including such issues as spatial/temporal dependence, multiple testing, Non-Gaussian Random Variables, bootstrap intervals, Bayesian precision intervals, Observational Error.

Verification of Extreme or Rare Events

Spatial Forecast Verification

User-specific verification


Group organizer: Eric Gilleland


Our next meeting is TBD

Paper/SeminarTime/Place PresenterPresentation slides
(if any)

Suggested Papers

If there are any other papers you would like us to discuss or add to the list, please email the organizer (Eric Gilleland).

A beer glass () by a paper indicates papers "on tap" for upcoming sessions.

Climate Model Evaluation

Not every paper in this list is specifically Climate Model Evaluation per se, but are listed here anyway.

Hueffer, K., M. A. Fonseca, A. Leiserowitz, and K. M. Taylor, 2013. The wisdom of crowds: Predicting a weather and climate-related event. Judgment and Decision Making, 8 (2), 91 - 105.

Livina, V. N., N. R. Edwards, S. Goswami, and T. M. Lenton, 2008. A wavelet-coefficient score for comparison of two-dimensional climatic-data fields. Q. J. R. Meteorol. Soc., 134, 941 - 955.

Moise, A. F. and F. P. Delage, 2011. New climate model metrics based on object-oriented pattern matching of rainfall. J. Geophys. Res., 116, D12108, 7 pp., doi:10.1029/2010JD015318

Reichler, T. and J. Kim, 2008. How well do coupled models simulate today's climate? Bull. Amer. Meteorol. Soc., 89, 303 - 311,

Woldemeski, F. M., A. Sharma, B. Sivakumar, and R. Mehrotra, 2012. An error estimation method for precipitation and temperature projections for future climates. J. Geophys. Res., 117, D22104, 13 pp., doi:10.1029/2012JD018062.

Tropical Cyclone (and similar) Forecast Verification

Aberson, S. D., 2008. An alternative tropical cyclone intensity forecast verification technique Wea. Forecasting, 23, 1304--1310.

Done, J., G. J. Holland, C. Bruyère, and A. Suzuki-Parker, 2011. Effects of climate variability and change on Gulf of Mexico tropical cyclone activity. Proc. Offshore Technology Conference, 22190 (

Gneiting, T., Stanberry, L. I., Grimit, E. P., Held, L. and Johnson, N. A. (2008). Assessing probabilistic forecasts of multivariate quantities, with applications to ensemble predictions of surface winds. Test, 17, 211-235. (Also has comment papers and a rejoinder)

Holland, G. J., J. Done, C. Bruyère, C. Cooper, and A. Suzuki-Parker, 2010. Model investigations of the effects of climate variability and change on future Gulf of Mexico tropical cyclone activity. Proc. 2010 Offshore Technology Conference, 20690. (

Moskaitis, J. R., 2008. A Case study of deterministic forecast verification: tropical cyclone intensity. Wea. Forecasting, 23, 1195--1220.

Extreme and Rare Events

Atger F, 2001. Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlin. Proc. Geophys., 8:401--417

Brooks HE, M Kay and JA Hart, 1998. Objective limits on forecasting skill of rare events. 19th Conf. Severe Local Storms, Amer. Met. Soc., 552--555

Ferro, C.A.T., 2007. A Probability Model for Verifying Deterministic Forecasts of Extreme Events. Wea. Forecasting, 22(5):1089--1100, DOI: 10.1175/WAF1036.1

Ferro, C.A.T. and D.B. Stephenson, 2011: Extremal dependence indices: improved verification measures for deterministic forecasts of rare binary events. Accepted to Wea. Forecasting. Available at Early Online Release from:

Friederichs, P., M. Göber, S. Bentzien, A. Lenz, and R. Krampitz, 2009. A probabilistic analysis of wind gusts using extreme value statistics. Meteorol. Z., 18 (6), 615--629, DOI 10.1127/0941-2948/2009/0413.

Stephenson DB, B Casati, CAT Ferro and CA Wilson, 2008. The extreme dependency score: a non-vanishing measure for forecasts of rare events. Meteorol. Appl., 15:41--50, DOI: 10.1002/met.53

Spatial Forecast Verification

Elmore, K. L., M. E. Baldwin, and D. M. Schultz, 2006. Field significance revisited: Spatial bias errors in forecasts as applied to the Eta model. Mon. Wea. Rev., 134, 519--531.

A list of spatial verification papers that is relatively up-to-date can be found at the ICP web site

Past Papers/Presentations

Paper/SeminarTime/Place PresenterPresentation slides
(if any)
2013 Series on Climate Model Evaluation
Taylor, K. E., 2001. Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res., 106, (D7), 7183 - 7192. Wednesday, 12 June 2013 at 10:00am/FL3-1021 Barb Brown -
Gilleland, E., 2013. Testing competing precipitation forecasts accurately and efficiently: The spatial prediction comparison test. Mon. Wea. Rev., 141, (1), 340 - 355. Monday, 22 April 2013 at 10:00am/FL3-1029Eric Gilleland -
Seminar Presentation: Some insights into cloud forecast biases and skill: Putting data and methods under the spotlight (pdf announcement/abstract)Tuesday, 23 August 2011 at 3:00pm/FL2-1001Marion Mittermeier -
2011 Series on Tropical Cyclone (and similar) Forecast Verification
Discussion of methods used officially by the National Hurricane Center Thursday, 6 October 2011 at 10:00am/FL3-1029 Mrinal Biswas-
Dupont, T., M. Plu, P. Caroff, and G. Faure, 2011. Verification of ensemble-based uncertainty circles around tropical cyclone track forecasts. Accepted to Wea. Forecasting, Available at Early Online Release Thursday, 18 August 2011 at 3:00pm/FL3-1029 Eric Gilleland-
Davis, C. A., W. Wang, J. Dudhia and R. Torn, 2010: Does increased horizontal resolution improve hurricane wind forecasts? Wea. Forecasting, 25, (6), 1826--1841. Thursday, 7 July 2011 at 10:00am/FL3-1021 Ligia Bernadet-
Colle, B.A. and Charles, M.E., 2011: Spatial Distribution and Evolution of Extratropical Cyclone Errors over North America and Adjacent Oceans in the NCEP Global Forecast System Model. Wea. Forecasting, 26, 129--149. Available at Early Online Release from: Early Online Release Thursday, 26 May 2011 at 2:00pm/FL3-1029Eric Gilleland-
2008 Series on Spatial Forecast Verification
Keil C and GC Craig, 2007. A displacement-based error measure applied in a regional ensemble forecasting system. Mon. Wea. Rev., 135:3248--3259. Tuesday, June 3 at 12:15/EOL-Atrium (FL1-2198)Matt Pocernich-
Seminar Presentation: Image Warping for Forecast Verification (abstract, txt)Tuesday, June 3 at 10:45am/FL-MSR (FL2-1022) Johan Lindström (pdf)
Seminar Presentation: High-resolution time lagged ensembles: walking the resolution-predictability tightrope (abstract, txt) Wednesday, April 23 at 3:00pm/FL-MSR (FL2-1022) Marion Mittermaier -
Seminar Presentation: A wavelet-based verification approach to account for the variation in scale representativeness of observation networks (abstracts, txt) Tuesday, April 8 at 3:00pm/FL-MSR (FL2-1022) Barbara Casati -
Venugopal, V.; Basu, S.; Foufoula-Georgiou, E. 2005. A new metric for comparing precipitation patterns with an application to ensemble forecasts. J. Geophys. Res. 110: D8, D08111 10.1029/2004JD005395. Friday, April 4 at 10:00am/FL1-2133 David Ahijevych (ppt, pdf)
Roberts NM and HW Lean, 2008. Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev., 136:78--96. Thursday, March 13 at 3:00pm/FL3-2072 Eric Gilleland pdf

You can subscribe or unsubscribe from our email list at*

Please also send an email to Eric Gilleland to explain who you are and why you want to be on the list to help protect against spam.

*Email addresses are retained only for announcing pertinent information about the reading group (e.g., meeting times, papers to be read, etc.).