Forecast Evaluation and Applied Statistics
- Related Links
Providing meaningful information to scientific developers and operational end users
Welcome to the homepage of the Forecast Evaluation and Applied Statistics group of RAL's Joint Numerical Testbed (JNT). This group was formerly part of the Weather Systems and Assessment Program (WSAP), and was formed out of the earlier forecast verification group within RAL. The new name better reflects the wide variety of projects and activities undertaken by us; including extreme value statistical analysis applied to weather and climate data, weather modification, and the development of a state-of-the-art suite of software tools for performing forecast verification.
The page is organized into a tabbed front page giving information about: relevant software (including many software packages designed by our staff), activities of our group (as well as some related activities by other groups in the Colorado area), related links, and people. Links to pages about specific projects involving our group are given at the right along with upcoming workshops being largely organized by us.
The roots of applied statistics in RAL are in forecast verification, which is the process of determining the quality of forecasts. Below is some further information about this subject.
Statistical verification of forecasts is a critical component of their development. Improvements can be made by evaluating forecast products throughout the development process as deficiencies in the algorithms are discovered. Verification is also beneficial to forecasters and end users because verification findings supply them with objective data about the quality or accuracy of the forecasts, which can feed into decision processes (Brown, 1996).
Forecast verification software initially developed for the WRF model. Developed by John Halley Gotway, Lacey Holland, Randy Bullock, David Ahijevych, Barbara Brown, Chris Davis, and Eric Gilleland
Developed primarily by our staff include:
Extremes Toolkit (extRemes)
Graphical User Interface software for weather and climate applications of statistical extreme value analysis. Developed by Eric Gilleland, Rick Katz (ISSE), and Greg Young.
Forecast verification analysis and graphical functions. Developed by and still maintained by Matt Pocernich.
Other statistical software developed at NCAR
The following reading groups are organized within the Forecast Evaluation and Applied Statistics group.
The following is a schedule of meetings organized in full or in part by the Forecast Evaluation and Applied Statistics group.
- ICP Fall Workshop, 24-25 August, 2009, Boulder, Colorado.
- DTC Verification Workshop, 26-28 April 2009, Boulder, CO.
Selected seminar presentations by our staff.
Extreme events in climate and weather an interdisciplinary workshop (10w5016), Banff Centre, Banff, Alberta, Canada, 22--27 August 2010. Gilleland, E. Some extreme value problems in climatology (pdf)
90th American Meteorological Society Annual Meeting, 16-21 January 2010, Atlanta, Georgia. Gilleland, E, D Ahijevych, BG Brown, B Casati and EE Ebert. Spatial Forecast Verification. 20 January 2010 (pdf).
The Fourth International Verification Methods Workshop. Finnish Meteorological Institute (FMI), Helsinki, Finland, 4-10 June 2009. Gilleland E, J Lindström and F Lindgren. Spatial Forecast Verification: The Image Warp. (pdf)
The 19th TIES Conference 8-13 June 2008 Kelowna, BC Canada. Gilleland E. Spatial Extremes in Atmospheric Problems (invited talk, pdf, References pdf). Lindström J, E Gilleland and F Lindgren. The Image Warp for Gridded Forecast Verification. (invited talk, ppt, pdf).
2007-08 Program on Risk Analysis, Extreme Events and Decision Theory, Opening workshop, 16-19 September, 2007, Statistical and Applied Mathematical Sciences Institute (SAMSI), Research Triangle Park, North Carolina, U.S.A. Gilleland E, Extremes and Atmospheric Data (invited talk, pdf)
Schedule of Seminars
|Are our Air Traffic Networks Particularly Vulnerable to Spatial Hazards? (abstract)||Tuesday, February, 15 at 1:30pm/FL2-1022 (Refreshments at 1:15pm)||Wilkinson, Sean M.|
|Circular CAR Modeling of Vector Fields (abstract)||Tuesday, November, 2 at 3:00pm/ML-Chapman||Modlin, Danny|
|Spatio-Temporal Modeling of Precipitation Using Gaussian Markov Random Fields (abstract, pdf)||Wednesday, September 17 at 1:30pm/ML-Chapman (refreshments at 1:15pm)||Johan Lindström|
|Fast Estimation of Spatially Dependent Temporal Trends Using Gaussian Markov Random Fields (abstract, pdf)||Wednesday, September 17 at 10:30am/ML-Chapman||David Bolin|
|Between residual tail-dependence and Hüsler-Reiss triangular arrays: Tail-dependence parameters and limiting multivariate EV distributions under a new second order condition. (abstract (pdf))||Friday, July 11 at 10:00am/ML-Chapman||Rolf-Dieter Reiss|
|Statistical interpretation of NWP model output: A survey of methods||Monday, June 16 at 10:00am/FL-MSR (FL2-1022)||Laurie Wilson|
|Image Warping for Forecast Verification (abstract, txt)||Tuesday, June 3 at 10:45am/FL-MSR (FL2-1022)||Johan Lindström|
|High-resolution time lagged ensembles: walking the resolution-predictability tightrope (abstract, txt)||Wednesday, April 23 at 3:00pm/FL-MSR (FL2-1022)||Marion Mittermaier|
|Rainmakers, scientists and statisticians: The challenges in designing a weather modification experiment (abstract, pdf)||Wednedsay, April 9 at 3:00pm/FL2-1001||Matt Pocernich|
|A wavelet-based verification approach to account for the variation in scale representativeness of observation networks (abstract, txt)||Tuesday, April 8 at 3:00pm/FL-MSR (FL2-1022)||Barbara Casati|
|30 July - 6 August||Brian Ancell, Texas Tech University, Atmospheric Sciences Group, Dept. of Geosciences, Lubbock, Texas|
|1 June - 15 August||Christopher Bednarczyk, Texas Tech University, Atmospheric Sciences Group, Dept. of Geosciences, Lubbock, Texas|
|1 June - 15 August||Roelof Burger, University of Witwatersrand, School of Geography, Archaeology and Environmental Studies, Johannesburg, South Africa|
|15 August - 14 September||Sara Jane Lock, University of Leeds, School of Earth and Environment, Leeds, United Kingdom|
|15 - 26 August||Marion Mittermeier, Mesoscale Model Development and Diagnostics Group, NWP, Met Office, United Kingdom|
|18 September - 1 October||Don Morton, University of Alaska, Arctic Region Supercomputing Center, Fairbanks, Alaska|
|27 June - 6 August||Esa-Matti Tastula, University of South Florida, College of Marine Science, Saint Petersburg|
|17 - 23 July||Marcus Walter, Cornell University, Earth and Atmospheric Sciences, Ithaca, New York/Good Morning America, New York, New York|
|1 - 5 November||Danny Modlin, North Carolina State University, Dept. of Statistics, Raleigh, North Carolina|
|31 March - 27 April||Barbara Casati, Ouranos Consortium on Regional Climatology and Adaptation to Climate Change, Montreal, Canada|
|12 - 25 April||Marion Mittermaier, Mesoscale Model Development and Diagnostics Group, NWP, Met Office, United Kingdom|
|2 - 6 June||Johan Lindström, Mathematical Statistics, Centre for Mathematical Sciences, Lund University, Lund, Sweden.|
|10-20 June||Laurie Wilson, Science and Technology Branch, Environment Canada, Dorval, Québec, Canada.|
|11-15 July||Rolf-Dieter Reiss, University of Siegen.|
|3 August - 12 October||Johan Lindström, Mathematical Statistics, Centre for Mathematical Sciences, Lund University, Lund, Sweden.|
The following regularly scheduled seminars are offered by other similar institutions, but are not necessarily affiliated with the Forecast Evaluation and Applied Statistics group.
Gilleland, E., 2011: Spatial Forecast Verification: Baddeley's Delta Metric Applied to the ICP Test Cases, Accepted to Wea. Forecasting
Gilleland, E. and Katz, R.W., 2011: New software to analyze how extremes change over time. Eos, 11 January, 92 (2), 13--14. (pdf)
Gilleland, E., D.A. Ahijevych, B.G. Brown and E.E. Ebert, 2010: Verifying Forecasts Spatially. Bull. Amer. Meteor. Soc., October, 1365--1373.
Heaton, M.J., M. Katzfuss, S. Ramachandar, K. Pedings, E. Gilleland, E. Mannshardt-Shamseldin, and R.L. Smith, 2010: Spatio-Temporal Models for Large-scale Indicators of Extreme Weather. Accepted to Environmetrics
Gilleland, E., J. Lindström, and F. Lindgren, 2010. Analyzing the image warp forecast verification method on precipitation fields from the ICP. Wea. Forecasting, 25, (4), 1249--1262.
Abeysirigunawardena, D.S., E. Gilleland, D. Bronaugh, 2009. Extreme wind regime responses to climate variability and change in the inner-south-coast of British Columbia Canada. Atmosphere-Ocean, 47(1):41--61.
Ahijevych, D., E. Gilleland, B.G. Brown, and E.E. Ebert, 2009. Application of spatial verification methods to idealized and NWP gridded precipitation forecasts. Wea. Forecasting, 24 (6), 1485--1497.
Gilleland, E., D. Ahijevych, B.G. Brown, B. Casati, and E.E. Ebert, 2009. Intercomparison of Spatial Forecast Verification Methods. Wea. Forecasting, 24, 1416--1430, DOI: 10.1175/2009WAF2222269.1.
Gilleland E, TCM Lee, J Halley Gotway, RG Bullock, and BG Brown, 2008. Computationally efficient spatial forecast verification using Baddeley's Δ image metric. Mon. Wea. Rev. 136(5):1747--1757.
Zheng X, and RW Katz, 2008. Simulation of spatial dependence in daily rainfall using multisite generators. Water Resources Research, 44 (in press).
Zheng X, and RW Katz, 2008. Mixture model of generalized chain-dependent processes and its application to simulation of interannual variability of daily rainfall. J. Hydrology, 349:191--199.
Apipattanavis S, G Podesta, B Rajagopalan, and RW Katz, 2007. A semiparametric multivariate and multisite weather generator. Water Resources Research, 43(11): W11401, doi:10.1029/2006WR005714.
Furrer EM, and RW Katz, 2007. Generalized linear modeling approach to stochastic weather generators. Climate Research, 34:129--144.
Habib E, CG Malakpet, A Tokay, and PA Kucera, 2007. Sensitivity of streamflow simulations to temporal variabilitiy and estimation of Z-R relationships. J. Hydrologic Engineering (in review).
Harper BR, RW Katz, and RC Harriss, 2007. Statistical methods for quantifying the effect of the El Nino-Southern Oscillation on wind power in the North Great Plains of the United States. Wind Engineering, 31:123--137.
Lamptey BL, RE Pandya, TT Warner, R Boger, RT Bruintjes, PA Kucera, A Laing, MW Moncrieff, MK Ramamurthy, and TC Spangler, 2007. An Africa initiative sponsored by the University Corporation for Atmospheric Research. Bull. Amer. Meteor. Soc. (in review).
Davis CA, BG Brown, and RG Bullock, 2006a. Object-based verification of precipitation forecasts, Part I: Methodology and application to mesoscale rain areas. Mon. Wea. Rev. 134:1772--1784.
Davis CA, BG Brown, and RG Bullock, 2006b. Object-based verification of precipitation forecasts, Part II: Application to convective rain systems. Mon. Wea. Rev. 134:1785--1795.
Gilleland E and TL Fowler, 2006. Network design for verification of ceiling and visibility forecasts, Environmetrics 17(6):575--589.
Katz RW and M Ehrendorfer, 2006. Bayesian approach to decision making using ensemble weather forecasts. Weather and Forecasting, 21:220--231.
Gochis D, PA Kucera, and co-authors, 2005. Meeting summary of UCAR/NCAR Junior Faculty Forum on Future Scientific Directions: The water cycle across scales working group. Bull. Amer. Meteor. Soc. 86:1743--1746.
Gilleland E and D Nychka, 2005. Statistical Models for Monitoring and Regulating Ground-level Ozone, Environmetrics 16: 535--546.
Katz RW, GS Brush, and MB Parlange, 2005. Statistics of extremes: Modeling ecological disturbances. Ecology, 86:1124--1134.
Stephenson A and E Gilleland, 2005. Software for the Analysis of Extreme Events: The Current State and Future Directions, Extremes 8:87--109.
Kucera PA, CB Young, and WF Krajewski, 2004. Geographic Information System based studies of radar beam blockage: A case study of Guam. J. Atmos. Oceanic Technol. 24:995--1006.
Ahijevych DA, E Gilleland, BG Brown, EE Ebert, L Holland, and C Davis, 2008. Intercomparison of spatial verification methods. 88th Annual American Meteorological Society (AMS) meeting, New Orleans, Louisiana. 9.1 Probability/Statistics conference.
Gilleland E and RW Katz. "Analyzing seasonal to interannual extreme weather and climate variability with the extremes toolkit (extRemes)", 18th Conference on Climate Variability and Change, 86th American Meteorological Society (AMS) Annual Meeting, 29 January - 2 February, 2006, Atlanta, Georgia. P2.15
Gilleland, E., L. Chen, M. DePersio, G. Do, K. Eilertson, Y. Jin, E.L. Kang, F. Lindgren, J. Lindström, R.L. Smith, and C. Xia, 2011. Spatial Forecast Verification: Image Warping. NCAR Technical Note, TN-482+STR, 30pp. (to appear)
Gilleland E, 2008. Confidence intervals for forecast verification. Submitted to NCAR Technical Notes (pdf)
Books and Book Chapters
Gilleland E, D Nychka, and U Schneider, 2006. Spatial models for the distribution of extremes, Hierarchical modelling for the Environmental Sciences: statistical methods and applications, Edited by JS Clark and A Gelfand. Oxford University Press, New York pp.170--183. ISBN 0-19-8569671
- Cooperative Institute for Research in Environmental Sciences (CIRES)
- Developmental Testbed Center (DTC)
- Weather Systems and Assessment Program (WSAP)
- Weather Research and Forecasting (WRF) Model Support
- Data Assimilation Testbed Center (DATC)
- U.S. Air Force Weather Agency (AFWA)
- American Meteorological Society -- Committee on Probability and Statistics
- National Oceanic and Atmospheric Administration (NOAA)
- NCAR's Geophysical Statistics Project (GSP)
- ADDS -- Aviation Digital Weather Service
- AWC -- Aviation Weather Center
- RTVS -- Real-Time Verification System
- Forecast Verification -- Issues, Methods and FAQ
Staff and Primary Contacts
- BROWN, Barbara | JNT DIRECTOR | ph: 8468 | email: bgb
- NEWMAN, Kathryn | ASSOC SCIENTIST II | ph: 2741 | email: knewman
- Ahijevych, David, Mesoscale and Microscale Meteorology, National Center for Atmospheric Research, Boulder, Colorado.
- Casati, Barbara, Ouranos Consortium on Regional Climatology and Adaptation to Climate Change, Montreal, Canada
- Bernadet, Ligia, NOAA Research-Earth System Research Laboratory/Global Systems Division, Boulder, Colorado, U.S.A.
- Brooks, Harold, Head, Mesoscale Applications Group, NSSL/FRDD National Weather Center, NOAA, Norman, Oklahoma.
- Davis, Chris, Mesoscale and Microscale Meteorology, National Center for Atmospheric Research, Boulder, Colorado.
- Ebert, Beth, Bureau of Meteorology Research Centre, Weather Forecasting Group, Melbourne, Victoria, Australia.
- Katz, Richard W., Institute for Mathematics Applied to Geosciences, National Center for Atmospheric Research, Boulder, Colorado.
- Lindgren, Finn, Department of Mathematical Sciences, Norwegian University of Science and Technology, Trondheim, Norway.
- Lindström, Johan, Mathematical Statistics, Centre for Mathematical Sciences, Lund University, Lund, Sweden.
- Mahoney, Jennifer, NOAA Research-Earth System Research Laboratory/Global Systems Division, Boulder, Colorado, U.S.A.
- Mittermaier, Marion. Mesoscale Model Development and Diagnostics Group, NWP, Met Office, United Kingdom