DTC Verification Workshop 2008

  • About
  • Schedule
  • Logistics
  • Register

16-18 April 2008, National Center for Atmospheric Research (NCAR), Foothills Laboratory, EOL-Atrium, Boulder, Colorado.

The purposes of this meeting are to demonstrate the Model Evaluation Tools (MET) to the community; discuss new state-of-the-art forecast evaluation methods that should be included in future releases; and to discuss user requirements for verification tools and future system components (e.g., database, display, distributed processing capabilities).

Topics that will be considered include:

Evaluation of Ensemble and Probabilistic Forecasts

Confidence Intervals and Hypothesis Testing -- Including such issues as spatial/temporal dependence, multiple testing, Non-Gaussian Random Variables, bootstrap intervals, Bayesian precision intervals, Observational Error.

Verification of Extreme or Rare Events

User-specific verification

System Components and Architecture

Workshop organizers: Lacey Holland, Barbara Brown, and Eric Gilleland

Agenda for DTC Verification Workshop (2 April 2008)

Updated April 4, 2008.

All sessions will take place in the EOL Atrium in Building 1 of NCAR's Foothills Lab

pdf version of the agenda.

Presentation slides and discussion questions (where available) may be obtained in pdf form by clicking on the titles in the web version of the schedule below.

We also had bonus talks by Mike Charles, Martin Goeber, and Ian Jolliffe on the current NCEP verification system, grid-box mean vs. point verification, and rank histograms, respectively.

Wednesday, April 16
Intro, goals, and overview verification in the DTC (Chair: John Halley Gotway)
8:00 - 8:30Coffee and tea
8:30 - 9:00Introduction and goalsBarbara Brown (NCAR; Boulder, CO)
9:00 - 9:20DTC OverviewLouisa Nance (NCAR; Boulder, CO)
9:20 - 9:40A verification service for DTCJennifer Mahoney (NOAA/ESRL; Boulder, CO)
9:40 - 10:00MET overviewLacey Holland (NCAR; Boulder, CO)
10:00 - 10:30COFFEE BREAK
10:30 - 11:15Development of a verification method testbedMike Baldwin (Purdie Univ.; W. Lafayette, IN)
11:15 - 12:00Discussion - general topics Barbara Brown (NCAR; Boulder, CO)
12:00 - 1:30LUNCH
Spatial verification methods (Chair: Barbara Casati)
1:30 - 2:15Spatial methodsBeth Ebert (BOM; Melbourne, Australia)
2:15 - 3:00Overview of spatial method intercomparison Dave Ahijevych (NCAR; Boulder, CO)
3:00 - 3:15COFFEE BREAK
3:15 - 4:00Discussion Eric Gilleland (NCAR; Boulder, CO)
Verification of probability and ensemble forecasts (Chair: Barbara Casati)
4:00 - 4:45Probability and ensemble methodsLaurie Wilson (Env. Canada; Montreal, Canada)
4:45 - 5:15R methods and rank histogramsMatt Pocernich (NCAR; Boulder, CO)
5:30 - 7:00Reception - Foothills Lab Cafeteria
Thursday, April 17
Verification of probability and ensemble forecasts (cont.) (Chair: David Ahijevych)
8:00 - 8:30Coffee and tea
8:30 - 8:50Spread-skill relationshipsTom Hopson (NCAR; Boulder, CO)
8:50 - 9:10Verification of Hydrologic Forecasts in the NWSYuqiong Liu (NOAA/OHD)
9:10 - 9:45Spatial evaluation of ensemble forecastsBill Gallus (Iowa State, Univ.) and Barbara Casati (Ouranos, Canada)
9:45 - 10:30Discussion - Ensemble and probability forecasts Lacey Holland (NCAR; Boulder, CO)
10:30 - 10:45COFFEE BREAK
Verification service (Chair: David Ahijevych)
10:45 - 11:45Verification Service PlansSean Madine (NOAA/ESRL, Boulder, CO)
11:45 - 12:30Discussion - Verification service
12:30 - 2:00LUNCH
Inference - Confidence intervals and hypothesis tests (Chair: Jennifer Mahoney)
2:00 - 3:00Inference methodsIan Jolliffe (Univ. of Exeter, UK)
3:00 - 3:30Bootstrap methodsEric Gilleland (NCAR; Boulder, CO)
3:30 - 4:00COFFEE BREAK
4:00 - 5:00Discussion - inference methods Matt Pocernich (NCAR; Boulder, CO)
Friday, April 18
Observation uncertainty (Chair: Eric Gilleland)
8:00 - 8:30Coffee and tea
8:30 - 9:00Observation uncertainty overviewBarbara Brown (NCAR; Boulder, CO)
9:00 - 9:20Estimation of observation uncertainty in data assimilationJosh Hacker (NCAR; Boulder, CO)
9:30 - 10:00Verification of cloud parameters: a future role for satellite imagery?Marion Mittermaier (Met. Office; UK)
10:00 - 10:30COFFEE BREAK
10:30 - 11:00Wavelet reconstruction of sparse observationsBarbara Casati (Ouranos; Canada)
11:00 - 11:45Discussion - obs uncertainty issues Beth Ebert (BOM; Melbourne, Australia)
11:45 - 1:15LUNCH
Cyclone tracks, extremes and standards of reference (Chair: Lacey Holland)
1:15 - 1:30NCEP Cyclone track evaluationGeoff DiMego
1:30 - 2:00Verification of extremesBarbara Casati (Ouranos; Canada)
2:00 - 2:30Types of reference forecasts: making the appropriate choicesMarion Mittermaier (Met. Office; UK)
2:30 - 3:15Discussion - extremes and standards of reference Martin Goeber (DWD; Offenbach, Germany)
3:15 - 3:30COFFEE BREAK
Summary and discussion
3:30 - 4:15Summary of priorities and recommendationsBarbara Brown    


The workshop will be held at the National Center for Atmospheric Research (NCAR) Foothills Laboratory in the northeast part of Boulder, Colorado. The main meeting sessions will take place in the EOL Atrium in Building 1. The Foothills Campus has an on-site cafeteria where breakfast and lunch may be purchased.

Remote Attendence

For those who wish to attend, but cannot make it to Boulder, we will be employing ReadyTalk. Please email Barbara Brown (bgb " at " ucar "dot" edu) if you would like to attend the meeting in this way.


Arrangements have been made with the Best Western Golden Buff Lodge for a special lodging rate of $59.00 per night. The Golden Buff is located at 1725 28th Street, Boulder, CO 80301, and is on the NCAR shuttle route and near to many shopping areas and restaurants. You can make reservations with them at bwgoldenbuff@yahoo.com or at 303-442-7450 (1-800-999-2833). For the DTC verification workshop use Group # 506476; for the Intercomparison Project use Group # 506475 to get the special rate.


NCAR provides a regular shuttle service that has a stop across the street from the Golden Buff Lodge, at 28th Street and Canyon Blvd. Shuttle schedules and maps can be found at http://www.fin.ucar.edu/sass/transport/shuttserv.html.

An NCAR van will be doing a special pick-up at the Golden Buff every morning during the workshop to take workshop participants to the NCAR Foothills Lab. The van will pickup at the Golden Buff at 7:55am. Pickup from Foothills will be at:
Wednesday: 7:05pm
Thursday: 5:05pm
Friday: TBD

Email: Barbara Brown   (bgb@ucar.edu)
to register for the workshop.

Please provide complete contact information, and indicate your primary areas of interest for this workshop (e.g., system components; probabilistic forecasts; extremes; uncertainty).