You are here: NCAR •
RAL •
Strategic Plan
• RAL Strategic Plan 2009-2013

Goal Area: Testing, Validating & Verifying Advanced Numerical Forecasting Techniques

"Maintain and expand a central collaborative function within NCAR and a distributive network of collaborators for developing, testing, and validating numerical forecast systems important to operational decision makers and the international research community."

NCAR will facilitate a collaborative testing and evaluation function for numerical prediction systems that will maintain a neutral position between all potential developers of these systems. This will allow for a trusted, distributed facility that developers and the operational community can rely on for unbiased assessment of the operational prediction systems and potential new additions to or components of those systems. In addition it will provide a wide range of potential users including researchers and developers with access to the state of the art codes that make up these prediction systems including code access, documentation, user support and forecast evaluation tools. Generally these prediction systems will focus on Numerical Weather Prediction (including space weather) but may include other forms of automatic prediction such as expert systems.

Motivation

A long–standing goal within the numerical modeling community and particularly within the community of operational users of numerical models is to have a group that is expert in testing and evaluating all types of numerical techniques to serve in providing unbiased information to operational entities in deciding on optimal configurations for their needs, determining requirements for new research, maintaining code repositories for both research and operational users and providing support for these communities via help desks, workshops and tutorials. Additionally, developers in the research community required a more efficient method of responding to operational requirements, getting access to operational codes, and collaborating on new research work with the WRF model. The seeds for such a group were planted in 2004 in RAL with the creation of the Joint Numerical Testbed (JNT) and the transfer of the newly–created WRF Developmental Testbed Center (WRF DTC) into the JNT from ESSL/MMM. No development was allowed in the JNT except for verification and evaluation tools, and each member of the group was sworn to neutrality regarding the testing of any numerical technique. This concept has been very successful and a strong trust relationship has developed between the JNT and its sponsors.

The JNT is beginning to expand significantly now. A Data Assimilation Testbed Center (DATC) was added in 2006 and other elements are under development as discussed below. With the use of forecast models of ever finer resolution, it has become apparent that the community needs a new set of tools to evaluate the accuracy of the forecast. Additionally, it is widely recognized that forecast quality must be assessed within the context of the purpose for which the forecast is used. Thus, a rainfall forecast that is optimum for one user (for example, a watershed manager), may not be optimum for another user (for example an air traffic controller). Development of new verification tools is an activity spread across many of the RAL management units, and is also part of the JNT role.

Near-Term Objectives

WRF Developmental Testbed Center (2009–2013)

"Provide testing, evaluation and support services to all developers and users of the WRF modeling system."

NCAR (RAL and ESSL), NOAA and the Air Force Weather Agency (AFWA) will continue a collaborative testing and evaluation function for WRF model configurations that will maintain a neutral position between all potential developers. This will allow for a trusted, distributed facility that developers and the operational community can rely on for unbiased assessment of the operational prediction systems and potential new additions to or new components of those systems. In addition it will provide a wide range of potential users including researchers and developers with access to the state of the art software that make up these prediction systems including code access, documentation and user support

Actions:

2009 – Respond to requests from AFWA to coordinate procedures for testing the Gridded Statistical Interpolation (GSI) including hiring necessary staff and coordinating the formation of a single code repository to serve the WRF DTC and its NOAA counterpart at the Global Systems Division in Boulder, CO.

2009 – Evaluate hurricane intensity tests under the NOAA Hurricane Forecast Improvement Program (HFIP) test plan established in 2008 across the seven test teams. After completing the first phase deliverable in March 2009, coordinate transfer of this function to the new JNT element that will focus on all tropical cyclone model testing and evaluation (see below).

2009–2013 – Review, rank and commission new test and evaluation tasks from all sponsors of the WRF DTC based on available resources and input from the WRF DTC Advisory Board and sponsors.

2009–2013 – Provide continuous support to the research and operational communities via support desks, tutorials, workshops, visitor programs and code repositories.

Targeted Sponsors: NOAA (NWS, OAR), AFWA, NCAR base (visitor program), FAA.

Anticipated Collaborators: ESSL/MMM, NOAA GSD, NOAA/NCEP, NOAA AOML/HRD, NRL, many universities.

Specific Measurements of Success: New hires for GSI testing and evaluation on board by February 2009, completion of HFIP test report (first phase) by April 2009, GSI co–shared code repository established by October 2009, positive feedback from WRF DTC visitors, workshop/tutorial attendees.

Advanced Verification Techniques and Tools (2009–2013)

"Develop, test, and implement as a community resource a suite of advanced verification tools with an emphasis on the special needs of both the research and operational sectors."

Verification of system performance is essential to all developers and operational users. In particular, it is increasingly important to utilize verification approaches that are user–specific, and that provide information about the quality of the forecast relative to purposes for which the forecast information is used. In response to JNT and community needs, the DTC has developed and implemented the state–of–the–art Model Evaluation Tools (MET), which includes traditional and new forecast evaluation methods and is freely available to the operational and research community.

Actions:

2009 – Evaluate the Network Enabled Verification Service (NEVS) as a potential JNT service for the future including its relationship with the MET capability.

2009–2010 – Design and implement a test and evaluation function to specifically serve the space weather and tropical cyclone stakeholder communities in harmony with the two new components in the JNT focused on these groups.

2009–2013 – Extend the spatial verification effort to include a variety of other methods. Follow–up on the recommendations of the 2007 National Research Council (NRC) in their study titled "Completing the Forecast" by expanding our research on approaches that are appropriate for ensemble and probabilistic forecasts, including forecasts of probability distribution functions. Additional work will focus on techniques for appropriately comparing forecasting systems (e.g., development and application of statistical hypothesis tests and confidence interval approaches) and the evaluation of forecasts on multiple spatial scales.

2009–2013 – Coordinate with the international community through conferences, working groups, committees and tutorials to continue to assess the needs of the community that can be addressed through expansion of MET and implement these added capabilities as resources allow.

Targeted Sponsors: NSF, NASA, NOAA, FHWA, AFWA, private sector weather service providers and the industry.

Anticipated Collaborators: Meteorological Service (Canada), ESSL. (MMM, HAO), IMAGe, University of Colorado, Bureau of Meteorology Research Centre, Australia, UK Met Office, NOAA Space Weather Prediction Center, NOAA/TPC, NOAA AOML/HRD, NOAA Storm Prediction Center, NOAA Earth System Research Laboratory, Purdue Univ.

Specific Measurements of Success: Number of international users for MET and positive feedback from this user group, success in implementing MET for hurricane and space weather verification.

Data Assimilation Testing and Evaluation (2009–2013)

"Conduct tests and evaluations of various data assimilation systems to determine optimal configurations for specific circumstances."

The Data Assimilation Testbed Center (DATC) within the JNT is a separate entity from the WRF DTC but serves a similar function, focusing on testing and evaluation of data assimilation systems including the NCAR WRF VAR system and the NCEP GSI system. The DATC works closely with data assimilation components in the NOAA GSD DTC and data assimilation developers in the research community. Computing resources will come primarily from the NCAR computer though some tests may be done on computer resources available to the DTC in other facilities. Main functions of the DATC include maintenance of data assimilation systems including code repositories, testing and evaluating new data assimilation techniques or configurations, transfer of data assimilation technology to operational facilities and providing support via help desks, tutorials, workshops and code repository access.

Actions:

2009 – Respond to specific sponsors (AFWA, CAA, AMPS) to test, evaluate and maintain their data assimilation systems. Collaborate with the WRF DTC and other developers to coordinate and provide data assimilation code support to AFWA and the community.

2010–2010 – Review and improve the DATC verification package.

2010–2013 – Provide community support on data assimilation components. Review, rank, and commission new test and evaluation tasks from all sponsors of the DATC based on available resources. Establish an objective testing and evaluation system for data assimilation components.

Targeted Sponsors: AFWA, NOAA, Navy, NSF.

Anticipated Collaborators: MMM, DTC, NOAA/GSD, NOAA/NCEP.

Specific Measurements of Success: Ability to respond to data assimilation developments and growth of DATC testbeds. Build strong trust relationships with sponsors and the community. Provide data assimilation reference configuration and support to research and operational communities.

Space Weather Testing and Evaluation (2009–2013)

"Establish a new functional element in the JNT termed the Space Weather Prediction Testbed (SWPT), develop a community consensus for how it should operate, prioritize needs for technology transfer, and begin to test and evaluate space weather models for operational use."

SWPT is a proposed new facility where new space weather prediction technology is tested and evaluated for possible advancement into operations. SWPT will evaluate space weather prediction technology to provide guidance to the operational centers for improving the operational systems, and to the research community for use of the models in research applications SWPT will work with the operational centers at NCEP, and AFWA to perform tests prior to decisions for changes in the operational system. This will include verification that the changes will provide measurable improvements in the operational model through a series of tests known as pre–implementation testing

Actions:

2009 – Hold community workshop and build consensus on functions, priorities and relationships to the space weather community through a SWPT Management Plan. Also seek consensus on draft of first article to be tested and procedures for the test.

2009 – Develop an Implementation Plan for a SWPT with broad functions similar to other JNT components and funding level in the $1.0––1.5M range starting in 2010.

2009–2010 – Establish the new unit and add tasking as resources are available.

2011–2013 – Build functionality based on available resources to mature the unit and build trust within the community.

Targeted Sponsors: AFWA, NOAA NCEP/SWPC.

Anticipated Collaborators: NASA Community Coordinated Modeling Center, HAO, Boston Univ. Center for Integrated Space Weather Modeling, Univ. of Michigan Center for Space Environment Modeling, Univ. of Colorado Laboratory for Atmospheric and Space Physics.

Specific Measurements of Success: Ability to respond to the space weather community expeditiously and build strong trust relationship within the community by 2012.

Tropical Cyclone Testing and Evaluation (2009–2013)

"Form a new component of the JNT to test and evaluate tropical cyclone modeling systems consisting of: data assimilation, atmospheric model, ocean model, storm surge, ensembles and model post–processing algorithms."

NOAA's Hurricane Forecast Improvement Program (HFIP) is expanding rapidly. RAL has been asked to form a new element within the JNT to provide testing and evaluation of hurricane model configurations. This activity was initially started in 2008 within the WRF DTC to test the effect of resolution against the forecast of hurricane intensity. After the March 2009 deliverable now assigned to the WRF DTC is completed, the primary function for tropical cyclone testing and evaluation will migrate to this new unit.

Actions:

2009 – In collaboration with NOAA GSD, coordinate tests of the Flow–following fine–volume Icosahedral Model (FIM) and the GFS global models at varying resolutions over the hurricane test case set defined for the phase one tests in 2008. Complete this evaluation of model comparisons by April 2009.

2009–2013 – In collaboration with NOAA HFIP sponsors, determine test and evaluation tasking priorities, deliverable schedule and agreed on general methods. Coordinate tests, conduct evaluations and deliver final test reports.

2009–2013 – Establish a "quick response" test and evaluation function within this new component to decrease the response time for test results (preliminary version) from roughly one year to about four months.

Targeted Sponsors: NOAA HFIP.

Anticipated Collaborators: NOAA GSD, NOAA AOML/HRD, AFWA, Navy, India Meteorological Department.

Specific Measurements of Success: Growth of test and evaluation program and ability to meet timelines and deliverables as specified.

Frontiers

National Unified Operational Prediction Capability (2010–2013)

"Establish a component of the JNT to test and evaluate the next generation numerical model under the auspices of the National Unified Operational Prediction Capability (NUOPC). Components are yet to be determined, but likely will be weather, climate, space weather, embedded regional, chemical, oceanographic and biological models with associated ensembles in an Earth System Modeling Framework (ESMF)."

NUOPC is a national initiative led by NOAA, AFWA and the U.S. Navy to design and implement the next generation national prediction capability. Although not yet fully defined, the architecture for the system is likely to be global and may be built using an ESMF architecture. The JNT has been asked to participate in this initiative to define the requirements for transition from today's modeling system to NUOPC. Other elements of NCAR (CISL and ESSL) are working in a similar manner to help define the architecture and the potential for the use of global ensembles as forecast tools.

Actions:

2009–2010 – Work with and within the NUOPC committees to define transition issues and recommendations for mitigating them. Also assist in defining the necessary support functions for the forthcoming national implementation.

2010–2013 – As directed by NUOPC management and as resources allow, accept tasking for testing and evaluation of NUOPC model configurations.

Targeted Sponsors: NOAA, Navy NRL, AFWA, FAA.

Anticipated Collaborators: ESSL/MMM, CISL/ESMF Group, university partners, NSF.

Specific Measurements of Success: Ability to respond expeditiously to NUOPC committee and tasking requests, level of participation in testing and evaluating NUOPC in the JNT or by participating as a distributed element of a larger complex of test and evaluation components.