AutoNowcaster
The primary role of the nowcast environment is to collect weather data, execute algorithms for producing a combined thunderstorm forecast, and provide a graphical display tool for viewing the various datasets.
Data incorporated into the nowcast environment includes WSR-88D (NEXRAD) radar, satellite, sounding, and surface data.
The software applications in the nowcast environment include algorithms for identifying and tracking thunderstorm movement, identifying boundaries, wind retrieval from radar, as well as a fuzzy-logic engine which allows the user to combine the weighted outputs from the various algorithms to produce a single, combined forcast. Subsequent verification of generated forcasts are available both visually and statistically.
In realtime operations the nowcast environment initiates and maintains process control through an auto-restart mechanism. The auto-restarter provides the user with an automated, hands-free forecasting environment.
Many components of the nowcast environment also can be operated in archive mode. Archive mode allows the user to re-run algorithms with newly specified parameters and to view historical datasets.
The key hardware components of the Auto-Nowcast Environment are:
- A collection of UNIX workstations on a local area network
- A mechanism for archiving data
The key software components of the Auto-Nowcast Environment are:
- Data ingest
- Data distribution
- Process control mechanism
- Analysis algorithms
- Graphical data displays
The key data components of the Auto-Nowcast Environment are:
- Doppler radar data
- Sounding data
- Surface data
- Satellite data
Other features include:
- System monitor
Field Projects
|
Project |
Year |
Sponsor |
Operational Site Location |
|
RAPS |
1995 |
Federal Aviation Administration |
Northeastern Colorado |
|
Memphis |
1996 |
Federal Aviation Administration |
Memphis, Tennessee |
|
SCAN |
1997, 1998, 1999, 2000 |
National Weather Service |
Sterling, Virginia |
|
ATEC/WSMR |
1997, 1998, 1999, 2000, 2001, 2002, 2003 |
U.S. Army |
White Sands Missile Range, New Mexico |
|
ATEC/Aberdeen |
1998 |
U.S. Army |
Aberdeen Proving Ground, Virginia |
|
ATEC/RTTC |
1999, 2000 |
U.S. Army |
Redstone Technical Test Center, Alabama |
|
Summer Olympics 2000 |
1999, 2000 |
World Weather Research Program |
Sydney, Australia |
|
2002, 2003 |
Federal Aviation Administration |
Lincoln Labs at Massachusetts Institute of Technology |
Getting Started
Logging onto the Auto–Nowcast Machines
Because the Auto–Nowcast Environment involves many complex algorithms, the power of many machines or hosts are combined to make up the entire environment. All machines in the Auto–Nowcast Environment are set up to operate under the "nowcast" user login. See your site administrator for the password for this login. Some of the Auto–Nowcast Environment machines are configured with auto–login at boot–up some are not, depending of the needs of the particular machine and the local area network configuration at the site. In an auto–login setup, booting the machine will automatically result in a login session as user "nowcast" and an will initiate an X–windows session. On machines which are not configured for auto–login at boot–up, the user must manually login as user "nowcast" and start X–windows using the following commands:
| login: | nowcast |
| password: | (see site administrator) |
| startx |
Once the X–windows session is started (either manually or via auto–login) the user will see a console window for system messages, an xterm for running UNIX commands, and a window manager tool bar. (look on reflect...)
Starting up the Auto–Nowcast Environment
The Auto–Nowcast Environment can be managed by any host through command line program execution; however, an X interface with pull–down menus is available on the control host. The control host can be determined by typing the command:
echo $CONTROL_HOST
The realtime Auto–Nowcast Environment is started from the control host either by typing the command:
niwot_startup
or via a pulldown menu on the $CONTROL_HOST:
It only takes a few minutes for all of the Auto–Nowcast Environment process to start up. Several graphical windows will appear on the control host at startup. Each of these windows and their role in the Auto–Nowcast Environment will be discussed in detail below. Because each of the algorithms within the Auto–Nowcast Environment requires different amounts of input data for their processing, the results of the algorithms will not begin to arrive until the Auto–Nowcast Environment has been up and running for a while. Some algorithms will produce data within six minutes, others will not produce data until there is significant weather.
Shutting Down the Auto–Nowcast Environment
The realtime Auto–Nowcast Environment is shutdown from any host by typing the command:
niwot_shutdown
or via a window pulldown menu on the $CONTROL_HOST:
Operations Guide
Operations Guide (for downloadable pdf, click here)
- Hardware
- Software
- Environment Config
- Data Display
- Algorithms
- Installation
- Monitoring
- Data Architecture
- Procmap and the Auto–restarter
References
The Auto-Nowcast Environment is a product of the following body of research:
- Bankert, R.L., 1994: Cloud classification of AVHRR imagery in maritime regions using a probabilistic neural network. J. Appl. Met., 33 (8), 909-918.
- Dixon, M., and G. Wiener, 1993: TITAN: Thunderstorm identification, tracking, analysis and nowcasting - a radar-based methodology. J. Atmos. Ocean. Tech., 10 (6), 785-797.
- Knight, C.A., and L.J. Miller, 1993: First radar echoes from cumulus clouds. Bull. Amer. Met. Soc., 74 (2), 179-188.
- Mueller, C.K., J.W. Wilson and N.A. Crook, 1993: Utility of soundings and mesonets to forecast thunderstorm initiation. Weather and Forecasting, 8 (1), 132-146.
- Purdom, J.F.W.: Subjective interpretations of geostationary satellite data for nowcasting. Nowcasting, K. Browning, Editor, Academic Press, 149-166.
- Tuttle, J.D., and G. Brant Foote, 1990: Determination of the Boundary Layer Airflow from a Single Doppler Radar. J. Atmos. Ocean. Tech., 7(2), 218-232.
- Crook, N.A., and J.D. Tuttle: Short-term Forecasting of Summer Precipitation using Echo Extrapolation, Storm Characteristics and Model Output.
- Crook, N.A., and J. Sun: Forecasting Storm Growth and Decay using Low-level Radar Data and the Adjoint Method.
- Crook, A., 1994: Numerical simulations initialized with radar-derived winds. Part I: Simulated Data Experiments. Mon. Wea. Rev., 122 (6), 1189-1203.
- Crook, A. and J. D. Tuttle, 1994: Numerical simulations initialized with radar-derived winds. Part II: Forecasts of three gust-front cases. Mon. Wea. Rev., 122, 1204-1217.
Additional related research includes:
- Delanoy, R. L. and S. W. Troxel, 1993: A machine intelligent gust front algorithm for Doppler weather radars. Preprints, Fifth International Conference on Aviation Weather Systems, Vienna, VA.
- Gould, K., C. K. Mueller and J. W. Wilson, 1993: Rules for short-term forecast of thunderstorm initiation and evolution: procedures and automation. Preprints, 13th Conference on Weather Analysis and Forecasting including Symposium on Flash Floods, Vienna, VA.
- Henry, S. G., C. Mueller and J. W. Wilson, 1996: Base-line statistical evaluation of the automated thunderstorm nowcaster. Preprints, 15th Conference on Weather Analysis and Forecasting, Norfolk, VA.
- Henry, S.G. and J.W. Wilson, 1995: The automatic thunderstorm nowcasting system. Preprints, 27th Conference on Radar Meteorology, Vail, CO.
- Henry, S. G., 1995: Evaluation of automated, short-term, thunderstorm forecast rules. Preprints, 6th Conference on Aviation Weather Systems, Dallas, TX.
- Henry, S. G. and J. W. Wilson, 1993: Developing thunderstorm forecast rules utilizing first detectable cloud radar-echoes. Preprints, 5th International Conference on Aviation Weather Systems, Vienna, VA.
- Henry, S. G., 1993: Analysis of thunderstorm lifetime as a function of size and intensity. Preprints, 26th International Conference on Radar Meteorology, Norman, OK.
- Nelson, E., W. Deierling and C. Kessinger, 2009: Utility of total lightning measurements for very short term forecasts of convection growth and decay at White Sands Missile Range. 4th Conference on the Meteorological Applications of Lightning Data, Amer. Meteor. Soc., Phoenix, AZ, 12-15 January 2009.
- Mueller, C. K., S. G. Henry, et.al., 1997: The Memphis ITWS convective forecasting collaborative demonstration. Preprints, 7th Conference on Aviation, Range and Aerospace Meteorology, Long Beach, CA.
- Mueller, C., T. Saxen, R. Roberts, J. Wilson, T. Betancourt, S. Dettling, N. Oien, and J. Yee, 1993: NCAR Auto-Nowcast System. Weather and Forecasting, 18 (4), 545-561.
- Rinehart, R. E., 1979: Internal storm motions from a single non-Doppler weather radar. NCAR/TN-146+STR, 262 pp.
- Saxen, T.R., C.K. Mueller, T.T. Warner, M. Steiner, E.E. Ellison, E.W. Hatfield, T.L. Betancourt, S.M. Dettling, and N.A. Oien, 2008: The Operational Mesogamma-Scale Analysis and Forecast System of the U.S. Army Test and Evaluation Command. Part IV: The White Sands Missile Range Auto-Nowcast System. J. Applied Meteorology and Climatology, 47 (4), 1123-1139.
- Sun, J., and N. Andrew Crook, 1997: Dynamical and Microphysical Retrieval from Doppler Radar Observations Using a Cloud Model and Its Adjoint. Part I: Model Development and Simulated Data Experiments. J. Atmospheric Sciences, 54 (12), 1642-1661.
- Sun, J., and N. Andrew Crook, 1998: Dynamical and Microphysical Retrieval from Doppler Radar Observations Using a Cloud Model and Its Adjoint. Part II: Retrieval Experiments of an Observed Florida Convective Storm. J. Atmospheric Sciences, 55 (5), 835-852.
- Wilson, J. W., and C. K. Mueller, 1993: Nowcast of thunderstorm initiation and evolution. Weather and Forecasting, 8, 113-131.
- Wilson, J. W., and W. E. Schreiber, 1986: Initiation of convective storms by radar-observed boundary layer convergence lines. Mon. Wea. Rev., 114, 2516-2536.
Frequently Asked Questions
Network Communications
Q. Will the radar data degrade the network performance of my LAN?
A. No. The Auto–Nowcast Environment exists on two separate local area networks. The radar data is broadcast on a dedicated LAN which does not interfere with the LAN on which your other machines reside.
Q. What should I do when I cannot login to one of the Auto–Nowcast Environemnt host machines?
A. If the host which contains the shared project space is down then it will not be possible to login to any other Nowcast hosts. This is because the .cshrc for all hosts are linked to the .cshrc in <shared project space host>:$PROJ_DIR/.cshrc.If you can't login to any nowcast host, try rebooting the host of the shared project space.
If the problem is isolated to one host, try rebooting that host.
Data Ingest & Management
Q. What should I do when a dataset is late or missing?
- A. Scream.
- If recent data files are not being written to disk in the data directory location, check to see if the data disk is full.
- If it is a NEXRAD radar dataset which is missing, make sure the radar data are being broadcast on the ethernet or via LDM.
- If everything looks ok up to this point, then the process responsible for generating this particular dataset may be having problems. First, determine the responsible process by examining the input/output tables in sections on Analysis Algorithms and Data Ingest. Once you have identified the responsible process, follow the diagnostic procedures for a missing or restarting process.
Q.How can I be sure that radar data is coming into a Auto-Nowcast host?
A.First, check the raw radar data directories directories to see when the last data files arrived. A second check can be run on the hosts which have the dual ethernet cards installed for receiving the radar data. This check requires root login privileges.
To find out whether a host has 2 ethernet connections (expected: eth0 and eth1), run the command:
ifconfig -a
If 2 ethernet connections are not listed, this host is not directly receiving the radar data feed so this check is not valid. If both eth0 and eth1 are listed, look for which one has "Bcast" set to xxx.xxx.199.xxx or xxx.xxx.197.xxx. This is the port which would be receiving radar data. To check that data is actually arriving on that port, run the command (for eth0, replace eth1 below with eth0):
tcpdump -i eth1 udp port 3280 If radar data is arriving, you will see a data stream similar to the following:
16:35:06.810338 128.117.199.255.3280 > 128.117.199.255.3280: udp 1450
16:35:06.810338 128.117.199.255.3280 > 128.117.199.255.3280: udp 998
16:35:06.820338 128.117.199.255.3280 > 128.117.199.255.3280: udp 1450
16:35:06.820338 128.117.199.255.3280 > 128.117.199.255.3280: udp 998
If radar data are not being broadcast on the ethernet port, you will need to check on communications upstream of the Auto-Nowcast Environment.
If you are receiving radar data via an LDM then you can type ldmadmin watch to see if bundles of radar data that are being received for a particular radar ID. The output of ldmadmin watch looks like this:
Oct 13 20:34:50 pqutil: 11144 20031013203446.553 NEXRD2 125000 L2-BZIP2/KIND/20031013203501/125/0
Oct 13 20:34:52 pqutil: 12277 20031013203448.979 NEXRD2 181020 L2-BZIP2/KLOT/20031013203139/181/20
Oct 13 20:35:01 pqutil: 12226 20031013203457.569 NEXRD2 181021 L2-BZIP2/KLOT/20031013203139/181/21
Oct 13 20:35:09 pqutil: 11593 20031013203506.282 NEXRD2 181022 L2-BZIP2/KLOT/20031013203139/181/22
Oct 13 20:35:10 pqutil: 11322 20031013203506.248 NEXRD2 125001 L2-BZIP2/KIND/20031013203501/125/1
Oct 13 20:35:14 pqutil: 8206 20031013203511.688 NEXRD2 181023 L2-BZIP2/KLOT/20031013203139/181/23
Q. What happens to the Auto-Nowcast Environment when a data feed goes down?
A. Because so many processes in the Auto–Nowcast Environment rely on the radar data feed, very few processes will continue to generate results if the radar feed goes down.
If other data feeds go down, some algorithms which rely on those data may output "missing data" datasets, while others will wait for the input data to become available before producing any output.
In general, the Auto–Nowcast Environment will continue to run, and when data become available again the algorithms will resume outputing meaningful results.
How much data is generated each day?
This will vary depending on the type and number of datasets at your installation.
Q. What should I do when one of the data partitions becomes full?
A. If the "/home" or cross–mounted shared data partition disk is 100% full, you should check that the events list is not saving so many cases that the data scrubber cannot free up sufficient disk space. If this is the case, the older cases should be archived and removed from $DATA_HOME/params/events.list.
If the "/" (a.k.a. "root") partition is 100% full, you must reboot the machine.
Process Control
Q. What should I do when a process is restarting or missing?
A. Following are some actions to take for diagnosing restarting or missing processes:
- Make sure the application executable exists in $RAP_BIN_DIR or $RAP_SHARED_BIN_DIR
- Make sure that the application instance in the process table matches the instance in the application start script.
- Examine the application log file.
- Run the process startup script manually to see what errors are printed out. To run the process manually first kill the process by running the appropriate kill script, then in an xterm, re–start the process using the appropriate start script.
To figure out the correct kill and start scripts for the process, examine the Host and Script specifications listed in the sections on Analysis Algorithms and The Project Directory.
NOTE: Be sure to kill and re–start the process on the correct host. The logical host is specified either in the start script or in the corresponding parameter file. The physical host is then determined using the UNIX echo command. For example: echo $TITAN_HOST
Q. What happens to the Auto–Nowcast Environment when a process in the procmap is restarting continuously or is missing?
A. The process will not be able to generate its expected output. This may also affect other "downstream" processes which need that process's output.
In this situation, it is important to diagnose the source of the problem. For information on trouble shooting a process which is failing, see the discussion above.
Q. What is the difference between Mode 1 and Mode 2 process restarts?
A. Mode 1 process restarts are due to a process missing from the procmap. This occurs if a process exits prematurely.
Mode 2 process restarts are due to a process not registering frequently enough with the procmap (check the Heartbeat time in the procmap window). In this case, the process is believed to be "hung" and is stopped and restarted by the auto–restarter.
Q. What does it mean when I get the message "No more processes" or "Cannot exec" or "Cannot fork" when I try to run a command?
A. This usually indicates that the host has run out of processes, i.e., the process table is full. In order to run the command you need to free up some process space. If you are running extra processes, which are not part of the Auto–Nowcast Environment, try to exit them. If that does not free up sufficient process space, try running the command exec top.You can start terminating processes until you can begin executing commands in the xterm. By examining the output of top you may also be able to determine which processes are filling up the process table and diagnose the problem causing the table to fill up.
Q. The Auto–restart log for the previous few days is empty. Is this a problem?
A. The auto–restarter only outputs to the log file when a process is restarted. An empty log file indicates that no processes were restarted, the system is up and fine.
Configuration
Q. Can I rearrange the location of datasets or add new machines to the Auto-Nowcast Environment?
A. Yes. In order to relocate processes and datasets to a new machine the following must be edited:
- process lists in $CONTROL_DIR/proc_list
- host data lists in $DATA_HOME/data_lists
- URLs in param files for applications which access the data.
It may be necessary to do a host_mkdata to install new data param files and do host_shutdown and host_startup to install new runtime process lists. Also you must restart all processes which access the data.
Q. Can I get software upgrades?
A. When the Auto–Nowcast Environment is installed at a field site, a snapshot of the application software is stored on disk so that any software problems that arise can be debugged with a version of the software which is running in the field. In the meantime, these same applications undergo changes at NCAR as a part of the ongoing effort to improve the software.
Depending on the magnitude of the changes that have taken place in the software at NCAR, an upgrade for any one application may be a simple matter of taking a new snapshot and copying the executable to a field machine. However, upgrading an entire Auto–Nowcast Environment may require reconfiguring the Environment to incorporate enhancements to the operating system, networking, interprocess communication, and process control.
In order to maximize stability in the Auto–Nowcast Environment, we do not install full software upgrades during an operational season. A full upgrade and reconfiguration typically takes four weeks of NCAR staff time.