By: Steve Whitlock
The technology known as SCADA (supervisory control and data acquisition) is unquestionably one of the greatest innovations for increasing the efficiency of a water/wastewater utility. But the very nature of the technology involved in monitoring and controlling a utility’s remote operations makes it vulnerable to the ravages of natural disasters.
The 16th Special Operations Wing of the U.S. Air Force is located at Hurlburt Field. The more than 8,000 highly trained military personnel, following their motto, “Any time, any place,” specialize in conducting unconventional warfare at a moment’s notice. Not surprisingly, the wing’s philosophy of readiness extends well beyond those units involved in military operations and is evident in all departments located on the base.
Geographically situated near Fort Walton Beach in Florida’s Panhandle region, Hurlburt Field has encountered more than its fair share of tropical storms and hurricanes and 2004 was certainly no exception.
But thanks to the “ready for anything” attitude that permeates the atmosphere, water utilities personnel have learned how to minimize the damages that these natural disasters inflict on their SCADA system.
According to base civil engineer Lt. Col. William Kolakowski, the base began installing SCADA in 1995 to help manage their freshwater wells and elevated storage tanks, as well as their wastewater lift stations. Later that year, the base was on the receiving end of a one-two punch when Hurricanes Erin and Opal struck the northern Gulf of Mexico coast just a few weeks apart. The utility had an opportunity to implement and evaluate the procedures for mitigating hurricane damage. While the utility personnel were pleased by the overall results, they were aware that some enhancements were needed to further reduce potential damage.
Ready for Ivan
When Hurricane Ivan came calling in September 2004, Hurlburt Field’s water and wastewater utilities were ready.
Hurlburt Field’s SCADA system, manufactured by Data Flow Systems (DFS), is used extensively throughout the utility and is an essential ingredient in the utility’s normal operations. Remote terminal units (RTUs) installed at distant locations allow personnel to monitor and control practically all conditions and functions at 29 of their most vital wastewater lift stations.
At freshwater wells, pumps and chlorination functions are monitored and controlled. Storage tank levels are controlled to maintain system pressure. Because the DFS system is modular, offers a simplified path for upgrade and resists the pitfalls of obsolescence, the lessons learned nine years ago can still be applied today.
The RTUs in use at Hurlburt Field are of two basic designs. A stainless steel enclosure houses a backplane in the traditional models. Function modules, including radio modules, power supplies and analog and digital input/output modules plug into card edge connectors mounted on the backplane. This design is used at most of the freshwater sites as well as some of the older lift stations. Newer and upgraded lift stations use a “TAC Pack” RTU that installs directly in the control panels. The TAC Pack, along with enhanced telemetry monitor/control capabilities, is actually a fully functional pump station controller that includes many of the essential items normally found in the control panel. The TAC Pack includes HOAs, elapsed time meters and a phase monitor, resulting in a greatly simplified control panel. While the system supports network communications, the RTUs in the Hurlburt Field system communicate to the central by radio, so there is no concern about downed telephone lines or damaged communications cable.
Minimizing weather
The first steps in minimizing the operational effects of tropical weather events were taken well before the onset of hurricane season. After 1995, retention walls that had been built around lift stations in low-lying areas were raised, since Opal’s storm surge exceeded some of their heights. Control panels in these locations were raised in an effort to keep them above water level. Also, because spare parts can be the key to quick post-storm recovery, the utility conducts an annual physical inventory to assure that an adequate supply of spares are on-hand at a time when large-volume damages can occur.
When the approaching storm is imminent and the base residents have been evacuated, the “ride out team” goes to work.
From the SCADA system’s central controls, an operator shuts down the water supply to base housing. All wastewater lift stations are pumped dry, so that wet wells and mains are completely empty. As the utility supervisor explained, prepping the lift stations in this manner eliminates the possibility of sewage spills in case a tidal surge floods the lift station. Because the well and mains have been emptied of sewage, spills that occur during the storm consist only of seawater and don’t require hazardous material cleanups or reports to the Department of Environmental Protection.
The team then begins to visit each remote location. Realizing that certain types of components are especially vulnerable to the effects of storm surge, electronic equipment is targeted for removal and safe storage. After Hurricane Andrew in 1992, some South Florida utilities reported ongoing failures in telemetry units and control panels for nearly a year after the storm. Most of these failures were the long-term results of saltwater corrosion that slowly incapacitated instrumentation and electronic devices.
Hurlburt Field’s SCADA system makes it simple to prevent losses like these.
Preventing loss
The components housed in the standard RTU are removed by hand and stored in a safe location. Most RTUs can be depopulated in less than a minute. When the function modules and backup batteries are removed, only a backplane with card edge connectors remains, which can be hosed clean and dried with a common hair dryer after the storm has passed. The newer TAC Pack RTUs make the procedure even easier for the ride out team. These units can be removed, without the use of tools, in just a few seconds.
In addition, because TAC Pack RTU/pump station controllers include many of the vital electric control panel components, there is little left in the panel that is vulnerable to the effects of the storm. The use of the TAC Packs “saves a lot of damage that the control panel would usually have,” according to the utility supervisor.
When Hurricane Ivan had passed, the ride out crew took an inventory of damages. The team concentrated first on the water and power utilities, because these must be fully functional before the base populace and the aircraft can return. Hurricane Ivan’s gusts were clocked at over 130 mph at Hurlburt Field, and the storm surge topped some of the 61?2-ft retention walls that had been built around lift station locations.
Fortunately, the steps taken prior to the storm kept the most susceptible equipment out of harm’s way.
According to the utility supervisor, “Damages were very, very minimal.” Upon surveying the damage, the team first visited the SCADA central site, which was left running on generator power (some of the less vulnerable RTU locations are monitored throughout the storm) and confirmed that it was operating properly. Next, the team went to the major lift station and water RTU sites. They cleaned and dried the RTU enclosures and control panels, aligned antennas that had been misdirected by the intense winds and restarted the telemetry. The entire procedure averaged only about 30 minutes per site.
All the SCADA RTUs survived unscathed and only a few electrical control panels, whose electronic components could not be removed, received any significant damage from the hurricane. The preventive measures taken saved thousands of dollars in essential equipment. More importantly, they enabled the utility to return to normal operations in the shortest possible time, allowing other base functions to return to normal.
Some other area utilities did not fare as well.
“Some of the utilities were hit really hard,” commented the utility supervisor. Public Affairs Director Capt. Thomas Knowles confirmed that, at the request of Florida Rural Water Association, Hurlburt Field loaned 20 generators to one Panhandle utility because “they lost everything.”
The utility supervisor added that multiple DFS SCADA users in the region supported each other by making spare parts available to other DFS user utilities. “We’re all one family,” he said.
Lessons learned
It is in the nature of the men and women who work at Hurlburt Field to shun complacency and constantly seek improvement. So it is no surprise that the base continuously uses “lessons learned” to improve its hurricane preparation and recovery procedures. As a result of Hurricane Ivan experiences, 15 remote sites that currently employ emergency generator power will be augmented over the next months until all lift stations, freshwater wells and storage tanks are equipped with generator power. The SCADA system will not only enable water utility personnel to monitor and control their remote site equipment during power outages, but will interface with the generators themselves, assuring their proper operation and maintenance.
The SCADA system also will see employment outside the water and wastewater utility. Current plans include the installation of over 100 emergency generators at key locations throughout the base, enhancing various operations during power failures. Additional RTUs will be installed at these sites, endowing the same monitor and control capabilities to Hurlburt Field’s power production department.
Preparation, experience and implementation are essential components in the unique, strategic military operations for which the 16th Special Operations Wing is known. By executing the same ideology when confronted by an overwhelming force, Hurlburt Field’s utilities personnel have learned to survive and prevail in their battles with Mother Nature.
About The Author: Whitlock is vice president of customer relations at Data Flow Systems, Melbourne, Fla., and can be reached at 321/259-5009 or by e-mail at [email protected]
