IT leaders offer insights about how to best protect systems should disaster strike.
Ridgewood High School I.T. Director Sam Lewis knows he can’t leave his tech infrastructure unprotected from power outages and power surges.
Teachers at the 900-student Norridge, Ill., campus rely on their tablet computers every day to wirelessly beam multimedia presentations onto their projectors. If a power outage knocked out the wireless network, it would immediately bring class lessons to a halt and force teachers to ditch their prepared lessons. Worse, power fluctuations can abruptly shut down or fry IT equipment, resulting not only in downtime but also in the potential for lost data and expensive repairs.
Lewis proactively safeguards his IT equipment with uninterruptible power supplies, which offer surge protection and battery backup. If it’s a momentary outage, UPS systems keep everything up and running until the problem is resolved. If it’s a prolonged outage, the battery backup gives IT staffers enough time to safely shut down servers and computers until power returns.
“We have invested in a lot of expensive equipment, and we can’t always trust the power coming in. So for protection, we rely heavily on UPS systems,” Lewis says.
Weathering the Weather
During the summer, heat can strain power grids and cause blackouts and brownouts as people dial up their air-conditioning. Hurricanes, tornadoes and lightning can also knock out power or create surges or sags. To protect IT equipment, school districts must not only invest in UPS devices, but also develop a comprehensive data backup and recovery strategy that includes everything from storing data in tape libraries to building backup data centers.
About 2,000 thunderstorms are occurring worldwide at any one time, producing 100 lightning strikes per second. The United States faces 15 to 20 million ground strikes a year, with Florida averaging the most strikes.
Source: National Lightning Safety Institute
Last year at Ridgewood, Lewis replaced the school’s aging six-year-old UPS with a new American Power Conversion InfraStruXure system that includes an APC Symmetra unit, a 20-kilowatt solution that gives the 14-server data center one and a half hours of battery backup. Lewis also recently purchased lower-end APC Smart-UPS 1500s for his six wiring closets to protect new networking equipment.
For disaster recovery, Lewis backs up data to two tape drives using Symantec’s Backup Exec software. The district’s storage area network also has fail-over, so if servers go down, the data is still accessible in the remaining servers. Lewis also plans to look into off-site disk-based storage.
IT administrators say the technology is expensive, but worth the investment. For example, the West Des Moines School District in Iowa recently bought a new Powerware 9170+ to protect the district’s network backbone and phone system (a Cisco Systems switch). The IT staff also invests in UPS devices to protect servers and lower-cost networking equipment.
“I haven’t calculated return on investment, but it’s a necessary evil,” says Scott Crothers, network engineer for the district. “It’s just something I have to do. I’d rather be prepared than have a power fluctuation blow out a piece of equipment.”
To help you protect equipment and data from power failures and other disasters, these IT practitioners offer the following best practices.
1. Don’t overload a UPS.
If you connect 1,500 total watts of equipment on a 1,000-watt UPS, it will overload the system and may not work, Crothers warns. In fact, you should purchase a UPS that can handle more wattage than is currently needed in case the data center expands in the future, he says.
1,630 tornadoes hit the United States in 2008, causing 125 deaths. It was the 10th most deadly year since reliable recordkeeping began in 1953.
Source: National Climatic Data Center
Crothers suggests figuring out how much battery life you want, then doubling that number to be safe. For example, when he bought the Powerware 9170+ for his district’s network backbone, he wanted 20 minutes of battery life — the amount of time it takes to boot his phone system. He decided to shoot for 40 minutes, but he overestimated and wound up with two hours of battery life. That’s fine, he says,because it’s always safer to have too much battery life, rather than too little.
To fine-tune the amount of battery backup and type of UPS you need, ask a vendor to send an electrical engineer to calculate the amount of power your equipment consumes, Lewis says.
2. Use a generator as backup.
This will keep operations up and running during a long power outage. Lewis connected his generator to the APC Symmetra device, so if power goes out, the UPS will carry the load until it automatically switches over to generator power after about 15 seconds.
Rule of Thumb:
Figure out how much battery life you want, then double that number to be safe.
— West Des Moines School District’s Scott Crothers
Photo Credit: Scott Sinklier
3. Consolidate your devices.
Consolidate multiple, small UPS systems into one larger, high-end UPS in the data center. Not only is one UPS easier to manage, but a high-end model offers more features, including longer battery life and an LED screen that shows how much power the equipment is drawing, says Jim Mattson, senior network engineer at Killeen Independent School District in Killeen, Texas. A few years ago, Mattson consolidated about 20 smaller APC UPS devices into one APC InfraStruXure system.
“It’s easier to manage one APC from one central graphical user interface,” Mattson says. “Aesthetically, it’s very clean. The room is not full of wires.”
4. Take advantage of remote administration tools.
UPS vendors provide management software for users to remotely monitor the devices and shut down equipment if needed, says Brian Hoyt, network administrator of Roland Park Country School in Baltimore. Hoyt bought network management cards for his APC UPS devices, so if power goes down, the devices alert him over e-mail.
Rule of Thumb:
Run a live test to find out the true battery life your UPS will supply in your environment.
— Roland Park Country School’s Brian Hoyt
APC also sells network management cards that monitor environmental factors, such as temperature. If it reaches a threshold that could lead to equipment failure, the devices can also be programmed to e-mail IT administrators, he says.
It’s also important to connect the UPS units to the network because the equipment can self-test and alert you if there are any problems with batteries, Lewis continues.
5. Do a test run.
When the UPS is installed, test the battery to find out how much battery life you actually have, Hoyt says. He initially performed a rough calculation on how much power his phone system would draw, based on the manufacturer’s documentation. He believed he would get 16 hours, but he actually got only four.
To determine battery life, APC’s web software lets the UPS pretend a power failure has occurred. It runs on the battery until there’s only 5 percent of the charge left and then flips back to regular power.
6. Choose disk over tape.
Crothers believes disk-based storage is better than tape because the backup and restore times are much faster and more reliable. Crothers purchased an EMC Avamar solution with 6 terabytes of hard-drive space. Backups that used to take 14 hours on tape now take just two to three hours, he says. In addition, restoring files used to take 15 to 20 minutes. Now, with the disk-based backup system, restores take just two to three minutes.
7. Buy low-end UPS systems for desktops.
Most school districts will focus their UPS purchases on more expensive technology, such as servers and networking equipment. But if funds are available, it’s worthwhile to purchase small, entry-level UPS devices for desktop computers and peripherals. The UPS devices can protect everything from computer monitors to hard drives and increase a computer’s longevity, Mattson says.
“We save money in the long run because the more times a machine ends up in the repair shop, the more it costs the taxpayer.”
Severe weather damage to U.S. property in 2006:
- Lightning: $63.8 million
- Tornadoes: $752.3 million
- Flash floods: $2.1 billion
- Tropical storms and hurricanes: $2.4 million
$11.7 billion - Total damages to U.S. property and crops from severe weather in 2006
Recent Hurricane Damage
- Hurricane Ike (2008): $13 billion to $21 billion (estimated)
- Hurricane Katrina (2005): About $125 billion, making it the most expensive natural disaster in U.S. history
- Hurricane Wilma (2005): $16 billion
- Hurricane Rita (2005): $16 billion
- Hurricane Andrew (1992): $27 billion
Sources: Risk Management Solutions; National Climatic Data Center