Anticipating Disaster

Floods, high winds, electrical storms and snowstorms take a daily toll on just about any higher education IT department, producing power outages, physical damage to devices and facilities, and staffing shortages. Some university IT operations are too familiar with extreme conditions, such as hurricanes, wild fires, earthquakes — even volcanic activity.

Universities in locations that get more than their fair share of calamity have found that virtualization, reliable power sources, remote recovery sites and alert notification systems are more than just words on paper. These technologies all play a part in getting their schools’ IT operations through the roughest of times. Disaster recovery, they say, isn’t a one-time plan; it’s a continual process.

Surprise, Surprise

When Hurricane Katrina roared through New Orleans on Aug. 29, 2005, the impact on Tulane University at first seemed relatively minor. But then the levee protecting the city crumbled, the water rose and the rest is history. The school’s uptown campus didn’t reopen until January 2006.

“What surprised us most in the Katrina disaster was that our traditional means of communicating — cellular phones and the university e-mail system — weren’t available to us,” reports Adam Krob, director of the university’s end-user support.

Tulane’s data center, located on the 14th floor of a downtown building, remained high and dry and suffered no damage. However, the building had no power, and the floodwater destroyed the connection to campus.

“One thing we learned is that you really need not just a backup site but a real production system with seamless fail-over for things like e-mail and web communications,” Krob says.

Tulane now uses VMware to virtualize its servers so processes can be easily replicated in different physical locations; has a commercially hosted backup hot site; and contracts with service providers for redirecting its e-mail server in the event of a failure. Besides providing fail-over capabilities, virtualization also reduces the number of total servers required and lowers hardware costs and electricity usage.

Trial by Fire

Threatened by earthquakes and wildfires, the University of San Diego (USD) uses a combination of mirroring capabilities, physically separate server facilities around campus, remote storage facilities and emergency-notification capabilities.

USD implemented an emergency-notification service in February 2007, and in October 2007 was able to warn students and faculty to stay away from the campus during a series of wildfires that swept through Southern California.

“The timing was fortuitous, as many other universities scrambled to put something together,” remembers USD Vice Provost and CIO Christopher Wessells.

In light of the surrounding risks, servers are distributed across four data centers around campus, allowing for redundancy and fail-over capabilities. The university is working to bolster such redundancy by maintaining copies of all applications and databases on a remote storage-array device that can be drop-shipped anywhere at a moment’s notice.

USD is also considering reciprocal “safe harbor” agreements to trade data center space for servers with geographically distant counterparts who are unlikely to deal with the same types of disaster.

“It’s cheaper than contracting with the large data center DR providers like SunGard for a backup site that we may never use,” says Wessells.

Trouble in Paradise

Garret Yoshimi, director of technology infrastructure for the University of Hawaii (UH) has seen his fair share of extreme conditions (see sidebar). “A particular disaster is not going to be one of the scenarios in your plan,” he warns. “Disasters don’t cooperate like that. You need the flexibility to respond to the event that is actually in your face.

“Our focus is on incrementally improving the resiliency of our infrastructure and building in as much redundancy as possible,” he adds. Even the most resilient network is useless without power — a fact that recent flood and earthquake events underscored.

Following a 2006 earthquake-generated power failure, the university found itself in the dark. “We didn’t have an emergency generator then and couldn’t provide continuous power to our data center,” recalls Yoshimi. UH now has a 500-kilowatt generator. “The campus is now able to withstand a fairly substantial power outage, with the data center remaining fully operational,” reports Yoshimi.

UH overlays not one, but two hosted third-party notification services on top of an emergency alert system based on its own e-mail system.

Yoshimi says that assigning particular functions to individual staff members can cause problems. UH cross-trains its staff to perform many functions to ensure that someone with the appropriate skills will be available.

Backup in Florida

In 2007, Florida International University (FIU) in Miami embarked upon a three-year plan to build a backup data center in a hurricane-resistant bunker in Tallahassee, located in the northern panhandle of the state. The budget for the first year covered the university’s enterprise resource planning (ERP) system. Other critical systems were set to be added when the new fiscal year began in July. FIU hopes to have a thoroughly replicated data center completed by the end of the 2009–2010 fiscal year.

“We tried the ‘safe harbor’ approach first,” reports Min Yao, FIU’s CIO. “But everyone we talked to already either had a partner or a contract with a vendor.” The universe of appropriate schools is actually quite limited because they have to be the right size and in the right place. “We have 38,000 students and lots of records, which would be too much of a burden for smaller universities. But I do recommend looking into reciprocal agreements because they can be very cost-effective if you find the right partner,” adds Yao.

In any case, Yao concludes, regular planning sessions that review processes and contact information “are much more important than the technologies themselves.”

Wildfire Fuels Emergency Notification

Located in the foothills of Southern California’s San Bernardino Mountains, California State University, San Bernardino (CSUSB) is in the direct path of the notoriously hot, dry Santa Ana winds that fuel the wildfires that have burned down several buildings on the CSUSB campus in years past.

Last October, the university made it through the latest round of devastating wildfires relatively unscathed. But the wildfire experience, combined with a general mandate to improve emergency response, prodded CSUSB to beef up its emergency notification capabilities, says CIO Spencer Freund.

CSUSB decided to leverage its existing Cisco VoIP infrastructure by implementing CDW Berbee’s InformaCast, which is based on the Cisco Call Unified Call Manager server.

The InformaCast system pushes an audio stream and/or a text message to multiple IP phones, IP speakers, desktop clients and overhead paging systems within the campus. Administrators can select a prerecorded message or send a live broadcast through either a password-protected web page or the IP phone-services menu.

Recovery Plans

Armed with more robust communications capabilities, these five universities have increased confidence that they can keep students safe and minimize disruption when the inevitable disasters strike.

All recommend overlaying multiple notification systems that exploit every available modality, and subscribing to third-party services that can maintain communications when the local infrastructure goes down. They have learned to treat disaster preparedness as a constant assessment process. Disaster plans are just a starting point; real-world situations rarely fall in line with them.