The Education CIO's Guide to Recovering From Disaster

It was Sept. 17, 2004, and Ivan had just finished a two-day romp through Pensacola, leaving $12 million in damages to the university. In one building, desktop computers that staffers had wrapped in plastic and set under desks to keep “safe” were now wading in four inches of water. While the building's roof withstood high winds, the leaky windows and torrential rains flooded the room. More than 50 computers were submerged.

“You would pick up the computer and literally water would pour out of it,” Dieckmann recalls. It was just one of the many lessons learned about how to protect sensitive equipment, systems and computers, in addition to dealing with backup sites and communications systems. Even guidelines for reporting claims to the Federal Emergency Management Agency (FEMA) were revisited.

Just 10 months later, on July 10, 2005, Hurricane Dennis struck UWF. But this time the staff was better prepared. Today, the university continues to make disaster recovery improvements as administrators face the bulk of the hurricane season, which runs through November.

When it comes to disaster planning, West Coast universities face their own set of demons: earthquakes, mudslides, wildfires and other disasters that usually lack advance warning.

The devastating 1994 earthquake in Northridge, Calif., followed by the 6.9 magnitude quake a year later in Kobe, Japan, served as a wake-up call for IT administrators at the University of California, Berkeley.

The university's 250 buildings had been rated structurally sound to withstand an earthquake. But the two major disasters showed that those ratings were “way too optimistic,” explains Jack McCredie, associate vice chancellor and CIO at UC Berkeley. The campus computing and operations center, for instance, was located in the basement of Evans Hall, a 10-story building constructed in the 1960s. Prior to 1995, it had been rated as “fair” by structural engineers, meaning it could withstand a “pretty good shock,” McCredie explains. But after reports from Northridge and Kobe showed widespread brittle fractures in welded steel beam-to-column connections, the building was downgraded to poor, meaning a modest quake could cause major structural damage.

“We're located a half mile from the Hayward Fault, so a modest event there would render the computing infrastructure of the campus unusable,” he says.

Rather than risk disaster, McCredie began an eight-year odyssey to build a structurally sound, leading-edge computing center for the university. In July 2004, the computing and operations center moved into the third floor of a reinforced three-story building known as 2195 Hearst.

“We're now confident that when some event happens, we have the kind of facilities that will get through it,” McCredie says.

Whether dealing with a hurricane, tornado or earthquake, CIOs know that recovering IT and communication systems after a disaster means more than just salvaging equipment. Colleges and universities that can't recover technology systems quickly risk losing students because of missed classes or lack of equipment. The best defense is to have a plan before disaster hits because, as UWF's Dieckmann warns, it's hard to get creative during a disaster. “You want to go on autopilot.”

Here are some disaster planning tips from those who've learned them firsthand:

BEFORE THE STORM

Identify a Remote Backup Location

Ivan taught Dieckmann that even newer buildings could succumb to hurricane-force winds and caused him to rethink the practice of housing all his 120 servers in a single data center. He now plans to open a second data center off campus for backup systems and some production systems. “With new high-speed networks, and Internet 2 plus national Lambda rail [a high-speed national Ethernet-based network that runs over fiber-optic lines], you now have the network bandwidth to get the same performance as if it were on your own campus,” he says.

At the University of Miami, which survived Hurricane Andrew in 1992, CIO M. Lewis Temares makes sure the university's Web site is always available by backing up the site at a facility in Atlanta. If on-campus systems go down in a disaster, administrators can still provide news and instructions to its six campuses via its Web site.

Many backup systems that keep Web sites running and phone lines working are designed to run for several hours using a generator after an electrical failure. Ivan taught several university CIOs a big lesson–that campuses could end up going weeks without power.

At Brevard Community College (BCC), a Florida campus that survived a one-two punch by hurricanes Francis and Jean in September 2004, the clock was ticking on its IT systems after power went out. Using an emergency power supply that only lasts a few hours, Director for Data Services Tony Awtonomow had only enough time to take down systems in an orderly fashion. “We knew we couldn't communicate with the outside world–we couldn't even do a Web site. We had to rely on the media to communicate with students and staff,” he explains. Internet access was sporadic because power was spotty, he added.

After Jean, administrators made some changes. Instead of continuing to rely on one campuswide generator that used 1,000 gallons of diesel fuel every 14 hours, they invested $135,000 on two small, fuel-efficient generators for the telecom closet and the university's main server room.

Advise About Simple Safety Measures

Indian River Community College in Fort Pierce, Fla., suffered back-to-back disasters last September when Hurricane Francis hit, followed by Jean three weeks later, which together caused $7 million in damage to its four campuses.

Despite the short time between emergencies, the college made a few changes to its disaster plan. “We built emergency kits for every building with flashlights, plastic bags and tarps for computers and equipment, and tools,” explains Mike Gibbons, director of network support. The IT mantra for faculty and staff: “Cover it and keep it high.” Each building was also assigned a building manager to secure equipment and to stay in the building during a storm.

In most cases, the simple preventive measures worked. At the Bryan Administration Building, with offices for 48 administrators and staff, wind and water damage rendered the building a complete loss, but CIO Barry Keim's computer and telephone survived. “They were on my desk covered in plastic,” he says.

Prepare for FEMA and Insurance Claims

While FEMA quickly responds and provides aid to universities and businesses after a disaster, the federal agency's guidelines for granting financial relief are stringent. UWF's Dieckmann says it's good to know in advance the documentation you'll need to claim losses. For instance, claimants must take pictures of damaged equipment in its location before they start to remove it, he says, and be ready to explain how equipment was damaged, such as by rain, wind or flooding. “It's best to have templates in place ahead of time, so as teams are going around, they know what documentation they're expected to collect,” he adds.

DURING THE STORM

Keep Communication Open

Most CIOs say communication takes top priority during a disaster, whether the contact involves emergency response, parents, students or staff.

After Ivan, UWF administrators were hampered by dead phone lines and downed cell phone towers. So Dieckmann's team invested in five satellite phones for key administrators. For the rest of the staff, he purchased cell phones from multiple vendors. “That's important because a particular carrier might be having problems where others are not,” he explains.

Tracking the whereabouts of thousands of students during a disaster also presents a technology challenge. UWF built a temporary Web site where students and faculty could check in and post their whereabouts.

At the University of Miami, administrators can send recorded messages to the home phone numbers of all 30,000 students, parents and employees before and after a disaster. These automated calling services, usually hosted by an application service provider, provide e-mail and SMS text messages to cell phones, and also acknowledge that the messages were received.

AFTER THE STORM

Deploy Only Essential IT Personnel

Campus buildings can be dangerous after a hurricane–and not just from a structural standpoint. Faculty and administrators eager to recover their data can crash systems and risk further damage to equipment.

Dieckmann advises administrators to keep the campus closed after a big hit and bring back only the staff critical for the reopening. “The most critical staff to get back immediately are your network and systems people for basic infrastructure–in our case that's about a dozen people,” he says. “The more people you have on campus while you are trying to [restore systems], the more difficult it is.”

The help desk staff should return once infrastructure is restored. “Everybody else at the university who is trying to get their operations back online will need help from your help desk,” he says. Finally, the classroom technology staff can restore those systems before classes resume.

Yet IT leaders can't automatically count on 100 percent participation from IT staff immediately after a storm. CIOs must give employees time to address their own personal property issues, says BCC's Awtonomow. After Francis and Jean, “We split-shift our IT labor force,” he explains. “While one shift was working, the other was working on their home plans. You get around-the-clock coverage, while letting employees take care of their personal stuff.”

Restore Essential Systems First

One of the most important things that the University of Miami learned during Hurricane Andrew in 1992 was to get payroll systems up and running immediately. “Make sure people have money so they can buy what they need to board up their own homes, so that they can come to work. They have to take care of their own problems first,” Temares says.

Think Beyond IT

Keep the building infrastructure in mind, including something as mundane as carpeting. “It can cause a lot of disruptions [if you have to] move equipment while replacing carpet. For us, after Ivan, that was a huge amount of work,” says Dieckmann.

Salvage Systems Carefully

Remember the waterlogged computers left on the floor at UWF? Amazingly, 45 of the 50 soaked PCs were salvageable. “If you're careful to dry [PCs] out before powering back on, it's amazing how resilient some of these can be,” says Dieckmann.

Be Realistic on Timing

While the IT staff at UWF made many recommendations for disaster planning improvements after Ivan, not all of them were implemented before Dennis struck. “Funding becomes an issue,” Dieckmann explains. The secondary data center will trigger $100,000 to $200,000 a year in additional costs, he notes. Also, with the widespread structural damage caused by Ivan and other recent storms, construction contractors are at a premium.

“Our ultimate goal is: If campus was totally shut down, those core communication systems and the real key business systems like finance and HR would still be operational from some other site over the Internet,” Dieckmann says.