The K-12 Survivor's Guide to Disaster Recovery Planning

Behind these numbers, there was chaos. The U.S. Department of Homeland Security set up camp at the Poway Unified School District office in San Diego, "so we had to move," recalls Robert Gravina, the district's chief technology officer and CIO. "But we didn't know where we were moving." District officials ultimately rebuilt their emergency operations center at city hall and were functional within three hours.

Poway USD has had a disaster recovery plan in place since 2003, and everyone is clear on the priorities in case of emergency. But Gravina and his team don't spend much time preparing for the worst. Rather, they prepare for the best. "Some people spend a lot of money on disaster preparedness, and they may never have a disaster," he says. Instead, "spend it on best practices that you can use every day. They can keep your network stable, and then they're there in an emergency."

Those who have faced and recovered from disasters both natural and manmade have learned the hard way where to focus their attention – and what holds up in an emergency. Here are the top lessons they've learned.

Evacuate the Danger Zone

On Sept. 11, 2001, as the world was gripped by the shocking images of planes crashing into the World Trade Center's Twin Towers, students just blocks away were being evacuated from Stuyvesant High School and from PS 234, The Independence School.

Getting people to safety should be every district's top priority, says IT consultant Irwin Kroot, who was with the New York City Department of Education from 2003 to 2006 and served as CIO for nearly two years. Each New York City school has a safety plan, and the DOE's IT department maintains a centralized database with the key data from each school's plan, spelling out where
students and staff will go in an emergency.

Spread the Word

Parents in New York City can have emergency information sent to them via voicemail, e-mail or text. Automated emergency notification "wasn't a well-developed-enough technology five years ago, but now it's ubiquitous," Kroot says. "Even on a shoestring, that's something that any school district should be able to manage."

The DOE also disseminates emergency information through the city's 311 phone service.

Protect Your Data

In May 2007, a tornado wiped out 95 percent of the infrastructure of Greensburg, Kan. It destroyed all of the town's schools and buses and leveled many homes.

The ceiling in Staci Derstein's office caved in, which proved to be a blessing in disguise. The debris fell in such a way that it shielded her notebook computer, as well as a server stored in her office, from the elements. The server stored student information, explains Derstein, principal of Kiowa County Elementary and Junior High School, so the district was able to recover the data, which was also backed up offsite.

Poway USD regularly conducts both digital and tape-based backups, which are stored offsite. The district also virtualizes 90 percent of its servers and operates a private cloud, where everyone can access applications, databases and more, regardless of their physical location. Virtualization cuts costs and reduces maintenance on a daily basis, but it also provides for quick data recovery in an emergency, Gravina says.

"You aren't dependent on your network as much because the applications are on servers in the cloud," he explains. "These technologies help in an emergency, but they're also best practices for any IT organization."

The quality of the district's data is one of its most significant achievements, Gravina adds. Poway USD operates dozens of databases, which feed into a single "Staging Database" that pulls information from an authoritative source and updates it each night. Of the 275,000 messages that were sent out during the wildfires, only 15 phone numbers were wrong. "If you don't have clean data, you aren't going to be able to communicate with your end users," he says.

But don't go overboard with storage, warns Bill Rust, a research director at Gartner. "Are you backing up data that isn't essential?" he asks. Schools shouldn't waste time and money on "a brute-force backup of everything in the world."

Map Out Your Infrastructure

As Hurricane Katrina barreled toward the Gulf Coast in August 2005, officials in Mississippi's Gulfport School District moved computers to high shelves. But when the storm had passed, the Technology Support Services building, which housed the network infrastructure, was flooded with seven feet of water. "It was obliterated," recalls Terri Burnham, who was the district's technology director at the time but has since retired.

Superintendent Glen East asked district officials to come in two days later to help with the recovery efforts, but the Technology Support Services building was a shell. "I didn't even know where to go," Burnham says. "You don't plan for what happens if you lose everything."

The superintendent instructed the staff to outline what needed to happen to get the district up and running again. "For weeks, we brainstormed," Burnham says. "We couldn't even remember all that we had." But the exercise taught a valuable lesson: "Document everything."

Burnham says the district lost its passwords list and protocol information. "You don't even need to lose the network," she says. "If you lose a network engineer and don't have the passwords, you might not be able to get back in." But, she adds, if you have that information documented in a disaster recovery plan and you know your priorities, you can proceed in a logical fashion.

Have Mission-Critical Equipment at the Ready

Like Gulfport, Kiowa County Schools lost just about everything in the natural disaster that devastated its community. Staff paychecks were scheduled to go out just two weeks later, however, so getting a computer and accounting software to the clerk responsible for payroll became a top priority.

The lesson, Kiowa County's Derstein says, is that a strong disaster recovery plan should list mission-critical hardware and software, a budget and the names of all staff responsible for purchasing equipment.

Gravina says Poway USD has developed agreements with manufacturers who will make needed hardware and software available within hours. The district also has a backup generator so it can operate that equipment.

The New York City DOE, meanwhile, has established alternate headquarters at various locations in the city's five boroughs. The school department uses each facility on a regular basis, so the infrastructure is already in place. If an emergency strikes, that facility's usage can be modified so the DOE can set up its alternate headquarters there. The district also maintains a Synchronous Optical Network (SONET) ring that ties together all of its sites – including the alternate locations – so users can access critical data and applications.

"The business continuity piece is really important," Kroot says. "You make not only a Plan A and a Plan B, but [also] a Plan C and Plan D, just in case."

Identify People You Can Count On

When the wildfires raged throughout the San Diego area, more than 300 Poway USD families, including 40 staff members, lost their homes. You can't expect someone whose house is on fire to come into the office to restore an e-mail system. So the district cross-trains staff to ensure that multiple people know how to perform any given task, Gravina says.

It's also critical, Derstein adds, to identify leaders. "Who will be in charge of which aspects of the recovery effort?" she asks.

Gulfport School District's Burnham got help from a neighboring district that had been spared by Katrina. Karen McManus, the technology coordinator for the Long Beach (Miss.) School District, helped Burnham gather information and complete Gulfport's application for E-Rate funding. "You get lost in the details," Burnham says. "You have to have someone mentor you and help you get through the shock and keep you focused, and [McManus] was it for me."

Be Patient

"Disaster recovery isn't going to happen quickly," Burnham says. "I wanted to quit so badly, but I told [district leaders] I would stay on until it was finished."

The new building opened in August 2010 – five years, almost to the day – after Katrina. The satisfaction of having seen the reconstruction through to the end "is a big reason why I retired," she adds.