Disaster Recovery 101: How to Upgrade After a Storm

Disaster recovery always takes a high priority for IT professionals in the Midwest, but there was no question that we needed to design an effective DR plan following the devastating tornado that hit Joplin, Mo., in May of last year.

The storm, which took 158 lives and caused roughly 1,000 injuries, was the deadliest tornado in the United States since 1947.

Many at Pittsburg State have close ties to nearby Joplin. Before I came on board as CIO at Pittsburg State, I was CIO of Joplin Schools for 11 years, so I was well aware of the design of Joplin's IT infrastructure. Also, Tim Pearson, PSU's assistant director of infrastructure and security, is chief of the volunteer fire department in Carterville, Mo., a bedroom community of Joplin.

Keep in mind that although Pittsburg State is just 30 short minutes across the Kansas–­Missouri border from Joplin, it didn't sustain any damage that Sunday. But many of our colleagues living in nearby Joplin faced tragedy that evening.

Being that close helped us understand the gravity of the situation and the kind of DR plan needed for our campus.

After contributing to the Joplin Schools IT restoration efforts and interviewing Traci House, director of technology at Joplin Schools, we discovered that it's the common-sense things that make a difference. For example, we had phone numbers for our team members listed and distributed among IT leadership but realized that we didn't have the cell numbers of our major service providers and primary vendors, including our Internet service provider and networking manufacturer.

We also learned that as the Joplin Schools team started receiving equipment for restoration, they had no place to put it because of the lack of buildings left standing in the area. We realized that in an emergency we would need to lease a building where we could store IT equipment and set up a temporary command center. Knowing this, we reached out to a local realtor and added that firm's contact information to our DR plan.

Following the storm, we also stocked up on hard hats, flashlights and goggles. That way, our IT team could arrive at the DR site in the proper apparel and with basic recovery tools. We also realized that all staffers need to make sure they have their university ID cards on them — at all times. In a disaster zone, it might be necessary for members of our team to show workers and volunteers from the Federal Emergency Management Agency that they belong at the DR site.

Based on our experiences, here are some best practices we recommend for institutions when updating their DR plans:

Decide who should serve on the DR planning team.

For starters, include the CIO, director of infrastructure and security, network administrator, system administrators, database administrators, physical plant representatives, computer technicians, telecom specialists and procurement officers. You may also want to include partners that deliver cloud-based services.

A team member from the physical plant staff is especially important because if you have to build something makeshift, you will need their expertise. It's also important to work out an arrangement with the procurement staff so that if an event occurs, there's a policy for raising the dollar level of purchases. This will allow the response team to procure the equipment needed to get the new network in place.

Identify a plan and specific contacts for restoring mission-critical equipment and services.

With the increased popularity of mobile devices, include a plan for restoring wireless service to mobile users. While observing how Joplin worked so diligently to regain connectivity, it was evident that the sooner wireless access was established, the sooner the school district could regain a sense of normalcy.

At Pittsburg State, one of our unique majors is automotive engineering. PSU hosts an international mini-Baja racing ­competition on a remote Baja track that's not near an established wireless infrastructure. By providing support for the program, we've gained the expertise and equipment needed to establish wireless access in unique locations.

After working with Joplin Schools, we also realized the need to make provisions for backup equipment. Some of this equipment is affordable to keep on hand, but much of it is not. In an emergency we will work closely with our providers to use their demo equipment in the short term to establish connectivity.

Assign a specific date on the calendar for when the DR plan will be updated.

PSU's DR team meets twice a year to update the DR plan. These dates are added to each DR team member's calendar and typically occur in January and July. The plan must take into consideration any recent changes in the organization's infrastructure, including virtual servers and client virtualization, as well as public or private cloud services. For example, we just deployed a new e-mail and calendaring groupware system, and this critical information had to be added to the DR plan.

Determine where the IT staff will assemble if the institution's data center is destroyed.

We identified two different sites where we will meet in the event of a disaster. If both sites are destroyed, we'll meet virtually in the interim through the collaboration site ­chatter.com. One advantage of using the site is that it's possible to run a session on a mobile device through cellular service.

If the disaster is of such a magnitude that neither of the identified sites is available for a face-to-face team meeting, then we will meet at a site designated by the president's office, which will be widely publicized by radio, television and the Internet.

Decide on the messages the university will send out to the campus community, as well as the format.

Typically, most official campus messages come from the marketing and communications department on campus.

During the aftermath of the tornado, Joplin Schools had great success communicating via Facebook, Twitter and YouTube. We plan to use the same avenues, as well as a secure group share at chatter.com for internal DR team messaging. Each of these social media sites can be accessed via a mobile device using cellular service.

Understand that, especially in the first hour or two following a disaster, a great deal of the initial information is inaccurate, so be very clear with your IT team on who's authorized to speak for the university and on which communications platforms.

Plan to engage your brain.

Last, heed the advice of Tim ­Pearson, who says that a DR plan is a guide, not a cookbook. Organizations may have to change direction as events evolve and try things that aren't in the formal DR plan — and that's OK.

Don't expect a plan to give the team every answer. It should simply remind the group to ask the right questions as they begin the restoration process. There is no step-by-step list of directions out there for every disaster. Ultimately, the most critical components in restoring an institution's IT infrastructure are willing hearts and flexible problem-solvers.