The Woolsey Fire rages in the hills behind Pepperdine University in California on Nov. 9, 2018.

Planning and Preparation Allow Backup and Recovery Tools to Shine Under Duress

Pepperdine University and University of Michigan IT staff are prepared to keep operations running, even in a crisis.

Preventing data loss, whether caused by an act of nature, a hacker attack or a software upgrade gone wrong, is among an IT leader’s most important responsibilities

When systems are down, precious research data, course materials and identity information belonging to faculty, staff and students hang in the balance. 

Technology is so integral to higher education today that an extended shutdown could prevent students from completing their coursework on time, delaying their degrees and even jeopardizing student loans, says Ken Underhill, an instructor with online IT and cybersecurity education platform Cybrary

The increasing digitization of research data makes the stakes even higher, Underhill says. 

“A system outage could cause researchers to lose their data or to question the integrity of any findings from their research,” he says, potentially forcing researchers to duplicate their work.

Key to keeping data protected and accessible, of course, are backup and recovery tools, whether onsite, in the cloud — or, increasingly, both. 

But even though many of these tools are automated, IT teams still should follow best practices to ensure the best performance during the worst of times. 

MORE FROM EDTECH: Check out how recovery systems can protect universities from potential threats.

After Fire, Pepperdine IT Moves to Facilitate Online Instruction

For Pepperdine University, that test came in November 2018, when the Woolsey Fire ripped through the region and forced the university to halt operations at two of its five Southern California campuses, Malibu and Calabasas, on November 9. 

Ultimately, the portion of the fire that affected the Malibu campus was extinguished without human casualties or significant property damage to the university, says Gerard Flynn, senior director of IT administration and client services. 

But IT infrastructure that serviced the university was damaged and the campus was largely inaccessible. Remote classes were implemented to keep students on track until they could return on November 26. 

Gerard

Practice drills, supported by an emergency handbook, prepared IT staff to respond when two campuses halted normal operations, says Pepperdine's Gerard Flynn. 

“Data accessibility was key to everything during the response to the fire,” Flynn says. “We needed to know who was located at the Calabasas and Malibu campuses and make sure that they got the appropriate messages. Secondly, we needed to know which faculty needed access to teach remotely and provide them with licensing and training on web conferencing.”

With two campuses closed, he says, “we needed to direct IT staff to report to one of our three other campuses and help in any number of ways: troubleshooting network connectivity, staffing 24-hour telephone lines and developing and educating faculty about an academic continuity plan,” Flynn says. 

Another challenge: Many instructors were unfamiliar with online teaching platforms. They had to learn quickly and help students participate, all without access to equipment and materials that had been left behind on campus. Meanwhile, some staff members and students had been evacuated from their homes or lost them in the fire.

MORE FROM EDTECH: Check out these four models that are revolutionizing the way universities approach higher education.

Drills and Dry Runs Prepare IT Teams to Handle Crises Calmly

To get solutions in place, the IT team was under pressure to access systems and fix the damage remotely. It also had to license and configure a web conferencing tool that had only been pilot tested. 

In a remarkable 48 hours, staff developed a website and provided training that allowed courses to continue. Pepperdine’s emergency preparedness planning proved effective under duress. 

One big reason was that the university had developed an emergency handbook with procedures and contact information for staff, vendors and other stakeholders. It had also conducted drills to ensure staffers knew how to react in a crisis. “

Maintaining the emergency handbook and conducting drills were key to our response, because every staff member knew what to do and how to fill in for those severely impacted by the fire,” Flynn says. 

On the infrastructure side, network rerouting was set up, and Pepperdine maintained data redundancy at a recovery site outside of Southern California

61%

The percentage of Disaster Recovery as a Service users who reported no downtime in 2018, compared to 39 percent of non-DRaaS users

Source: Unitrends, Data Protection, Cloud, and Proof DRaaS Delivers — Unitrends 2019 Survey Results, June 2019

“Designing the network to automatically reroute should a connection fail was key to our maintaining access to data needed during the crisis,” Flynn says. 

And, although it wasn’t needed during the fire, the remote disaster recovery site provided peace of mind that operations could continue if the fire worsened.

Like Pepperdine, the University of Michigan deploys a multipronged approach to disaster preparedness and data backup and recovery, including stress testing and dry runs. 

With a core mission of creating and disseminating new knowledge through research, says CIO Ravi Pendse, it is critical that the university protect and archive information for future generations

Solid backups, effective recovery tools and well-tested plans are the first line of defense. But the ability to execute them calmly in an emergency is also important, says Pendse, who is also a computer science professor. 

To that end, the university’s IT department conducts stress tests, bringing systems down on weekends or after hours to measure, for example, recovery time. 

It also conducts tabletop exercises, presenting team members with scenarios so they can practice their response. 

MORE FROM EDTECH: Classifying campus data is key to an effective backup and recovery plan.

Communications and IT Staffers Are Integral to Disaster Recovery

It’s worth noting that the University of Michigan’s exercises include both IT and communications staff, a step that’s often overlooked. 

In any disaster, frequent communications are critical, Pendse says: “It’s really important to make sure you’re communicating what you know and that you’re doing that with calmness.”

Gerard Flynn
Every staff member knew what to do and how to fill in for those severely impacted by the fire.”

Gerard Flynn Senior Director of IT Administration and Client Services, Pepperdine University

At the end of the day, people — and their ability to respond effectively to a crisis — can be the determining factor in whether recovery technologies are able to deliver the security for which they’re designed. 

For the Pepperdine community, says Flynn, individuals’ grace and proficiencies made it possible to overcome the challenges wrought by the wildfire. 

“The IT staff responded selflessly,” he says. “Some were displaced themselves, but worked around the clock either at another campus, borrowing equipment from fellow Pepperdiners, or from wherever they were located. The quantity and quality of work produced in a short period of time were astounding.”

John Davis
Sep 23 2019

Sponsors