Preventing data loss, whether caused by an act of nature, a hacker attack or a software upgrade gone wrong, is among an IT leader’s most important responsibilities.
When systems are down, precious research data, course materials and identity information belonging to faculty, staff and students hang in the balance.
Technology is so integral to higher education today that an extended shutdown could prevent students from completing their coursework on time, delaying their degrees and even jeopardizing student loans, says Ken Underhill, an instructor with online IT and cybersecurity education platform Cybrary.
The increasing digitization of research data makes the stakes even higher, Underhill says.
“A system outage could cause researchers to lose their data or to question the integrity of any findings from their research,” he says, potentially forcing researchers to duplicate their work.
Key to keeping data protected and accessible, of course, are backup and recovery tools, whether onsite, in the cloud — or, increasingly, both.
But even though many of these tools are automated, IT teams still should follow best practices to ensure the best performance during the worst of times.
MORE FROM EDTECH: Check out how recovery systems can protect universities from potential threats.
After Fire, Pepperdine IT Moves to Facilitate Online Instruction
For Pepperdine University, that test came in November 2018, when the Woolsey Fire ripped through the region and forced the university to halt operations at two of its five Southern California campuses, Malibu and Calabasas, on November 9.
Ultimately, the portion of the fire that affected the Malibu campus was extinguished without human casualties or significant property damage to the university, says Gerard Flynn, senior director of IT administration and client services.
But IT infrastructure that serviced the university was damaged and the campus was largely inaccessible. Remote classes were implemented to keep students on track until they could return on November 26.
Practice drills, supported by an emergency handbook, prepared IT staff to respond when two campuses halted normal operations, says Pepperdine's Gerard Flynn.
“Data accessibility was key to everything during the response to the fire,” Flynn says. “We needed to know who was located at the Calabasas and Malibu campuses and make sure that they got the appropriate messages. Secondly, we needed to know which faculty needed access to teach remotely and provide them with licensing and training on web conferencing.”
With two campuses closed, he says, “we needed to direct IT staff to report to one of our three other campuses and help in any number of ways: troubleshooting network connectivity, staffing 24-hour telephone lines and developing and educating faculty about an academic continuity plan,” Flynn says.
Another challenge: Many instructors were unfamiliar with online teaching platforms. They had to learn quickly and help students participate, all without access to equipment and materials that had been left behind on campus. Meanwhile, some staff members and students had been evacuated from their homes or lost them in the fire.
MORE FROM EDTECH: Check out these four models that are revolutionizing the way universities approach higher education.
Drills and Dry Runs Prepare IT Teams to Handle Crises Calmly
To get solutions in place, the IT team was under pressure to access systems and fix the damage remotely. It also had to license and configure a web conferencing tool that had only been pilot tested.
In a remarkable 48 hours, staff developed a website and provided training that allowed courses to continue. Pepperdine’s emergency preparedness planning proved effective under duress.
One big reason was that the university had developed an emergency handbook with procedures and contact information for staff, vendors and other stakeholders. It had also conducted drills to ensure staffers knew how to react in a crisis. “
Maintaining the emergency handbook and conducting drills were key to our response, because every staff member knew what to do and how to fill in for those severely impacted by the fire,” Flynn says.
On the infrastructure side, network rerouting was set up, and Pepperdine maintained data redundancy at a recovery site outside of Southern California.
61%
The percentage of Disaster Recovery as a Service users who reported no downtime in 2018, compared to 39 percent of non-DRaaS users
Source: Unitrends, Data Protection, Cloud, and Proof DRaaS Delivers — Unitrends 2019 Survey Results, June 2019
“Designing the network to automatically reroute should a connection fail was key to our maintaining access to data needed during the crisis,” Flynn says.
And, although it wasn’t needed during the fire, the remote disaster recovery site provided peace of mind that operations could continue if the fire worsened.
Like Pepperdine, the University of Michigan deploys a multipronged approach to disaster preparedness and data backup and recovery, including stress testing and dry runs.
With a core mission of creating and disseminating new knowledge through research, says CIO Ravi Pendse, it is critical that the university protect and archive information for future generations.
Solid backups, effective recovery tools and well-tested plans are the first line of defense. But the ability to execute them calmly in an emergency is also important, says Pendse, who is also a computer science professor.
To that end, the university’s IT department conducts stress tests, bringing systems down on weekends or after hours to measure, for example, recovery time.
It also conducts tabletop exercises, presenting team members with scenarios so they can practice their response.
MORE FROM EDTECH: Classifying campus data is key to an effective backup and recovery plan.
Communications and IT Staffers Are Integral to Disaster Recovery
It’s worth noting that the University of Michigan’s exercises include both IT and communications staff, a step that’s often overlooked.
In any disaster, frequent communications are critical, Pendse says: “It’s really important to make sure you’re communicating what you know and that you’re doing that with calmness.”
At the end of the day, people — and their ability to respond effectively to a crisis — can be the determining factor in whether recovery technologies are able to deliver the security for which they’re designed.
For the Pepperdine community, says Flynn, individuals’ grace and proficiencies made it possible to overcome the challenges wrought by the wildfire.
“The IT staff responded selflessly,” he says. “Some were displaced themselves, but worked around the clock either at another campus, borrowing equipment from fellow Pepperdiners, or from wherever they were located. The quantity and quality of work produced in a short period of time were astounding.”
