Response Teams Stay Ready for Potential Campus Emergencies

When Richard Crim installed a network switch on a mobile trailer’s lightweight wall, he remarked, “I hope the trailer is earthquake proof.” A week later, an unusual earthquake hit Virginia, and Crim now hesitates to joke about the unexpected.

As the rare, magnitude 2 temblor hit, the trailer held its own, and the facility remained unharmed, according to Crim, who works as an information technology specialist at Lord Fairfax Community College, near Washington, D.C.

Still, engineers needed to evaluate the damage, so IT staff steered clear of the main data center. Nearby at a sister campus, a data center had to close for weeks because of structural damage.

In an instant, Crim’s lighthearted comment became very real.

“I thought I was joking, because earthquakes never happen around here, but without warning, we had to find a way to operate without entering our facilities,” Crim says.

Crim routinely brainstorms with his emergency response team about new threats. The team is continuously prepared for service attacks, hurricanes, intruders and even the bird flu.

Fortunately, earlier preparations for other potential disasters kept the college running. The school’s IT technicians used smartphones and the remote access applications LogMeIn and TeamViewer to assign passwords and manage enterprise applications and email. This kept students and staff productive during the setback.

“We were in the parking lot with our phones, doing everything we normally do at our desks,” Crim says.

Threat Management

Colleges and universities typically face a variety of threats that may jeopardize operations and safety, including natural disasters and man-made emergencies. As a result, many schools strive to preserve operations in an emergency’s aftermath.

Doing so represents a change in priorities for some educational institutions, according to Shawn McCarthy, research director at IDC Government Insights. Citing budget constraints, McCarthy notes that higher education historically underspends for continuity of operations compared with other sectors, including financial services. This is gradually changing, he adds, as universities and colleges evaluate today’s risks.

“Continuity of operations is all about risk management and how much you spend to manage threats across the organization,” McCarthy says. “It’s especially important to have a continuity plan to protect the financial records of a university and the learning systems.”

As for Crim, remote access is just one instrument in his toolkit to keep vital areas up and running. Web connectivity is critical to higher education, so he contracts for duplicate Internet services to complement the in-ground fiber connections that support essential email and hosted website resources. A local cable provider installed the second connection using above-ground lines.

“Now, if an earthquake or a backhoe ruptures the buried fiber, we’ll stay connected to the Internet,” Crim says. “Similarly, if a storm takes down the aerial lines, we’ll still have our original connection. We made sure that the two are fully redundant and don’t follow the same physical path to a central office.”

Lord Fairfax Community College also takes advantage of having four campus locations: the Fauquier and Vint Hill campuses, both in Warrenton, Va.; the Luray–Page County Center in Luray, Va.; and the Middletown campus in Middletown, Va. All but Vint Hill, the college’s newest campus, form a triangle with endpoints a few hours apart.

VMware vSphere provides server virtualization, so if one data center crashes, its services automatically trigger a failover response to one of the remaining sites.

“We don’t have to invest in duplicate hardware for disaster recovery or worry about copying backups to an alternative site,” Crim says.

Rewritten Rules

For some institutions, an extensive commitment to public cloud services rewrites strategies for continuity of operations and disaster recovery.

“Traditional disaster recovery is essentially dead within the context of fully virtualized, cloud- orchestrated solution models,” says Peter Young, senior vice president of analytics, planning and technology at the University of Maryland University College (UMUC), based in Adelphi, Md., near Washington, D.C. “We moved past planning for traditional disaster recovery to engineering for resiliency.”

The institution relies almost exclusively on Infrastructure as a Service and Software as a Service solutions to support administrative and educational resources for 85,000 students in 27 countries. Most of the students perform coursework exclusively online, so the college must provide accessible learning materials across a wide range of time zones.

“Trying to provide that level of service out of our own data center is not a tenable option,” says Jason Reed, vice president and chief technology officer at UMUC. “In a global environment, disasters happen somewhere almost all the time. For that reason, we re- engineered the whole notion of disaster recovery and business continuity.”

UMUC’s strategy relies on having hosted services running throughout the world, says Young. If a resource becomes unavailable at one data center, the workloads automatically shift to an alternative location.

“The distributed nature of hosted clouds means we can spread our risk to avoid single points of failure,” Young says.

The strategy eliminates staff time used on redundant hardware and software, and on the failover systems’ maintenance and tests.

Reed adds, “Now, our role is to manage the service-level agreements with our cloud providers and ensure that their disaster recovery and business continuity plans are up to date and properly certified.”

Protection Partners

While online universities such as UMUC think globally about uptime, some more traditional colleges act locally to protect operations.

Rebecca Sandlin, CIO at Roanoke College in west central Virginia, disagrees that failover sites should be in far-flung locations as a hedge against regional outages. Sandlin believes the optimum location is off campus but still within an easy drive for IT staff during a disaster.

She found that balance through a partnership with the college’s home city, Salem, which maintains what Sandlin considers a state-of-the-art data center a mile from campus.

“It’s far enough away to protect against many of our most likely disruptions, but close enough to rush over in an emergency,” she says.

Sandlin’s staff manages Cisco UCS blade servers virtualized with VMware vSphere. The college’s Tegile Systems storage combines high-performance, solid-state drives and spinning disks and continuously backs up information to paired appliances in the city’s data center. This ensures students and staff can continue their work without loss of valuable materials during an unexpected outage.

The college also maintains redundant fiber lines to the city’s data center as an additional safeguard when failovers occur. If an emergency impacts one connection, traffic automatically reroutes to the remaining pipeline.

Sandlin feels confident that these components can keep Roanoke College operative in an emergency, but cautions that technology alone is not enough.

“We test the system periodically to ensure everything is working as expected,” she says. “We also perform data restores. They’re a pain to do, but they’re the only way to be sure everything will work when we actually need it.”