Always Prepared: College IT Departments Get Ready for the Worst
If the day ever comes when a monster blizzard slams into New England, a fire erupts or any other emergency occurs, Tufts University officials say their campus is more prepared than ever.
The campus emergency alert system will immediately blast out phone calls, email and text messages to students, faculty and staff — and even to students' parents — providing news and important instructions.
The Tufts IT department already provides remote access to applications and data, so if its three Boston-area campuses should ever need to close temporarily, administrators and staff can continue to work from home, and use web-based collaboration tools to hold meetings. Students can continue to access educational materials and communicate with their professors through a web-based learning management system (LMS).
"You can't always predict what will occur, but we feel we are fairly equipped for as many contingencies as we can prepare for," says Dawn E. Irish, director of communications and organizational effectiveness at Tufts Technology Services.
Natural and man-made disasters at colleges and universities can — and frequently do — disrupt education, research and other important functions, such as IT, food, housing and healthcare services. When buildings are damaged, important administrative or student records can be lost.
Many colleges and universities, including Tufts, have made significant efforts to create and improve emergency operation plans, detailing how key officials will respond to and protect their campus community and property from any type of emergency that might arise. From an IT perspective, that includes building redundancy throughout the IT infrastructure and ensuring the availability of computer systems, applications, data and communications.
"Business continuity and disaster recovery are priorities; not just IT systems and services, but also communication with students, the administration, faculty and parents," says Theresa Regan, Tuft's director of enterprise infrastructure.
Improving Continuity of Operations
Over the past three years, the Tufts Technology Services team has modernized the aging data center on the Medford, Mass., campus, investing in new blade servers, a new storage area network, VMware virtualization software and new networking equipment.
The IT department has also leased space for a secondary data center in downtown Boston, where the IT staff has deployed backup IT infrastructure. When and if the main data center goes down, the backup data center can take over and keep IT operations running, Regan says.
Both data centers were designed with resiliency in mind. In the past 18 months, the IT staff has virtualized 90 percent of applications, including email, the LMS, enterprise resource planning (ERP) applications and websites. Virtual servers, databases and other data are automatically replicated to the backup data center, she says.
Each data center also is equipped with an uninterruptible power supply system, a backup generator and redundant connections to Internet service providers. Web application servers were deployed in a round-robin setup to allow for load balancing. If one server goes down, the others will pick up the workload.
"We want to ensure that our users never see an outage," Regan says.
Tufts — like many modern systems — is a global university, so 24/7 availability is critical to its students' and faculty members' success. Students and researchers worldwide must interact with information as well as their colleagues and peers, as though they are based on the main campus. If a disaster or some other emergency occurs and the campus must be closed for a day, or for several days, faculty and staff can work remotely by accessing applications and data over the web or through a virtual private network, Irish says. They can take advantage of desktop video conferencing software to hold meetings and use collaboration tools to share desktops and collaborate on documents. Using the LMS and other web-based tools, faculty can continue to teach and interact with students. Students can still access class files and upload homework.
"We've expanded the ability for people to access Tufts resources, regardless of what might be going on," Irish says.
U.S. 2013 Billion-Dollar Weather and Climate Disasters
The National Oceanic and Atmospheric Administration's National Climatic Data Center keeps track of all severe weather or climate events that have great economic and societal impacts. The U.S. has sustained 151 weather/climate disasters since 1980 where overall damages or costs exceeded $1 billion. The total costs of those events exceed $1 trillion. In 2013, NOAA recorded seven weather and climate disaster events with losses exceeding $1 billion.
Emergency Alerts
As for alerting 15,000 students, faculty and staff, Tufts administrators have used the campus emergency alert system to announce power outages and other emergencies. They also used the alert system last year after the Boston Marathon bombing. Administrators suspended all nonessential services on campus and kept the campus community updated with information and instructions, Irish says.
When an emergency happens, administrators can send short text messages, voicemail and email directing people to the university's website for more information. The university also uses Twitter and Facebook to deliver critical information. Tufts first standardized on a mass notification system in 2007, but upgraded to a new cloud-based system last summer because it was more user friendly and delivers messages more efficiently to the entire campus community.
"We wanted to make sure we can communicate in seconds," Regan says.
The cloud-based provider also offers continuity of operations benefits, Regan says, with IT infrastructure in multiple cities. If a major disaster occurs in Boston, the service will stay up and running, allowing administrators to continue to communicate with the campus community.
Digitizing Records
At the University of Oklahoma, one assistant dean is digitizing her college's paper records as another means of disaster preparation. Her lesson was hard learned.
One night in late 2009, a major snowstorm caused a leak in the roof of the three-floor administrative building within the university's College of Arts and Sciences. Water leaked onto the first floor, flooding the northeast portion, destroying old curriculum records and damaging current student scholarship submissions.
"The water drained for four hours before it set off an alarm. When I walked in, there were two to three inches of water and a huge mess," says Rhonda Dean Kyncl, assistant dean of the College of Arts and Sciences.
After Kyncl and her staff dried out the scholarship applications with large fans, most were salvageable and readable. The college had to contact the students who submitted those that weren't, and asked them to resubmit.
The water damage brought to light the need for a better record-keeping system, Kyncl says, so that spring, she met with the dean about digitally archiving everything.
The college deployed an enterprise content management (ECM) system in the fall of 2010, and migrated from an all-paper to an all-digital environment. With 7,000 students, the college has a lot of records.
"It's saved us a lot of money in printing and paper costs, and it saves us time in filing documents. We no longer do that," she says.
The move also saved physical space, with two former file storage rooms now used as academic offices. All records are stored on a server and backed up twice, once to another campus server and then to cloud-based storage offsite. Consulting a workflow designer, Kyncl looked at each paper form and process before designing an electronic workflow for each, including student records that detail meetings with advisers, check sheets that measure progress toward a degree, and class withdrawal forms that Kyncl must approve. Today, everything is done electronically.
"Withdrawal forms now take a quarter of the time they used to take," she says. "I can make the decision on the ECM system, and it generates an email to students with comments on the decision, and it's immediately sent to student records."
Safety from Tornadoes
The University of Alabama is improving its own disaster recovery plans by investing in a second data center.
In April 2011, the university was in the process of building a backup data center 200 miles away in Atlanta when a deadly tornado ripped through Tuscaloosa, destroying hundreds of buildings and homes along its path. The tornado just missed the Alabama campus, but it cemented the need for a secondary data center, Deputy CIO Scott Montgomery says. Besides tornadoes, fires, flooding and vandalism could all cause the main data center to suffer downtime, he says.
"If your data center is rendered inoperable for a month or two, you are closing your school down for a semester," he says. "When you add up the direct revenue lost, the reputational costs and the long-term ramifications for closing down, the costs are enormous."
Alabama's IT department did finish a secondary data center in a colocation facility in Atlanta. IT staff identified the university's most important applications — email, online learning, ERP and student recruiting and registration software — then set them up in a hot-site environment.
"The applications and data are replicated routinely," he says. "When a failure occurs, we just apply the latest logs to the databases to start the applications in the colocation facility."
To build resilience into the wide area network, the IT team also added a redundant network connection from an IT equipment room on the main campus to Atlanta.
The main data center houses 450 virtual servers, while the colocation facility houses half the number of virtual servers and about 250 terabytes of replicated storage.
The backup data center, which cost about $5 million and became operational last May, is well worth its price thanks to the peace of mind it brings. "It's like purchasing insurance," Montgomery says. "There is a risk-benefit calculation you have to go through. The probability of something happening is low, but the impact should something occur is very high. We looked at that and felt we had better set up a disaster recovery capability."
