Systems Survival Guide

The solo IT practitioner at Geneva Area City Schools works to build an infrastructure that will carry on through crashes and calamities.

“We tried to recover the server and couldn't. We had to start over with new equipment,” says Sidoti, who is the sole IT professional in the district, which has six schools  and serves almost 3,000 students from Geneva and several other communities in Ashtabula County, Ohio. “I went to the powers that be, and they looked at me and said, ‘How much is it going to cost?' It was an easy sell. It wasn't a ‘what if” scenario – we were in the middle of it. All our systems had been affected.”

Seeing the Big Picture

Sidoti did not propose a band-aid for the damaged systems. Looking beyond the emergency, he wanted to use the calamity as a catalyst for changes that would strengthen the district's IT infrastructure and help minimize the impact of equipment failures and other crises in the future. In Superintendent Mary Zappitelli, Sidoti had an ally who clearly understood the importance of continuity of operations (COOP) for both the educational mission and administration of the schools.

“Information technology is an integral part of running the district,” says Zappitelli. “On the administrative side, if our computers go down, all we can do sometimes is sit and look at each other.”

“In the classroom, if you want people to use the technology, it's got to work,” Zappitelli explains. “When technology is unreliable – that's what turns people off. The server disaster brought us to our knees, but what Scott has done will prevent the same sort of thing from happening again.”

With the go-ahead from administration, Sidoti purchased four HP ProLiant DL380 servers, three to run the applications that had resided on the scorched main server. The fourth server would support a new centralized backup solution, consisting of a CA ARCserve backup software suite and a Sun StorageTek SL24 tape autoloader connected to the server by an HP SC08Ge HBA storage controller.

The new system was designed to provide backup for all of the district's systems, including Windows 2008 R2 Microsoft Hyper-V, existing Novell networking and the newly installed Cisco VoIP phone system.

The CA software suite provides flexible backup that supports any storage medium or configuration. The Geneva district uses it with a Fujitsu Eternus iSCSI-based storage area network Sidoti had purchased the previous summer to provide easy access to and quick recovery of files from mission-critical applications.

The Recovery Process

The equipment arrived the day after Sidoti placed the order, and the grueling process of restoring information and rebuilding systems began. The main server had been connected to a tape backup system located in the same room, so there was no chance to restore from tape. The server itself was beyond repair, but Sidoti, with the help of a consultant, was able to recover almost all the data from its hard drive and migrate it to the new ProLiant servers.

As an added complication, the version of the Novell Netware network operating system used by the district was nearly obsolete. Sidoti, who is Microsoft-certified, had been contemplating a move to Windows well before the primary server cooked and crashed. It made more sense to invest in Windows Enterprise software and licenses immediately after the disaster than to restore the existing operating system and then replace it within a year, he says.

“I spoke to an outside consultant who told me it would take two months to get the Windows system up and running,” Sidoti says. “That just wouldn't work for us, and I decided to do it myself. In 16 days we migrated to Windows, and when the teachers came back they were able to function with all their files and data.”

“I had in-house help that didn't cost us anything,” Sidoti continues. “Five or six people from the district volunteered to help me and worked through the winter break. One high school teacher, Jarrod Burgard, put in more than 100 hours.”

Sidoti configured the Windows 2008 R2, including Active Directory, policies and security features and then had help from the volunteers in implementing the new operating system.

Be Prepared

Despite the lost server and the recovery process that lasted through winter break and beyond, Geneva Area City Schools is ahead of many school systems in planning for continuity of operations, according to Gartner analyst Roberta Witty. Relatively small organizations like school districts often start thinking about continuity of operations only in the wake of a catastrophe or serious system failure, says Witty.

“Because of horrible events like shootings and natural disasters, K–12 districts now have systems in place to send out emergency messages. But that doesn't mean they have a plan to keep operations going,” Witty says. “If there's no plan in place, restoring systems after a disaster comes with tremendous costs in terms of effort and expense, and in delays of important projects. The best advice is to build recovery into everything you do in IT, rather than thinking about it later.”

That's what Sidoti had been trying to do in the year before the December systems crash. The technology he found in the district was first-rate, he says. But, as a long-time networking consultant before he joined the school system, Sidoti recognized design defects and vulnerabilities that threatened to hobble the district's IT operations or bring them to a halt.

“The network and equipment here had loads of potential, but the systems fell short of what they could be and should be,” Sidoti says. “Good equipment only gets you a third of the way. You also have to think about design – your vision of the future – and then implementation.”

Uplifting Upgrades

When Sidoti arrived, Geneva had gigabit fiber running between its buildings, with Cisco switches throughout the network and a Cisco firewall. But the network was configured without any segmentation, so a problem anywhere was likely to spread to other locations and systems. As a step toward containing future system failures, Sidoti has segmented the high school and junior high portions of the network.

Software for essential systems, such as special education and professional development, had in the past been installed on designated PCs in each school in the district. If the designated computer crashed, one or more teachers lost access to vital information. To remedy the situation, Sidoti installed two application servers running Microsoft Hyper V virtualization software to replace the services previously running on PCs.

Staff can now log in to any PC and download the programs they need from the central server. The strategy provides increased failover for the systems as well as increased access for teachers and administrators. As part of the project, Sidoti purchased the Fujitsu SAN to ensure quick recovery of files should one of the critical systems fail. The SAN has, of course, become a crucial component of the district's new centralized backup solution.

Planning for Disaster

School districts with limited staff and resources should focus on backup as the best place to start thinking about a COOP strategy for information systems, says Linda Sharp, project director for crisis preparedness for the Consortium for School Networking.

“From an IT standpoint, backup is critical,” she says. “Districts need to take the time to create a plan that covers every possibility, from equipment failure to natural disaster. But backing up systems and data is a good place to begin. And when they do that, they need to think about where the backups are and if the backups have backups.”

For districts that can afford it, outsourcing backup to a hosted service is a way to ensure that back-up information is held at a safe distance from natural disasters and devastating events such as fires, Sharp says.

It's imperative to take a comprehensive look at how IT functions in the school district, says Witty. This fundamental step is sometimes overlooked in continuity planning, she says.

“Understand which systems are mission-critical and what the impact will be if the district doesn't have that technology,” Witty says. “Know how long the schools can go without the technology – and that varies. A major failure is not as bad during a vacation as it would be on opening day.”

Other IT professionals thinking about continuity planning should approach it as an integral part of optimizing their systems, says Sidoti.

“My best advice to people is to take a step back and analyze the whole process: your design, your equipment and your implementation,” he says. “You ask if you have the best architecture possible, how can you keep things up and running, and then, how can you back things up so you can recover from a disaster.”

Despite the constraints of K–12 district budgets, Sidoti cautions against making buying decisions on the basis of low cost alone, stressing that “cheap is not the same as cost-effective.”

Funding Finesse

Thanks to a recent influx of state funding to districts that serve students from low-income families, Geneva Area City Schools is in the midst of revamping all aspects of its infrastructure, including new construction and building consolidation as well as opportunities to update technology, Zappitelli says. In that context, Sidoti's far-reaching vision of how IT systems should be built is invaluable, she says.

“Scott looked at what we were spending before and said we need to use the dollars in a different way,” she says. “Sometimes the difference is not more funds, but the better use of them. By changing the way we cut up the pie of IT funding, he's made the systems work more effectively and made them less vulnerable to disasters as well.”

Disaster planning is important, but a large part of the process should involve preventing or minimizing the potential impact of system failures, Sidoti says.

“When you research COOP planning, you get a lot of information about disaster recovery,” he says. “That's important because you can't do anything about tornadoes or floods. But more of the focus should be on avoiding the disasters that come when equipment or systems fail.”