When it comes to data security, the buck stops with me – and that isn't a responsibility I take lightly.
As director of technology for the Hendrick Hudson School District, a five-school district in suburban Westchester County (45 minutes north of New York City), I am the person responsible for ensuring that the data belonging to more than 3,000 students and staff is secure.
Over the years I've had my share of close calls while doing data restores. Eventually I wanted better assurance of the integrity of the district's backups, so I decided to find a better way of doing it.
I knew I wanted disk-based backup for speed and reliability, and that I needed our data to be stored offsite in case of catastrophe. But I was certain that tape was a technology of the past and that the process we had in place – rotating tapes to different buildings – was prone to failure.
Online backup seemed to be the logical choice. In fall 2007, I began to research online backup service providers and solutions. I collaborated with members of my IT staff and a few parent volunteers who work in IT to develop a plan of action that would suit our district's size and our unique needs. By summer 2008 we were up and running with an online backup and recovery solution, and we've never looked back.
Our plan had several components that worked well for us and could be of use to other schools and districts considering a move to cloud-based backups.
Also in this Issue
To learn how other districts have put their disaster recovery plans into practice, see "Speaking from Experience."
Complete a Risk Assessment
The first step in creating a disaster recovery plan is determining risk, and it makes sense to do the same when considering any major process change, such as a new backup procedure. Risks are categorized very differently depending on a district's size, location and type.
For the Hendrick Hudson School District, the risk of tornadoes, hurricanes and flooding is low. But the risks of a nuclear incident or major power outage need to be considered, given the presence of the Indian Point nuclear facility within district boundaries, as well as our proximity to Manhattan, which places us on the densest electrical grid in the country.
Consequently, we had to think about worst-case scenarios – and the possibility that the district could be closed or inaccessible for a long time. If the worst happened, we'd need secure, reliable remote access to all key district data. Once we identified which files on each server were essential to business continuity, we decided it would be simpler and most cost-effective to back up everything but student data offsite. In an emergency, we reasoned, we could continue without the current year's student data, but having staff and office data readily available was crucial.
Write an RFP
Our initial request for proposals was based on what we knew at the time. In hindsight, I should have added greater and more specific demands.
First, consider the availability of plug-ins for the databases you're using. We had some issues when testing database plug-ins, so be very specific about which ones you run to make sure the backup vendor you choose has a plug-in that works for every database you have. Also, include open-file plug-ins to ensure that files left open on a workstation are backed up. Deduplication allows the vendor to eliminate duplicate copies of the same file at the backup location and can save money. Make it a requirement.
Insist that data be backed up to a secure location and specify what you need to do to access it remotely. Include a request for charges per server, per gigabyte and per plug-in, and ask about ancillary costs too. Watch for hidden fees, and be sure to require a fully functioning trial with live data.
Analyze and Evaluate Responses
This step is critical. Anyone who has ever reviewed RFP responses knows that vendors often exaggerate their ability to provide services. Review each response carefully, and be leery of grand promises and unrealistic claims (such as 100 percent anything). Check references, and ask the vendors for detailed breakdowns of costs using your specific data as a guide. Compare the responses with your risk assessment to determine whether the vendors can handle the worst you might throw at them.
It's also worth letting your auditors look at the responses to identify potential concerns about physical security or data procedures at remote locations.
Conduct a Live Trial
No vendor can predict with certainty how your data will compress and how its software will interact with your servers and environment. A trial is key.
I installed the products of our top three contenders on one server apiece and let them run for a full month. I watched the logs, looked at the daily e-mails, and did test restores along the way – from the console and from a remote host. This allowed me to see which interface worked best for our environment, which software was most reliable, and which was easiest to set up and manage. You can't get this depth of data without a live trial.
I still ran my tape-based backups during the trial and prescheduled trial backups to avoid conflicts.
Analyze the Results
Return to the criteria specified in your RFP to gauge vendor performance. After a month of backups and restores, you'll have a lot of data to review.
Develop a rubric of criteria by which to analyze the results. Often the interface of a product is very user-friendly and simple, but the software "under the hood" isn't robust enough. Make sure you choose a product that has more than you need so you can grow into it. This is a big investment of time and money, so don't rush the decision-making process.
Once you've chosen your vendor, negotiate the contract. New York laws prohibit school districts from signing multiyear contracts that lack an opt-out clause, so vendors are always hesitant to offer us the best possible pricing for a one-year agreement.
Remember, however, that the market is competitive. Bargain hard to get the best price and terms for your district, because switching vendors later isn't easy. Be sure to include some cushion in your data plan to accommodate future growth.
Implement in Phases
The hardest part of implementing an online backup service is "seeding" the larger servers. Seeding is the first full backup of each server. Backups of larger servers can take days and can slow down your Internet connection.
In our case, we did a local backup to an external drive over a weekend when utilization was low, shipped it to the vendor overnight, and had the company move the backup data to its site and sync with our server. Smaller servers backed up overnight the first time without a problem. All subsequent backups are differential only and have run efficiently.
As each server is brought onto the new system and tested, you can begin to phase out your tape-based system until you're 100 percent online.
The Hendrick Hudson School District has had to recover data that was backed up to the cloud more than a dozen times in the past three years. Each restore was completed quickly and fully, without incident.
Conduct a Live Test Annually
Assuming you don't have to contend with a real disaster, it's good practice to test at least one server annually for disaster recovery purposes.
Each year we do a full restore to a spare server. For larger servers, the vendor will need to ship an external drive overnight. Smaller servers can be restored overnight using your Internet connection.
With each test you'll learn which adjustments to make to your backup settings until you have a very reliable system in place.
- Location: The lasting effects of the Sept. 11 terrorist attacks and Hurricane Katrina on the communities and infrastructures they affected prompted our decision to put geographic distance between our facilities and our data.
- Cost: We lowered our backup costs on a per-gigabyte basis by carefully pruning and managing backups and by using built-in versioning and data deduplication functions.
- Convenience: Having the ability to select files and folders to back up and restore via a user-friendly interface is vital.
- Reliability: Online backup offers peace of mind and security that can't be accomplished with tape or disk-based onsite backup.