How will you install your next server? What if the uninterruptible power system (UPS) is maxed out, the air conditioner can barely cool the server room or another rack cabinet won’t fit through the door? We encountered all of these problems at Cincinnati Country Day School in Ohio, and the answer that made sense for us — both technically and economically — was server consolidation with VMware.
This case study documents our two-phase virtualization project, which cost $7,000 for the first phase, plus $90,000 over two years. The number of physical servers decreased from 16 to two, and we avoided $10,000 in costs for another UPS and air conditioning system.
Moreover, the hardware cost of the next virtual machine is zero, because the existing hardware can probably handle 50 to 100 virtual machines, and the useful life of the equipment will likely be five years. Other benefits include excellent performance and reliability, efficient administration, the ability to respond quickly and flexibly to user needs for additional server resources, and significantly reduced space, power and cooling requirements.
BUILDING THE ARCHITECTURE
At Cincinnati Country Day, we began our virtualization initiative two years ago. At that time, we were running 16 Compaq servers, which were each between two and four years old, substantially filling two rack cabinets. Each rack had a 3kVA UPS running at 80 percent capacity, making it impossible for the air conditioning to keep the room below 78 degrees. We were also anticipating the addition of two Hewlett-Packard (HP) DL380 servers as part of the implementation of a proposed schoolwide information management system.
Faced with the prospect of adding another equipment rack, another UPS and a larger air conditioner at a combined cost of more than $10,000, we chose to consolidate our hardware by installing VMware ESX Server software on an existing HP DL380.
The cost for this first phase of the project was under $7,000. That covered licensing and support costs for the ESX Server software, an additional 2- to 8-gigabyte RAM (we needed more RAM as we increased the number of virtual machines) and two 36GB hard drives for the DL380.
At the outset, we intended to virtualize our three oldest systems, all Compaq ProLiant 1600s that were hosting our public Web server, databases for the Business and Admission Offices, and our Equisys Zetafax network fax system. In fact, we were not able to virtualize our Zetafax server due to its reliance on a fax board that’s not supported on VMware ESX Server. (ESX Server has minimal support for add-on hardware, but this makes it a leaner, more efficient operating system.)
Virtual machines can be easily migrated to other ESX Servers, all of which support the same set of virtual hardware. ESX Server does support pass-through of SCSI devices, such as backup tape drives and their libraries. Ultimately, we worked around the Zetafax problem by virtualizing a Compaq ProLiant DL360 that had been our TrendMicro antivirus deployment server. We then moved the fax board and Zetafax software to the DL360.
We continued our virtualization efforts by migrating additional servers that had been running on older DL360s, such as Microsoft’s Internet Information Server, Dynamic Host Configuration Protocol (DHCP), print services, Active Directory domain controller and SQL 2000 Server, as well as Trend Micro’s OfficeScan and a FileMaker Server.
SCALING UP INFRASTRUCTURE
Another aspect of server virtualization soon became apparent: It is quick and painless to set up a new virtual server. Many server software applications work better when they have the operating system all to themselves.
When more than one server application was running on an old physical server, we allocated a separate, new virtual server to each. That required us to upgrade our DL380 ESX Server’s RAM from 5GB to 8GB. We felt we had reached the limits of this system with 15 virtual machines: The disk array was nearly full, and the baseline CPU utilization was 50 percent.
At that point, we began planning how best to scale up our VMware infrastructure so that we could virtualize the rest of our environment. Our experience with a single ESX Server over a period of a year demonstrated excellent stability, reliability and performance.
THE MAINTENANCE CHALLENGE
However, the difficulty with running a single ESX Server with locally attached storage is in maintaining it. A hardware upgrade or a VMware software upgrade requires shutting down all hosted virtual machines during the upgrade. By the same token, a hardware failure of the ESX Server would take all hosted virtual machines offline until repairs were completed. For that reason, we installed dual power supplies and a RAID-5 disk array on the DL380 ESX Server and upgraded its warranty to HP’s two-hour depot service.
To avoid these issues going forward, we decided to install two HP ProLiant DL385 servers, each connected to an HP StorageWorks MSA1500 storage area network. The SAN contains 14 300GB drives in a RAID-6 configuration with two online spares (2.8 terabyte capacity).
The total cost of the hardware, VMware software and support was about $90,000, and we completed the hardware setup in March 2006. By the end of April, we had virtualized the remainder of our servers, including two Microsoft Exchange servers, one Microsoft SQL server, three intranet Web servers, two file servers, the DHCP and print server, and one Active Directory domain controller. In all, there are 23 virtual machines on the two ESX Servers, which both run at an average CPU utilization of about 10 percent.
With dual ESX Servers and a SAN, we can do maintenance on one of the ESX Servers at a time, with zero downtime for the virtual machines. We only have to hot-migrate all the virtual machines to the other ESX Server using VMotion, whereby the RAM contents for a virtual machine are copied over the network from one ESX Server to the other.
VMotion uses a transaction log strategy to capture changes to RAM contents while the copy is in progress. When that is complete, the virtual CPU is suspended; the CPU state is copied to the other ESX Server and then is resumed.
Our VMware infrastructure project is complete, with six machines left unvirtual-ized. The Active Directory PDC emulator, the VirtualCenter management server and the Zetafax server will probably remain on separate physical machines. The PDC emulator should remain separate so that Active Directory and DNS can function if VMware is down. Similarly, it is helpful to have access to the VirtualCenter management server if the ESX Servers are offline.
We will probably virtualize our Backup Exec server and the two Senior Systems school information management servers as their hardware ages. Even so, all of our server equipment occupies about half of one equipment rack, and our power consumption is half of what it was originally.
Considering server virtualization? Add these five questions to your checklist:
1. What are the CPU, RAM, hard disk and network utilizations of your existing servers? This will determine the capacity requirements of your virtualization host systems.
2. What add-on hardware do your servers require? Some add-ons are incompatible with virtualization.
3. Which software vendors support running in a virtualized environment?
4. Where is your school or district in the server hardware purchasing cycle? If you’re about to buy new hardware, it’s time to consider virtualizing.
5. What is the scale of your proposed deployment? Begin by testing VMware Server, which is free. Then start slowly with a single VMware ESX Server or a full-scale redundant ESX server and a storage area network.
HOW IT WORKS
Most production servers run at a very low average level of CPU utilization with occasional bursts of activity. The VMware host operating system leverages this fact to manage a number of Windows or Linux guest operating systems (virtual servers) on a single physical server, sharing the CPU resources among them. Every virtual server is allocated its own share of physical RAM and one or more virtual disks, each of which is a large file on the host server’s physical disk array.
Through various sharing mechanisms among virtual servers, VMware can achieve up to a 30 percent savings in physical RAM utilization. Disk storage can be locally attached or reside on a storage area network. Virtualizing 10 to 30 guests on a single host has worked well in our experience. The limiting factors have been RAM availability and CPU utilization.
The VMware product line includes the free, low-end VMware Server and the enterprise-class VMware ESX Server. VMware Server, formerly called GSX Server, runs on top of a Windows or Linux host operating system. It is intended as a low-cost introduction to the benefits of virtualization.
VMware ESX Server is intended to provide the reliability and high performance needed in a production environment. ESX Server is part of VMware’s Virtual Infrastructure product suite, which also includes hot migration between ESX Servers, recovery of virtual machines from failed ESX Servers, dynamic load balancing of virtual machines among ESX Servers and a management console.
Jeffry A. Spain is network administrator at Cincinnati Country Day School in Ohio.