How to Track and Measure Education IT Initiatives

From planning through implementation, managing any complex IT project to its conclusion is always challenging. When it's done, it feels great. However, you should hold off on the celebration, because you're not done.

How do you know if your project is a success? The only way to know for sure is to conduct a careful analysis of your expected results. There is no perfect measuring stick to benchmark every IT initiative, but there are various metrics–from basic desktop scorecards to full-blown organizationwide Six Sigma initiatives–that can be used to measure IT projects from start to finish.

At the University of Georgia in Athens, Ga., Greg Topp, director of information technology planning and decision support, is helping to develop several performance management systems to ensure that the school's 33,000 students are getting what they need. “IT always has more demand than supply,” he says, “so we have to find a way to get focused on the higher needs of the university.”

Where to start? With a four-criteria Balanced Scorecard adapted from the business world to link operational processes to overall goals. Topp's group has identified financial performance/cost management, operational performance, people performance and asset management as areas to monitor. For each, he identifies accountability in tangible terms and then tracks results. Operational performance, for example, could include on-time delivery of technology, uptime and even the number of calls handled by the help desk.

The Balanced Scorecard is pivotal because it “represents a structured way to create a process and to create a link between operational performance and the goals of the organization,” Topp explains. Moreover, the university uses a Compact Planning process that builds a strategy to connect IT and its budget with the university's goals. It's what Topp calls “negotiated development of the priorities of the university.” This process, along with portfolio management, ties back into the Balanced Scorecard.

Finally, Topp praises the Capability Maturity Model, a flexible quality management system that has been used in business, government and education. This five-level scheme focuses on compliance and self-assessment of internal processes. As you climb the ladder from stage 1 to stage 5, you move from the creation of a stable environment and repeatable processes to a loop of continuous improvement that develops when you are operating effectively and spending more time on optimization. Topp also hopes to do some high-level electronic dashboarding so that any university administrator can get a good view of what's going on with IT performance.

At Bryant University in Smithfield, R.I., 3,000 notebook PC-toting students and 140 professors enjoy the technological trappings of what The Princeton Review calls the second most wired college in America. Art Gloster, vice president of information services, maintains that ranking with an elaborate system of IT goal-setting and results-tracking that's intertwined with the goals of the university's four other vice presidents.

Macro goals are devised in all-day retreats that cover not just a one-year plan, but a five-year plan. “I have to support the academic side of the institution,” Gloster says. “It's not just about IT itself.” The vice presidents meet two hours each month to track their goals as a group.

Gloster starts with specific goals and then defines minimum, target and superior results for each one. At the end of the academic year, points are awarded, with more points given for better results. Everything adds up to a final number that's reported to the president.

Gloster measures results in a variety of ways, including network uptime (the minimum acceptable is 99.9 percent on a 24-hour basis), the amount of time it takes to resolve problems, user satisfaction as gauged by in-house surveys of various subsets of the campus community (faculty, students, etc.), and even Bryant's current spot in the annual U.S. News & World Report rankings.

One of Gloster's recent big projects was to roll out a wireless network (with 230 hot spots) and Internet Protocol (IP) telephony system. “We saved $126,000 in personnel costs and project a four-year cost recovery followed by $265,000 in annual savings,” Gloster says. These numbers are pivotal because, unlike many other universities, Bryant actually puts depreciation on its books, affecting the bottom line well into the future.

IT performance is just as important at small liberal arts institutions such as Bowdoin College in Brunswick, Maine. With 1,650 students and 179 faculty members, the scale may be smaller, but the challenges are just as big.

CIO Mitch Davis has the responsibility of reporting directly to the president and the luxury of staying in close contact with the faculty, students and administration, who constantly provide feedback about IT initiatives. “We don't build anything to meet IT's needs,” he says. “We create solutions to serve Bowdoin's faculty and students, and we can easily see if we're succeeding.”

Davis cites the example of deploying a campuswide gigabit network. After conquering the financial aspects of the project, Davis looked not to benchmarks or traffic data to judge the network's success, but rather to the changes he saw on campus. “The network is faster and more secure now, so customers [faculty and students] are confident it will work, and they actually use it now,” he says. “That's what matters.”

Davis says the best way for him to measure success is to walk around the college and meet with the customers and listen to feedback from his staff. “Bowdoin wants to provide students with the tools they need to move forward in society,” he says. “Technology is just one of those tools, and it's always there just under the surface–all the technical resources that anyone would need.”