How to Track and Measure Education IT Initiatives

The only way to know for sure is to measure–and score–IT initiatives. Here are some options to get you started.

One objective way to know if an IT initiative meets its goals is to measure and score it, possibly with a Balanced Scorecard.

From planning through implementation, managing any complex IT project to its conclusion is always challenging. When it's done, it feels great. However, you should hold off on the celebration, because you're not done.

How do you know if your project is a success? The only way to know for sure is to conduct a careful analysis of your expected results. There is no perfect measuring stick to benchmark every IT initiative, but there are various metrics–from basic desktop scorecards to full-blown organizationwide Six Sigma initiatives–that can be used to measure IT projects from start to finish.

At the University of Georgia in Athens, Ga., Greg Topp, director of information technology planning and decision support, is helping to develop several performance management systems to ensure that the school's 33,000 students are getting what they need. “IT always has more demand than supply,” he says, “so we have to find a way to get focused on the higher needs of the university.”

Where to start? With a four-criteria Balanced Scorecard adapted from the business world to link operational processes to overall goals. Topp's group has identified financial performance/cost management, operational performance, people performance and asset management as areas to monitor. For each, he identifies accountability in tangible terms and then tracks results. Operational performance, for example, could include on-time delivery of technology, uptime and even the number of calls handled by the help desk.

The Balanced Scorecard is pivotal because it “represents a structured way to create a process and to create a link between operational performance and the goals of the organization,” Topp explains. Moreover, the university uses a Compact Planning process that builds a strategy to connect IT and its budget with the university's goals. It's what Topp calls “negotiated development of the priorities of the university.” This process, along with portfolio management, ties back into the Balanced Scorecard.

Finally, Topp praises the Capability Maturity Model, a flexible quality management system that has been used in business, government and education. This five-level scheme focuses on compliance and self-assessment of internal processes. As you climb the ladder from stage 1 to stage 5, you move from the creation of a stable environment and repeatable processes to a loop of continuous improvement that develops when you are operating effectively and spending more time on optimization. Topp also hopes to do some high-level electronic dashboarding so that any university administrator can get a good view of what's going on with IT performance.

At Bryant University in Smithfield, R.I., 3,000 notebook PC-toting students and 140 professors enjoy the technological trappings of what The Princeton Review calls the second most wired college in America. Art Gloster, vice president of information services, maintains that ranking with an elaborate system of IT goal-setting and results-tracking that's intertwined with the goals of the university's four other vice presidents.

Macro goals are devised in all-day retreats that cover not just a one-year plan, but a five-year plan. “I have to support the academic side of the institution,” Gloster says. “It's not just about IT itself.” The vice presidents meet two hours each month to track their goals as a group.

Gloster starts with specific goals and then defines minimum, target and superior results for each one. At the end of the academic year, points are awarded, with more points given for better results. Everything adds up to a final number that's reported to the president.

Gloster measures results in a variety of ways, including network uptime (the minimum acceptable is 99.9 percent on a 24-hour basis), the amount of time it takes to resolve problems, user satisfaction as gauged by in-house surveys of various subsets of the campus community (faculty, students, etc.), and even Bryant's current spot in the annual U.S. News & World Report rankings.

One of Gloster's recent big projects was to roll out a wireless network (with 230 hot spots) and Internet Protocol (IP) telephony system. “We saved $126,000 in personnel costs and project a four-year cost recovery followed by $265,000 in annual savings,” Gloster says. These numbers are pivotal because, unlike many other universities, Bryant actually puts depreciation on its books, affecting the bottom line well into the future.

IT performance is just as important at small liberal arts institutions such as Bowdoin College in Brunswick, Maine. With 1,650 students and 179 faculty members, the scale may be smaller, but the challenges are just as big.

CIO Mitch Davis has the responsibility of reporting directly to the president and the luxury of staying in close contact with the faculty, students and administration, who constantly provide feedback about IT initiatives. “We don't build anything to meet IT's needs,” he says. “We create solutions to serve Bowdoin's faculty and students, and we can easily see if we're succeeding.”

Davis cites the example of deploying a campuswide gigabit network. After conquering the financial aspects of the project, Davis looked not to benchmarks or traffic data to judge the network's success, but rather to the changes he saw on campus. “The network is faster and more secure now, so customers [faculty and students] are confident it will work, and they actually use it now,” he says. “That's what matters.”

Davis says the best way for him to measure success is to walk around the college and meet with the customers and listen to feedback from his staff. “Bowdoin wants to provide students with the tools they need to move forward in society,” he says. “Technology is just one of those tools, and it's always there just under the surface–all the technical resources that anyone would need.”

TIPS TO MEASURE YOUR PROJECTS

DO monitor time and money saved. When technology makes a process faster, that process becomes less expensive.

DON'T talk tech to those who don't care. Professors are interested in the interface of their new online chat system, not about the capacity or throughput of the server that hosts it.

DO pay attention to customer satisfaction. Use survey results to find out how the college community rates IT overall. You need to know.

DON'T operate in an IT vacuum. Integrate and measure your IT goals with those of academic and administrative departments. You're all in this together.

DO track library services and telephone systems closely. They're leading indicators of your overall IT performance.

DON'T fear the application of business-oriented performance metrics in your academic environment. Systems like the Balanced Scorecard and Six Sigma are adaptable enough to be useful on campus.

THE BBQ BENCHMARK

Bowdoin College CIO Mitch Davis likes to measure his success not with spreadsheets or bar charts, but with barbecues. Playing off the title of one of his favorite management books, Sacred Cows Make the Best Burgers by Robert Kriegel and David Brandt, Davis treats his staff to a “sacred cow barbecue” every time they find and eliminate an entrenched administrative process that's inefficient. The better they're doing, the more often they grill.

Davis cites Bowdoin's human resources processes as an example. Initially, he encountered resistance to changing the two-week, 40-step procedure required to bring in new employees and get their paperwork approved and signed. He and his staff convinced administrators by having them watch a 12-minute video, created by the audiovisual group, that documents each of the 40 steps and then explaining that HR and IT could implement an online system that would significantly streamline the process. The administrators agreed, the faster process is coming online very soon and Davis fired up the grill one more time, taking a unique approach to measuring–and celebrating–IT success.

Don Willmott is a New York-based technology writer.

Oct 31 2006

Sponsors