Measuring the Value of Technology Projects

INCREASINGLY, SCHOOL administrators and technology directors are being asked to respond to questions such as, “We bought 50 computers last year, so why aren’t the kids any smarter?”

It is generally agreed that providing a one-to-one computing environment for students has many benefits. But how does one go about evaluating the costs and benefits to determine the net value? In short, is the project worth implementing? Evaluating benefits in K-12 for many projects, including ones that involve technology, is particularly difficult, as the benefits are often qualitative.

To determine the value of implementing a proposed project or technology, one must look at costs and benefits.

IDENTIFYING BENEFITS

Many projects are not undertaken to reduce expenses or to enhance staff productivity. The real benefits are largely qualitative and relate to school goals and mandates. So how does one measure the benefits of providing equity of access, enhanced student learning or 21st century skills?

Before diving into lists of benefits, we need to understand some basics. First, the obvious: Technology is not a benefit unto itself. Schools were not established to house networks of computers, but are supposed to educate our children. In areas where computers can assist in the task of enhancing education, either directly or indirectly, that is good.

DISTRICT GOALS

The objective of educating our children is further defined and expanded upon by district goals. These goals, which should be specified in measurable terms, capture the vision for the district, along with state and federal mandates.

Many of these goals are generally documented as a part of a long-range plan covering a four- or five-year period. The goals are accomplished not by what we do, but by how we do it — by processes that relate to one or more of these goals or the overall objective.

In order to educate our children, processes are established to accomplish everything from getting kids to school, tracking students and administering funds to developing curricula and teaching. Since resources are limited, improved or totally reinvented processes either directly or indirectly improve the school’s ability to educate students. As Jack Grayson, founder and chairman of the Houston-based American Productivity and Quality Council (APQC), says, “Outcomes cannot be changed without changes in processes.”

Technology is one very important tool that can improve these processes. Making a process more efficient or effective either directly affects the ultimate objective of educating students or allows resources to be diverted to other processes that do affect this ultimate objective. While process improvement can result in progress toward goals, process transformation or reinvention can result in dramatic gains.

DETERMINING COST

In order to have a sustainable one-to-one project, the costs need to be calculated over the life of the project. Initial purchases, training and implementation costs should be amortized or annualized over the life of the cost element.

Also, ongoing costs must be added in. For example, initial amortized costs could include notebook PCs over four years, building upgrades over 15 years, initial teacher training over five years, and servers and network switches over five years.

Unbudgeted but real costs that should be considered are indirect labor and risk. Indirect labor is the hidden cost of time spent by users in training and handling system-related issues. These costs are often as large as or larger than the direct costs.

Risk is the probability of success for the project. Risk is a rather subjective factor and is best arrived at via consensus. While risk is not an actual cost, it can be treated as a cost when evaluating whether to move ahead with a proposed project or when comparing projects that are competing for the same funds.

Total Cost could be calculated as:

Total Cost = (Annualized Direct Costs + Indirect Costs) ÷ Probability of Success

Probability of success is a fraction between zero (no chance of success) and one (100 percent certain of success).

The CoSN-Gartner TCO tool is available to help schools determine current direct and indirect costs for technology and can be used to provide valuable input to the one-to-one project cost estimates. Visit http://classroomtco.cosn.org/gartner_intro.html. Also available from CoSN is a spreadsheet designed to help users calculate the costs of an educational technology system. The spreadsheet is available upon request from k12tco@alyrica.net.

MEASURING BENEFITS

First, any anticipated dollar savings need to be identified. These may include items such as cost avoidance of buying textbooks and freeing up space taken by a computer lab. Any benefits that result in cost savings, personnel productivity or incremental revenue should be identified and stated in dollar terms.

Qualitative benefits include items such as student achievement, political value, closing the digital divide, and many other benefits that need to be aligned with a prioritized list of the district, state and federal goals. The qualitative benefits, with anticipated results, should be stated in measurable terms as completely as possible.

Aligning the benefits with appropriate processes helps define value. Improved student achievement, as a major goal of most classroom computing projects, can be defined in terms of teaching and curricula processes and impact on specific goals, such as a five-point improvement in standardized test scores or increased attendance.

Once these measurable goals have been identified, they need to be aligned with the district, state and federal goals for the district, with relative importance associated with each goal. This benefit information can be scored in a fashion similar to the chart on page 42.

VALUE OF INVESTMENT

In Return on Investment (ROI) terms and adding in risk and qualitative benefits:

VOI = [(Total $ Benefit + $ Value of Total Score) ÷ Total Cost] x 100%

A value of investment greater than 100 percent has a positive value, and for competing projects, the project with the highest VOI has the best projected return on dollars spent. The hard part here is, of course, attaching a dollar value to the total score of qualitative benefits. While there may be no agreement as to what this relationship should be, the total score may still prove valuable when assessing projects that are competing for the same funding.

A more practical approach may be to first determine net cost, which is the Total Cost minus any benefits that can be stated in dollars — savings, staff productivity, cost avoidance or revenue increase: Net Cost = Total Cost less $ Benefit. From here, projects competing for the same funds can be compared and evaluated based on Net Cost and Total Score.

SUMMARY

There is no precise method to calculate the true dollar value of many proposed technology projects in an educational environment, as many of the benefits are qualitative. However, it is very helpful when evaluating these projects internally, and when explaining the value of projects with constituents, to apply a consistent methodology to fully determine all of the associated costs and benefits. When looking at qualitative benefits, they must be aligned with goals or objectives of importance to the school or district and described in measurable terms.