Measuring the Value of Technology Projects

It pays to measure the qualitative benefits and return on investment of technology.

INCREASINGLY, SCHOOL administrators and technology directors are being asked to respond to questions such as, “We bought 50 computers last year, so why aren’t the kids any smarter?”

It is generally agreed that providing a one-to-one computing environment for students has many benefits. But how does one go about evaluating the costs and benefits to determine the net value? In short, is the project worth implementing? Evaluating benefits in K-12 for many projects, including ones that involve technology, is particularly difficult, as the benefits are often qualitative.

To determine the value of implementing a proposed project or technology, one must look at costs and benefits.

IDENTIFYING BENEFITS

Many projects are not undertaken to reduce expenses or to enhance staff productivity. The real benefits are largely qualitative and relate to school goals and mandates. So how does one measure the benefits of providing equity of access, enhanced student learning or 21st century skills?

Before diving into lists of benefits, we need to understand some basics. First, the obvious: Technology is not a benefit unto itself. Schools were not established to house networks of computers, but are supposed to educate our children. In areas where computers can assist in the task of enhancing education, either directly or indirectly, that is good.

DISTRICT GOALS

The objective of educating our children is further defined and expanded upon by district goals. These goals, which should be specified in measurable terms, capture the vision for the district, along with state and federal mandates.

Many of these goals are generally documented as a part of a long-range plan covering a four- or five-year period. The goals are accomplished not by what we do, but by how we do it — by processes that relate to one or more of these goals or the overall objective.

In order to educate our children, processes are established to accomplish everything from getting kids to school, tracking students and administering funds to developing curricula and teaching. Since resources are limited, improved or totally reinvented processes either directly or indirectly improve the school’s ability to educate students. As Jack Grayson, founder and chairman of the Houston-based American Productivity and Quality Council (APQC), says, “Outcomes cannot be changed without changes in processes.”

Technology is one very important tool that can improve these processes. Making a process more efficient or effective either directly affects the ultimate objective of educating students or allows resources to be diverted to other processes that do affect this ultimate objective. While process improvement can result in progress toward goals, process transformation or reinvention can result in dramatic gains.

DETERMINING COST

In order to have a sustainable one-to-one project, the costs need to be calculated over the life of the project. Initial purchases, training and implementation costs should be amortized or annualized over the life of the cost element.

Also, ongoing costs must be added in. For example, initial amortized costs could include notebook PCs over four years, building upgrades over 15 years, initial teacher training over five years, and servers and network switches over five years.

Unbudgeted but real costs that should be considered are indirect labor and risk. Indirect labor is the hidden cost of time spent by users in training and handling system-related issues. These costs are often as large as or larger than the direct costs.

Risk is the probability of success for the project. Risk is a rather subjective factor and is best arrived at via consensus. While risk is not an actual cost, it can be treated as a cost when evaluating whether to move ahead with a proposed project or when comparing projects that are competing for the same funds.

Total Cost could be calculated as:

Total Cost = (Annualized Direct Costs + Indirect Costs) ÷ Probability of Success

Probability of success is a fraction between zero (no chance of success) and one (100 percent certain of success).

The CoSN-Gartner TCO tool is available to help schools determine current direct and indirect costs for technology and can be used to provide valuable input to the one-to-one project cost estimates. Visit http://classroomtco.cosn.org/gartner_intro.html. Also available from CoSN is a spreadsheet designed to help users calculate the costs of an educational technology system. The spreadsheet is available upon request from k12tco@alyrica.net.

MEASURING BENEFITS

First, any anticipated dollar savings need to be identified. These may include items such as cost avoidance of buying textbooks and freeing up space taken by a computer lab. Any benefits that result in cost savings, personnel productivity or incremental revenue should be identified and stated in dollar terms.

Qualitative benefits include items such as student achievement, political value, closing the digital divide, and many other benefits that need to be aligned with a prioritized list of the district, state and federal goals. The qualitative benefits, with anticipated results, should be stated in measurable terms as completely as possible.

Aligning the benefits with appropriate processes helps define value. Improved student achievement, as a major goal of most classroom computing projects, can be defined in terms of teaching and curricula processes and impact on specific goals, such as a five-point improvement in standardized test scores or increased attendance.

Once these measurable goals have been identified, they need to be aligned with the district, state and federal goals for the district, with relative importance associated with each goal. This benefit information can be scored in a fashion similar to the chart on page 42.

VALUE OF INVESTMENT

In Return on Investment (ROI) terms and adding in risk and qualitative benefits:

VOI = [(Total $ Benefit + $ Value of Total Score) ÷ Total Cost] x 100%

A value of investment greater than 100 percent has a positive value, and for competing projects, the project with the highest VOI has the best projected return on dollars spent. The hard part here is, of course, attaching a dollar value to the total score of qualitative benefits. While there may be no agreement as to what this relationship should be, the total score may still prove valuable when assessing projects that are competing for the same funding.

A more practical approach may be to first determine net cost, which is the Total Cost minus any benefits that can be stated in dollars — savings, staff productivity, cost avoidance or revenue increase: Net Cost = Total Cost less $ Benefit. From here, projects competing for the same funds can be compared and evaluated based on Net Cost and Total Score.

SUMMARY

There is no precise method to calculate the true dollar value of many proposed technology projects in an educational environment, as many of the benefits are qualitative. However, it is very helpful when evaluating these projects internally, and when explaining the value of projects with constituents, to apply a consistent methodology to fully determine all of the associated costs and benefits. When looking at qualitative benefits, they must be aligned with goals or objectives of importance to the school or district and described in measurable terms.

ENHANCE PROCESSES

The use of technology to enhance school processes can be categorized:

• Technology infrastructure improvement: cost savings, with computer services staff and user efficiencies reducing costs and providing better service

• Administrative computing applications: applications, such as accounting, providing reduced administrative costs

• Community outreach: Internet-based applications to communicate with parents, students and the community, which may have some cost savings benefit, but with more value related to community image and student performance

• Teacher computing: teacher productivity and tools designed to make teachers more productive and effective, reduce costs and provide information toward enhancing student achievement

• Student computing: focused on improving student achievement, with potential personnel cost savings

MEASURING MORE THAN TEST SCORES

In the fall of 2005, Scarsdale Public Schools in Scarsdale, N. Y., decided to measure the impact of technology on student achievement. Using a model designed by the Tri-State Consortium, an organization of high-performing schools in the New York, Connecticut and New Jersey region, Scarsdale’s staff focused on the use of IT to enhance student research.

District computer teachers and librarians gathered evidence that would show the impact of technology on its educational program. A team of 16 peers from neighboring districts examined evidence and interviewed teachers, students, board members, administrators and community members. The team focused on answering three essential questions:

• What evidence demonstrates that collaboration between the teaching staff results in high-quality technology and research projects?
• To what extent does the use of district technology resources support and enhance student research?
• To what extent do professional development opportunities support the use of technology and research?

The Tri-State team identified several strengths of Scarsdale’s IT program, including the depth and quality of student work, the knowledge and leadership of the IT staff and the broad support of the Board of Education and the community.

The Scarsdale district learned the importance of involving the whole community, allowing opportunities for districtwide conversations during the planning process and having a plan for follow-up prior to the evaluation.

The staff used the evaluation to engage in reflective practice, foster faculty collaboration and examine ways to make a meaningful difference in the lives of students.

Jerry Crisci, Director of Technology, Scarsdale Public Schools, N. Y.

ASSIGN RELATIVE IMPORTANCE TO EACH MEASURABLE GOAL

Goal Component: Cost savings
Importance [1 to 10]______
Fit [+ / 0 / −]______
Effect [1 to 10]______
Financial Benefit or Score______

Goal Component: Increased student achievement
Importance [1 to 10]______
Fit [+ / 0 / −]______
Effect [1 to 10]______
Financial Benefit or Score______

Goal Component: Reduced absentee rate
Importance [1 to 10]______
Fit [+ / 0 / −]______
Effect [1 to 10]______
Financial Benefit or Score______

Goal Component: 21st century skills developed
Importance [1 to 10]______
Fit [+ / 0 / −]______
Effect [1 to 10]______
Financial Benefit or Score______

Goal Component: Curriculum enhanced
Importance [1 to 10]______
Fit [+ / 0 / −]______
Effect [1 to 10]______
Financial Benefit or Score______

Goal Component: Political value by parent training
Importance [1 to 10]______
Fit [+ / 0 / −]______
Effect [1 to 10]______
Financial Benefit or Score______

Goal Component: Develop trained workforce, new skills gained
Importance [1 to 10]______
Fit [+ / 0 / −]______
Effect [1 to 10]______
Financial Benefit or Score______

Goal Component: Decreased dropout rate
Importance [1 to 10]______
Fit [+ / 0 / −]______
Effect [1 to 10]______
Financial Benefit or Score______

Goal Component: Breach digital divide; meet No Child Left Behind mandate
Importance [1 to 10]______
Fit [+ / 0 / −]______
Effect [1 to 10]______
Financial Benefit or Score______

Goal Component: Additional goal
Importance [1 to 10]______
Fit [+ / 0 / −]______
Effect [1 to 10]______
Financial Benefit or Score______

Total Score
Importance [1 to 10]______
Fit [+ / 0 / −]______
Effect [1 to 10]______
Financial Benefit or Score______

Goal component: List all district, state and federal goals that may be impacted by, for example, the one-to-one project.

Importance: The relative importance of this goal to the school

Fit: How the one-to-one project fits with this goal. A project may have no fit or even have a negative impact on a given goal.

Effect: The relative effect the one-to-one project has on achieving this goal.

Financial Benefit or Score: Anticipated cost savings, dollar value of productivity gains, plus any means of converting qualitative measures into dollars. Where benefits can’t be evaluated or agreed upon, score = goal importance X effect.

Rich Kaestner is TCO Project Director for the Consortium for School Networking, a Washington, D.C.-based nonprofit organization.

Oct 31 2006

Sponsors