Accountability Isn't a Four-Letter Word

WHEN A SALTY FOUR-LETTER word is uttered, it may draw unwanted attention to the speaker and create discomfort among the listeners. Although the word accountability actually has 14 letters, its mention often evokes similar reactions.

Yet demonstrating real return on investment (ROI) with technology is an issue schools can no longer avoid, deflect or soft-pedal. Funds are tight, so school boards and superintendents are demanding evidence of the effectiveness of every technology investment.

“The media and many parents, too, are increasingly unsatisfied with the ‘trust us’ answer,” says Curt Anderson, instructional technology coordinator for Millard Public Schools in Omaha, Neb.

With appropriate producing metrics in place, we can show how technology expenditures have a positive effect on the bottom line of education: raising student achievement. High-stakes tests aren’t the only option for measuring technology’s benefits. So where do we begin when we want to demonstrate results? And how can we frame our goals and evidence in a convincing fashion?

UPDATING METRICS

Many educators struggle to answer these questions, while others hope to avoid the debate altogether. Regrettably, school leaders often employ less important, imprecise or misplaced measures to arrive at their answers. Metrics employed by educational leaders have historically fallen into three categories:

Baseball provides helpful metaphors. Riding the pine bench gives us time to consider what’s happening on the field. These are Tier 1 metrics. But metrics of this nature often don’t show the true value of technology projects.

“For many years, technology evaluation has been all about counting and comparing — our student-to-computer ratio, how many computers we have on our network and how many classrooms have Internet access,” explains Bill Morrison, director of technology for the Rapides Parish School District in Alexandria, La. “The problem with these measures is that they say nothing about student outcomes, yet we persist in using such measures to determine program success.”

Yet, in both cases, we are not in a position to contribute to the team’s efforts to win. (See the “Getting Ready to Bat ” sidebar on page 47 for details on Tier 1 and Tier 2.)

“These metrics are popular because they are very easy to administer and collect,” Morrison continues. The type of measures increasingly demanded by many decision-makers and administrators fall into the “at bat” category. Employing these metrics shows that technology is a productive player that can get on base, get in scoring position, help the school district score and maybe hit a home run.

TIER 3: AT BAT

Tier 3, or “at bat,” measures tread on the prized ground of results-oriented thinking. They represent more powerful tools — metrics that, if employed, will begin to move schools steadily toward a producer mentality and away from a consumer mentality in educational technology.

“If student achievement is truly the ultimate goal,” says Hal Anderson, director of information systems for the Cheyenne Mountain School District in Colorado Springs, Colo., “then the ultimate measure of effective use of technology must be based on what the students have accomplished with the tools that are available to them, rather than being based on how successful we as educators may have been in the task of procuring those tools.”

These metrics fall into four categories: light, soft, medium and hard gains.

In the light gains arena, educators can develop paper-based or online surveys that capture student feelings about using technology or simply indicate a preference for learning with technology. Schools can measure motivational impact by comparing how long students may linger after class or stay focused on a given task compared with the same interest level or time on task during a traditional lesson. A principal or lead teacher may also gather compelling classroom anecdotes by interviewing classroom teachers and soliciting examples of learning moments or student success stories.

In the soft gains arena, educators can track attendance patterns during an ongoing technology-based activity and compare those data with traditional patterns to see if attendance has increased or instances of tardiness have decreased. Another metric involves tracking classroom assignment completion or homework submittal during a technology-based intervention and comparing those data with historical data. Are students turning in more homework or completing more assignments as a result of this intervention?

Learning efficiency (learning or accomplishing more in less than the expected time) is one of the easiest and most important gains to document. In order to measure learning efficiency, simply compare the amount of time it takes to accomplish a task or master a lesson standard using the technology with the amount of time it would take without the technology. Another method would be to measure the amount of material covered with the technology intervention as opposed to the amount of information covered in a traditional lesson without technology.

In the medium gains arena, consider administering a pre-test prior to a technology-supported lesson and then testing again to demonstrate that student gains were indeed realized. Measuring increased enrollment in upper-level courses requires the long view but remains a desirable goal. Educators must first present historical enrollment patterns and show that the intervention has in part led to increased enrollment of women, minorities or target populations in upper-level courses.

Next, imagine the appeal of a grant proposal that offers to reduce the number of Ds and Fs in a historically poor-performing class. This can be done by showing a historical tendency and aiming to reduce that trend through the introduction of a technology-based intervention.

In the hard-gains arena, educators enjoy many convincing options for measuring ROI. We can document concrete improvement of skills over time by referencing a rubric or projects in a learning portfolio.

In order to warrant continued funding for a project, consider using value-added assessment tools to demonstrate how a technology-based literacy program affects gains. If school leaders insist on standardized test score improvements before funding a key innovation, create a pilot project that focuses on improving the subcomponents of a test score, known as a content subscore.

Also, imagine crafting a technology proposal that aims to move students up in proficiency level on a state assessment. For example, if 10 students are on the bubble between partially proficient and proficient performance levels, concentrate on improving just those students with your technology intervention.

Educators no longer need to recoil from the challenges of accountability. However, schools do need to enlist more compelling metrics as they strive to demonstrate the learning benefits associated with technology investments.

“We shouldn’t throw up our hands just because impact assessment is difficult,” says Brent Wilson, professor of information and learning technologies at the University of Colorado at Denver. “We should build a reasoned case for impact that might draw on a whole array of measures — some better than others, but all useful in drawing a portrait of overall impact.”

Increasingly, the metrics that matter will be producing metrics.