EVALUATION OF PROGRAMS
The Secretary's Conference on Educational Technology-1999
Documenting the Effects of Instructional Technology: A Fly-Over of Policy Questions

by
Dale Mann, Professor Teachers College, Columbia University
New York, New York 10027-6696
and
Managing Director Interactive, Inc.
Huntington, New York 11743-3360

PREFACE

  1. EVALUATION RESEARCH ABOUT INSTRUCTIONAL TECHNOLOGY POLICIES

  2. PEDAGOGY

    1. The Efficacy of Instructional Technology
    2. Pedagogy: The Multiple Sources of Learning
    3. Pedagogy: Learning Outside the School
    4. Implications for a Research Agenda
      1. IT Effects @ Home
      2. IT Effects @ School
      3. The Effects of Serious Play
      4. Estimating the Critical Mass of IT
      5. IT Dosage
  3. POLITICS

  4. ECONOMICS

    1. Capitol Decisions and Capital Decisions
    2. The Public Benefits of Private Investment
    3. The Roles of Government
  5. FOUR INSTRUCTIONAL TECHNOLOGY ADDITIONS TO THE EXISTING CONCEPTION OF SCHOOLING

    1. Adding an Education Focus to Our School Focus
    2. Adding a Learning Focus to Our Teaching Focus
    3. Adding Homes to Schools: Adding Parents to Teachers
    4. Moving Learning to the Learner
  6. FOUR EVALUATION RESEARCH QUESTIONS FROM FOUR PERSPECTIVES


PREFACE

[October 22, 1707, The English Channel]

"Returning home victorious from Gibraltar... [Admiral] Sir Clowdisley...summoned all his navigators... The consensus placed the English fleet safely west of... the Brittany peninsula. But as the sailors continued north, they discovered to their horror that they had misgauged their longitude near the Sicily Isles [which] became unmarked tombstones for two thousand of Sir Clowdisley's troops... [and] four of the five warships."1

Appalled by the loss of lives and ships, in the Longitude Act of 1714, the Parliament promised a prize of £20,000 for a better way to navigate than throwing a log overboard and watching it drift off. Without a way to measure time accurately, ships could not determine noon and without a way to determine noon, they could not determine their east-west location. The lack of measurement had serious consequences.

I begin with two assumptions: first, that we want to do credible science and second that we want that science to advance the contribution that instructional technology might make to learning. Trying to advance instructional technology makes my perspective partisan or political in the sense that I am acknowledging a particular value interest in the outcomes of this application of science.


I. EVALUATION RESEARCH ABOUT INSTRUCTIONAL TECHNOLOGY POLICIES
"640K ought to be enough for anyone." Bill Gates, 1981

Doing research costs money and most evaluation research is paid for by clients---governments or private organizations. Adding a client may complicate evaluation in the same way that adding a patron complicates art: some clients and some patrons have expectations in addition to the outcomes of the otherwise pure event of science or art. The evaluation fish swims in the sea of politics, and should. Anne L. Bryant, executive director of the National School Boards Association says, "School Boards are going to be asking increasingly: 'Demonstrate to us that [computer-based instruction] has results."2

Imagine a senior government official on the cell phone, in a cab, running late on her way up Constitution Avenue to a congressional hearing. She knows she will face pointed questions about "All these computers we put in schools". If you answer the call, will you want to help address that skepticism? If you are good at helping, you might add seven figures to the appropriations authorization: if you are really good, you might add eight figures. If you want to help, you have to compel belief and that is likely to be more than good science, it is likely to address pedagogy, politics and economics, all at once.

In my view, evaluation research about education policy is intended to effect decisions and typically addresses pedagogy, politics and economics, all simultaneously.


II. PEDAGOGY
"There is no reason why anyone would want a computer in their home." Ken Olson, President, Chair and Founder of Digital Equipment Corp., 1977

    A. The Efficacy of Instructional Technology.

    The overriding question is how powerful is instructional technology? A second order question is, how do we know how powerful that technology is?

    First, I believe that instructional technology works.3 Instructional technology only works for some kids, in some topics and under some conditions but that is true of all pedagogy, all systems for teaching or learning. There is nothing that works for every purpose, for every learner and all the time.

    Emphasizing the things that instructional technology has not done has its political uses just as surely as saying that technology works. One continuity among critics of instructional technology is the idea that all teachers are always preferable to all machines, e.g., William L. Rukeyser's statement that, "The best teacher has always been a person, not a machine."4 The sub-title of the cover story of The School Administrator for April 1999 was "A Leading psychologist calls for slowing the rush toward computing". In that piece, the critic of instructional technology, Jane Healy acknowledges that, "...(W)ell implemented simulations and conceptually driven programs may improve learning---if a good teacher is in charge."5

    But what is known about the learning efficacy of such ubiquitous features of American schooling as the teacher-talk model of instruction? The 770 square foot classroom box? The 180-day (American) school year? We accept and even welcome critical attention to instructional technology that is seldom applied to the implacable regularities of American schooling. That leads to a paradox in which technology from the last generation has been proven inadequate and that from the next generation is unproven. With either negative data or none, the field is left to those who promptly make the next generation of technology the worst enemy of the current generation as in, 'Next year it will be cheaper, faster, smaller or even---more constructivist. So let's wait.'

    Our goal should be first, to understand the conditions of pro-social technology use and second to employ that understanding for learning improvement. Both require more penetrating analysis than has heretofore been the standard.


    B. Pedagogy: The Multiple Sources of Learning.

    The first thing to be understood is that there are many, many sources of learning. Technology needs to be disentangled from the other sources. Some are inside and many are outside the school. Parents educate, the family educates, the media educates and so on. Children learn from their teachers, from textbooks, from homework, from the Channel One TV on the wall and they learn from computers.6


    C. Pedagogy: Learning Outside the School.

    Thus, the contribution of instructional technology is best understood in a context that includes the contribution of all the educators.

    Since James S. Coleman's 1966 analysis, it has been generally acknowledged that about 30% of the variation in children's educational achievement comes from their experience in school and 70% comes from other experiences, especially their families, the culture they live in (the media, etc.) and their peers. Coleman's insight works both ways. Families that support learning, advance their children's educational achievement: those that do not or that hinder or disrupt learning, impede their children's educational achievement. In later writing, Coleman called the leverage that families apply to their children's learning, "family capital".


    D. Implications for a Research Agenda.

    There are several implications for an agenda of evaluation research about instructional technology.


         •  1. IT Effects @ Home.
           

      We need to account for all the educators---school effects and home effects. How many studies are there of the amount of learning at home that is supported by instructional technology?



         •  2. IT Effects @ School.
           

      Inside the school, we need to find ways to measure technology effects separate from teacher or textbook effects. How many studies attempt to measure the amounts of these various phenomena and associate them with outcomes?



         •  3. The Effects of Serious Play.
           

      Except for the 'learning-should-hurt' crowd, most educators recall what coaches and early childhood educators have never forgotten---play is a child's work. Entertainment is correctly pilloried as passive and generally purposeless. But active play is strongly connected to learning of all sorts.7 The fact that we do not know much empirically about what and how children are learning from technology-delivered serious play applications outlines the limits of our imagination.



         •  4. Estimating the Critical Mass of IT.
           

      We do not have any very good way to answer school board questions about, How much is too little? Few school boards would accept a coach assertion that 42 minutes a week of basketball practice, in groups of 24, will result in 42 offers of college athletic scholarships. Why then do we allow policy makers to believe that 42 minutes a week of "Computer" from the "The Computer Teacher" in "The Computer Center" will change children's school performance?



         •  5. IT Dosage.
           

      Most current evaluations assume that if a school has bought a site license for Electrified Reading: Release 2.0 then the teachers are using it and the children are experiencing (some unknown amount) of it. A generation of so-called "implementation research" suggests the fallacy of that assumption yet we do not have good measures of children's exposure or even of teachers use of programs. We need much more attention to elapsed time, exposure effects, dosage effects.



III. POLITICS
"The wireless music box has no imaginable commercial value. Who would pay for a message sent to nobody in particular?" Advice to David Sarnoff in the 1920's.

Politics is the process through which values are authoritatively distributed for a society. Whatever its interpretation in the popular culture, politics has deep implications for the purposes served by government action, for example, which children get what quality of schooling and which not? If measurement is the essence of science, benefit is the essence of public service, analysis may document who benefits from a particular program; politics determines how that benefit will be distributed.

Consider a choice among three public policy options: What would advance the children's interests more:

  1. Higher pay for already employed teachers?
  2. The same amount of money but spent only to hire additional teachers? Or,
  3. The same amount of money spent on instructional technology?

Or, assume that the policy goal is to "integrate technology into the classroom". The instant consensus is that can be done only by more professional development for teachers---more in-service, more released time, more contracts to Teacher Centers or, my own favorite, more subsidies for graduate school tuition. We never consider options such as:

  1. Giving teachers a computer to take home over the summer (and trusting their professionalism, curiosity and commitment)
  2. Putting computers into the classroom and letting the kids explore them and co-teach, co-learn with their teachers.
  3. Using technology to teach technology---for example, by producing CD-ROM role-playing simulations about what happens in classrooms under different conditions and with different teacher choice-consequence paths.

Neither do we consider making technology so 'transparent' that it does not require training, for example, ATMs. For the most part, policy choices are limited by political power and by the conventional conceptions of education as schooling and of learning as teaching. They all lead to the same labor-intensive conclusions and all are centered on teachers not on learning. We do not, for example ask the following question:

Under what conditions are protein-based teaching systems preferable to digital learning systems?

It is at least possible that digital systems do things that RLHB's

  • should not bother to do (keep records)
  • do not want to do (drill children)
  • can not do (have infinite patience) or
  • do not do reliably (treat all children as though they can all learn)?

Or consider "The Learning Odyssey", a complete curriculum for grades 4 though 9, produced by the Agency for Instructional Technology (AIT) that was originally aimed at the home-schooling market. The topics include language arts, math, science, history, art, music, technology and personal development. All subjects are aligned to state content standards. Teacher comments on student work are available by e-mail. Subscription prices are $150/month; $350/3 months; $900/9 months; and, $1,100/year. As part of the price: AIT will pay for a child to be tested with any standardized test required by a local jurisdiction.

AIT describes the "Learning Odyssey" as a replacement for school. "..(L)earning need not be school-based...schools must reinvent themselves as institutions with a far greater purpose, or cease to exist."8

Or consider another volatile issue, violence and the Internet. Post-Littleton, the concern to minimize the sources of violence in children's lives is likely also to generate unintended negative consequences. Our understandable attention to the harmful examples of Internet use may cripple wholesome applications of the same technology

We can help children by minimizing violence but also by maximizing good. If we would not ban all pharmaceuticals because some are hallucinogens, then we should also differentiate between pro- and anti-social applications of telecommunications. Except for a few one-off examples of good video games (Tetris, Carmen Sandiego), we have no systematic understanding of the good that can be done through learning related games and the Internet.

In order to encourage more wholesome development of these technologies, we need to understand how they work. To continue the pharmaceutical analogy, we need to identify the active ingredients in these applications followed by clinical trials to document their effects. We need clinical trials that (1) identify and measure the active ingredients of instructional technology and (2) that document the gains associated with amounts of their use.

Whatever the case, analysts, researchers and/or evaluators do not have the right to make deeply political, deeply value choices. Analysts are not elected or authorized by any constituency to make official decisions. Doctorates are not licenses to usurp state legislatures, local boards of education or even superintendents and principals. The role of analysis is to inform decisions, NOT to make them.


IV. ECONOMICS
"I think there is a world market for maybe five computers."
Thomas Watson, Chairman, IBM, 1943

A third of a century ago, James Coleman and Lawrence A. Cremin tried to teach us that we needed a more generous vision of education than one centered exclusively on schools. In addition, we need a more generous vision of progress than that which depends solely on public funds. There are partners who share important and child-centered purposes and they are in the private sector, perhaps especially in technology in the private sector.


    A. Capitol Decisions and Capital Decisions.

    Public and private decision-makers are interested in answering the same question: What works? They are because both are making investment decisions. "We need information to show what works and what doesn't. If we had empirical data, policy-makers would be more willing to fund technology and voters would be much more willing to pay."9

    The interest in 'what works' goes beyond public policy. There is now a category of inquiry called "curriculum due diligence". Because potential investors have a right to know about the integrity of what is being offered, banks and brokers retain curriculum analysts to document those companies in the learning business can actually deliver what they claim to sell---learning. Whether the goal is benefit to the constituents or return on investment, the interest in efficacy is the same.

    And in both instances there are competing uses for the same funds. Linda Roberts, the director of the Office of Educational Technology in the US Department of Education said, "School districts will be called to task for 'What are you doing with your money and what difference does it make?"10 In the public sector, the (implicit) questions are: More highways or more schools? More lights on police cars or more computers in classrooms? More scholarships for college students or more professional development for teachers? The private sector compares buying a magazine to creating software, or starting a chain of day care centers to creating an Internet homework helper site. All of those decisions can be illuminated by data about outcomes for learning.


    B. The Public Benefits of Private Investment.

    Where has there been more good for schooling? (A)The clouds of quarter million dollar grants from foundations and State Departments of Education to fund (non-profit) pilot projects and experimental activities or (B) two guys named Jobs and Wozniak trying to breadboard a "home computer" in a Cupertino garage?

    We should at least admit that in a capitalist society the engine of innovation---and yes, largely of improvement---is the profit motive. The need for parents to be better connected to their children's learning is widely acknowledged but who has invested more in creating a curriculum of the home? State departments of education or the Lightspan Partnership? (Hint: Creating the 100+ CD ROMs that support reading, language arts and mathematics, grades k through 6 has cost Lightspan $150 million.


    C. The Roles of Government.

    Recall that the Internet started out as a Defense Agency Research Projects Administration experiment to connect weapons labs. The Food and Drug Administration supervises clinical trials of privately developed pharmaceuticals on behalf of the public.


V. FOUR INSTRUCTIONAL TECHNOLOGY ADDITIONS TO THE EXISTING CONCEPTION OF SCHOOLING
"Computers in the future may weigh no more than 1.5 tons.""Popular Mechanics", 1949

    A. Adding an Education Focus to Our School Focus.

    There are a lot of educators in every society---the TV, newspapers, parents, religious and cultural institutions, video games, sports and the general culture. Every time we hear "education" and automatically think "school" we are diminishing the prospects of improvement.


    B. Adding a Learning Focus to Our Teaching Focus.

    The frontal act of instruction, the uncertain business of trying to require children to learn particular things is very difficult (ask any teacher). What if we re-conceptualized "the teaching of children" as "the facilitation of learning"? Likely, it is more possible to arrange learning than it has been to force teaching.


    C. Adding Homes to Schools: adding Parents to Teachers.

    Schools and homes remain isolated from each other. And, despite there centrality in the lives of children, we have never had very good bridges between the two. Using Lightspan's Achieve Now! Schools lend children a Sony Playstation (retail cost, $100) as a platform for learning-related video games that are launched by the teacher in the classroom but then completed by children and their parents at home. In a pre/post and experiemental/control evaluation, the children and schools with this home-school-home connection performed better on reading, language arts and mathematics than did those without.


    D. Moving Learning to the Learner.

    One certain consequence of digital technology is that learning will go to the learner. In the earliest times, boys went with their fathers and uncles to observe the hunt; girls went with the mothers and aunts to discover which plants were edible. The artists of the cave walls moved learning inside. The creation of the common school still required learners to go to the site of learning and to dependent on the knowledge masters. Dependency makes learners vulnerable to the political (and ethnic and class and gender) prejudices of the masters.

    Digital communications reverses that commerce (with the Internet, learning goes to the learner) and dramatically transforms that politics. Digital learning can be "The 4 'Anys'---Any Learning, Any Time, Any Place to Any One". The democratizing impacts of that reversal are heartening but only dimly perceived. And the consequences for schools and universities, conceived as physical spaces, have not begun to be imagined although their consequences are probably captured by the observation about technology as a 'train'--you will either be on it or under it.


VI. FOUR EVALUATION RESEARCH QUESTIONS FROM FOUR PERSPECTIVES
"But what is it good for?" Comment on the microchip from IBM Advanced Computing systems Division, 1968
  • From the perspective of science, How certain, how unambiguous, how compelling are the data?
  • From the perspective of pedagogy, What implications for practice can be derived from the evaluation?
  • From the perspective of politics, What political values are served?
  • From the perspective of economics, What are the cost or financial implications?

About the Author: Dale Mann is a professor in the Program in Educational Administration at Teachers College, Columbia University and Managing Director of Interactive, Inc., a technology development and evaluation firm Huntington, New York.


FOOTNOTES

1. Dava Sobel, Longitude: The True Story of a Lone Genius Who Solved the Greatest Scientific Problem of His Time, New York, Walker & Co., 1995, pp 11-13. Sobel continues, "[the night before, the Admiral had been approached by a sailor]...who claimed to have kept his own reckoning of the fleet's location during the whole cloudy passage. Such subversive navigation by an inferior was forbidden in the Royal Navy...[the Admiral] had the man hanged for mutiny on the spot."

2. Pamela Mendel, "Issues That Defined the Year for Schools and Computers", The New York Times, Wednesday, December 30, 1998.

3. For some examples, see Charol Shakeshaft's paper also submitted for The Secretary's Conference on Educational Technology and attached herein. For a recent summary of 264 research reviews and reports, see Jay Seven-Kachala and Ellen R. Bialo, 1999 Research Report on the Effectiveness of Technology in Schools, 6th Edition, Software & Information Industry Association, Washington, D. C., 1999.

4. William L. Rukeyser, "Computers' Role in Education", the website of "Learning in the real world", Woodland, CA,.

5. Jane M. Healy, "The Mad Dash to Compute", The School Administrator, v. 4, n. 56, pp. 6-10. C. F., J.M. Healy, Failure to Connect: How Computers Affect Our Children's Minds - for Better and Worse, New York, Simon & Schuster, 1998.

6. Many school people have adopted a constructivist interpretation of the work of children while continuing to focus on teachers as the sole source supply of children's learning. As technology becomes more sophisticated (for example, through artificial intelligence), this singular fealty to the school's employees will be harder to maintain. For example, the experience of the Internet suggests the powerful extent to which technology itself teaches with or without the mediation of adults.

7. C.F., Dale Mann, "Serious Play", Teacher College Record, v. 79, n. 3, Spring 1996, pp. 446-469.

8. Michael Sullivan, Executive Director, AIT, "The Heller Report: Internet Strategies for Education Markets", v. 4, n. 2, August 1998, p. 4.

9. Lieutenant Governor Kim Robak, Nebraska, Quoted in Kerry White, "A Matter of Policy," Education Week, November 10, 1997, v. XVII, n.11, "Technology Counts: Schools and Reform in the Information Age," p.6.

10. Pamela Mandels, "U.S. Official Calls for Studies of Technology in the Classroom", New York Times, April 27, 1998, p. B27.

 Previous Table of Contents Next 

 
Print this page Printable view Send this page Share this page
Last Modified: 08/23/2003