Rossi and Freeman (1993) define evaluation as "the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of ... programs." There are many other similar definitions and explanations of "what evaluation is" in the literature. Our view is that, although each definition, and in fact, each evaluation is slightly different, there are several different steps that are usually followed in any evaluation. It is these steps which guide the questions organizing this handbook. An overview of the steps of a "typical" evaluation follows.
Evaluators often talk about two different types of evaluations: "formative" and "summative." A formative evaluation is usually conducted in the early stages of a program and addresses questions about implementation and ongoing planning. This type of evaluation typically examines process rather than product. In the case of technology programs, a formative evaluation might ask if equipment was received on time, if the expected number of teachers received professional development, what skills teachers learned from professional development, or if the equipment is being used in the classroom as planned. Formative evaluations are useful for various purposes. For example:
The following are some examples of formative evaluation:
Example A - "Why didn't they show up?" At the beginning of the school year, Smalltown High School decided to provide the first part of a two-part teacher professional development series on the integration of technology into the classroom. While the teachers attending the professional development were satisfied and seemed to benefit from it, the turnout was lower than expected. Upon further investigation, the evaluators found out that, since the professional development was offered after school, many teachers who were involved in leading extra-curricular activities could not attend. Learning this, evaluators surveyed teachers about their preferences and availability. From the survey results, the decision was made to have both Saturday training and training during the summer.
Example B - "Things don't always go as planned." During the summer, Big City Elementary completed all its wiring. The school's plan was to install equipment in the fall and provide professional development for teachers that would teach them how to use the new equipment in the spring. In December, as part of its overall evaluation plan, the school assessed whether the equipment had been installed as intended. It turned out that most of the computer equipment was backordered and would not be installed in time for the professional development. Evaluators determined that, because of the delay, prices had dropped and better equipment could be obtained at a similar cost. The school quickly updated specifications and was able not only to receive the equipment in time for the spring professional development, but to purchase better equipment than originally intended.
Summative evaluations assess program outcomes or impacts. To determine the relationship of different factors to outcomes, similar to formative evaluations, some information used in summative evaluations is collected early in the life of a program (e.g., baseline data, test scores). Unlike formative evaluations, however, a portion of the information is collected after the program has been completely implemented and adequate time has passed to expect outcomes to occur. In terms of evaluating educational technology, a summative evaluation might ask if teacher technology skills improved as a result of a professional development activity, if teachers are using technology to a greater extent in their instruction, or if technology improved student motivation or performance.
Some advantages of summative evaluations include:
Examples of summative evaluation follow:
Example A - "Which type of professional development is better?" Administrators at Smalltown High had two different approaches to providing professional development to teachers on the use of technology in their classrooms. One group felt that, since many teachers did not have extensive computer knowledge, it was important to teach them how to use the software before teaching them how to integrate the software into instruction. This group proposed a six-week schedule (one session per week), with three sessions on the software itself and three sessions on integrating it into teaching. Another group felt that teachers would learn how to use the software as a result of it simply being available, so this group proposed a full six sessions focusing solely on curriculum integration. The school decided to test both options. Teachers signed up for professional development and then were randomly assigned to one of the two types. One year after participating, evaluators surveyed the teachers on the use of different types of software in their instruction and observed teachers' use of technology in the classroom.
Example B - "Test scores, test scores." One year after spending a large amount of money on technology, parents of students at Big City Elementary started asking whether the new technology would improve test scores. As the school planned on adding new technology grade-by-grade, a decision was made to follow cohorts of students over time to assess changes in math and reading scores. Specifically, the school was interested in comparing the class of students going through the school two years before the technology was implemented to the class two years after the technology was implemented. To assess long-term impacts, the school would monitor test scores of the students through graduation from high school.
-###-
![[Table of Contents]](/icons/up.gif){kind=icon}
[Where Do I Start?]