![[Contents]](/icons/up.gif){kind=icon}
[Chapter 6: Cross-Case Analysis 3: Part 1 of 3] Assessment of Student Performance April 1997
The Adequacy and Politicization of the Timeline for Reform
The adequacy of the amount of time allowed for development, introduction, and institutionalization of assessment reform can have a dramatic impact upon a state's ability to sustain its reform efforts and to meet its various objectives. In particular, sufficient time must be allowed for developing, piloting, and refining assessments and for collecting baseline data if a state is to monitor student progress accurately.
The pressure to produce results is typically intensified when the introduction of the performance assessment takes place in the political realm. Furthermore, changing political climates can threaten the reform.
Three of the six state-initiated performance assessments included in this study resulted from an act of the legislature — those in Arizona, Kentucky, and Oregon. In the case of Oregon, the state department of education, district officials, and teachers worked toward the stated goals of an act of legislation only to have those goals dramatically revised a couple of years later. In 1991, the Oregon legislature enacted the Oregon Educational Act for the 21st Century, mandating, among other reforms, a system of student certification. Four years later, however, the legislature revisited the Act, eliminating the Certificate of Initial Mastery outcomes and replacing them with content standards.
Such vacillations in the elements of a reform effort leave teachers, administrators, and district and state officials with a dilemma regarding how much time and effort they should invest in the reforms. At the schools in Oregon, Arizona, and Kentucky participating in this study, the political uncertainty underlying the performance assessments resulted in teachers' skepticism about the longevity of the new assessments and desired student outcomes. The Oregon high school that had participated in the state's assessment reform effort withdrew from the pilot assessment development project. Teachers at the Arizona school said they would not invest substantial effort on ASAP until it was clear the assessment was "here to stay," and a school board member in the district visited in Kentucky noted that the number of representatives in the Kentucky legislature who supported KIRIS was dwindling and that KIRIS could be dramatically altered or even abolished in the not too distant future. In such atmospheres, teachers and administrators we interviewed were cautious about investing their time and effort to support the state's objectives, as those objectives may soon change or be eliminated.
New York's, Vermont's, and Maryland's timelines for reform are, as yet, unthreatened by politicization of their reform efforts. In the case of New York and Vermont, the assessment systems are voluntary. Furthermore, additional components are only gradually being added to the assessment system. In addition, in the case of New York, the reform efforts are primarily local, leaving little room for large-scale disenchantment. In the case of Maryland, the system is mandatory and is being implemented state-wide. However, Maryland's investment in public relations, in avoiding sensitive topics, and in involving teachers and local district educators in formulating the system has paid large dividends in terms of the lack of outright public opposition.
Professional Development
Professional development clearly is a critical component of a state's efforts to introduce assessment reform. Our findings indicate that teachers' understanding of the assessment — its purposes, format, pedagogical underpinnings, scoring procedures, and consequences — and their ability to work with the assessment are crucial to progress toward attaining the state's purposes for the assessment system.
Professional development is perhaps most crucial to those assessments that have among their purposes the goal of influencing classroom curriculum and instruction. As has been noted, findings regarding facilitators and barriers in assessment reform with respect to this purpose will be discussed in Chapter 6, and for this reason the discussion of professional development as a facilitator of reform will be cursory in this chapter. In general terms, however, it should be stressed that the professional development provided to teachers typically serves to acclimate them to the new assessment environment, thereby increasing the likelihood that they will sympathize with (or at least understand) the state's other objectives and decreasing the likelihood that they will actively oppose the new assessment.
In the five states introducing performance assessment systems statewide, teachers identified, at a minimum, value in the professional development they received with respect to the mechanics and logistics of the new assessment. Furthermore, teachers in Kentucky and Vermont identified additional, more general value to professional development activities concerning the assessment.
Kentucky's and Vermont's continual refinement and provision of professional development marked a clear shift in the attitudes of the teachers we interviewed in these two states. From one year to the next, teachers said that they had become more comfortable with and more proficient in the use of performance assessments. They also stated that they improved their skills in scoring portfolios during these professional development sessions. By providing professional development opportunities that led teachers to improve their skills in working with portfolios, these two states were also, at least in theory and over time, furthering the technical quality of their assessments in terms of the interrater reliability achieved.
School-initiated performance assessments are typically developed to meet four purposes, in some combination:
Our findings suggest that, at the school level, these four purposes are fairly compatible with each other; that is, it is possible to use a single assessment reform to meet all of them. Several factors, however, contribute towards a school's ability to meet these purposes. These factors include:
Each factor, as it serves as a facilitator or a barrier, is considered below.
Waivers
Waivers can serve as a facilitator of assessment reform by freeing schools from external mandates that are either incompatible with the reform or that compete too much with the reform for limited teacher time. In our sample, three of the four elementary schools involved in school- or nationally-initiated, school-adopted assessment reform received waivers from district-mandated report card systems. The one school-initiated high school system has obtained a waiver from district assessment requirements. Waivers obtained by the seven schools that are involved in school- or national-level assessment systems are reviewed in Exhibit5-3.
In addition to the above waivers obtained by schools from district requirements, the Anton, Iowa school district, of which Noakes Elementary School is a part, applied for and received a five-year waiver from state requirements to "report measures of student progress" at the time the district elected to participate in the New Standards Project.
These two types of waivers allow teachers to formulate their curricular materials, performance standards, and assessment systems more logically than they believe they could in the absence of the waivers; in turn, they are better able to achieve the purposes stated for school-initiated assessment reforms. Teachers using narrative report cards (as is the case at all three schools with waivers from district reporting requirements) testified to the lack of a link between performance-based assessment and traditional report cards. Teachers at other schools without these waivers expressed the same reaction. For instance, teachers at Thoreau would like to use narrative report cards (and, indeed, briefly experimented with them at one point) but say that the university system in the state was unwilling to work with narrative reports. (Thoreau is one of two high schools among these seven schools. Using narrative reports in lieu of traditional report cards may pose unique problems at the high school level as colleges and universities seek comparable data about students across schools.) Teachers at Sommerville (the other high school) also identified an incompatibility in assessing students' performance in an alternative way but reporting it in a traditional way.
The problem with not obtaining waivers from testing or reporting requirements is highlighted by Cooper Middle School. Teachers involved in developing and implementing Cooper's extensive performance assessment system testified to the difficulties inherent in teaching their students skills for taking the Iowa Test of Basic Skills (ITBS) while also preparing them for the types of assessments they normally conduct in their classrooms (the content coverage and the types of skills required for the two types of testing systems are different). In addition, students spoke about the difficulties in understanding the presentation of their grades by certain subject areas when much of the teaching and learning that goes on in their classrooms is thematic.
Availability of Information and Resources
The most important information resource available to four of the seven participant schools comes through their participation in national-level education reform efforts: Ann Chester and Noakes in NSP, Cooper in CES, and Sommerville in Pacesetter. However, it is important to note that the other three school-level reforms studied here (Ni?os Bonitos, Thoreau, and Park) also draw upon the work of NSP and CES to some extent in developing their performance assessments (Ni?os Bonitos works with NSP, Park with CES, and Thoreau with both). Teachers at all seven schools are accustomed to seeking information from outside sources and using what they find useful to guide their reform efforts.
Teachers at Ann Chester, Noakes, Cooper, and Sommerville have all participated in the national conferences, symposia, and institutes offered by the national-level reform effort in which each school participates. The impact these professional development opportunities have had on teachers' abilities to construct and use assessment tasks will be explored in Chapter 6.
The extent to which participation in national reform efforts has fostered each school's ability to change or alter assessments varies across the four schools. At Noakes and Ann Chester, for example, participation in NSP is an integral piece of the school's reform efforts. At Noakes, teachers' adoption of NSP tenets and methods is central to reform, and at Ann Chester, NSP is compatible with and supports a district-initiated reform effort, the Applied Learning Program, which focuses on capacity building at the local level.
Similarly, at Sommerville High School, the Pacesetter mathematics program has a dramatic impact upon how math is taught to enrolled students, but there is little spillover of the curriculum, instructional methods, and assessment practices from Pacesetter into the school's other math courses. However, this lack of school-wide impact is not at odds with the school's goal for Pacesetter: to teach math better to a particular group of students.
Cooper Middle School also benefits from the information resources it gains through its participation in the Coalition of Essential Schools, which provides information to support the school's goal of more fully integrating curriculum, instruction, and assessment. For example, Cooper's Rite of Passage Experience and Planning Backwards are based upon the Coalition principles. However, the professional training support Cooper obtains is far less extensive than the support received by the three other national reform effort participants included in this study.
Cooper teachers articulated the need for much more professional development and support on an on-going basis. The stress generated by the lack of professional support was evident in the decision of Cooper teachers to go back to teaching mathematics in a traditional way, as they felt that they had insufficient mastery in teaching mathematics through thematic units (as they had been doing).
Teachers without ready networks to turn to have also found outside sources of information to be useful as they develop and implement their performance-based assessments. Teachers at Ni?os Bonitos Elementary School described the difference between developing their school's language arts learning outcomes and portfolio assessment in the late 1980's with developing their mathematics outcomes and portfolio assessment several years later. When they went to work on the language arts assessment, they found no models to guide them. Though they obtained some assistance (from outside experts, including Dennie Wolf and Grant Wiggins) in developing the portfolio assessment, teachers basically worked alone to develop the school's language arts learning outcomes, scoring rubrics, and assessments. In contrast, when they turned to mathematics a few years later, teachers had the resources of the NCTM standards and content areas. They also were able to tap the experiences of colleagues in other schools, for the school was no longer alone in its shift towards performance-based assessment. Ni?os Bonitos is now refining its assessments and content standards based upon the work conducted by the New Standards Project. In addition, the school participates in the National Alliance for Restructuring Education and has access to the expertise and models of educational reform of that organization.
Teachers at Thoreau High School, which pioneered its Rite of Passage Experience more than 20 years ago, "went it alone" for years. (Thoreau's principal tells the story of the school's application to be an associate member of the newly-founded CES. The application described the school's Rite of Passage Experience and the philosophy underlying the school, both of which resonated perfectly with the principles upon which CES was founded. The principal received a phone call from a dumbfounded Ted Sizer, who asked, "Who are you guys and how did you do all this?") They now participate (in small ways, according to teachers and the principal) in both CES and NSP. However, these teachers are accustomed to doing ROPE their way and remain convinced that they know how to do it best. They say they adopt ideas from CES, NSP, and other efforts they read about, but they are not interested in "swallowing hook, line, and sinker" something new. Teachers at Park Elementary School, also a long-time leader of child-centered education reform, are similarly accustomed to working within their group to identify appropriate methods of integrating assessment with instruction, turning to outside resources as they find them to be useful. For example, the New York City Assessment Network provides the individualized support and guidance Park teachers need in order to adapt the Primary Learning Record to their particular classroom situations.
In each of these seven cases, informal and semi-formal support networks and professional development sessions have facilitated schools' efforts to develop assessment systems.
Time and Organizational Structures
Perhaps the most difficult barrier to confront assessment reform initiated at the school level is finding the time for teachers to design, to experiment with, and to refine the reform itself. If any single obstacle to assessment reform was mentioned at every site participating in this study (both at the seven sites with school-initiated assessments and at the nine schools involved in district- and state-initiated assessments), it was the factor of insufficient time.
In contrast to state-initiated assessment reforms, however, schools do enjoy the luxury of undemanding timelines for introducing the reforms: though it may be difficult for teachers to find enough time to devote to designing the reform, they are typically not under pressure to introduce it by any specific date. Teachers at Ni?os Bonitos took four years to develop their language arts learning outcomes and portfolio assessment; teachers at Park "dabbled" in using the PLR and joined other teachers who were using it voluntarily if and when they decided the tool would be valuable to the teaching and learning that goes on in their classrooms; and teachers at Thoreau make adjustments to ROPE as they decide they are necessary, but remain content with the current version until they have time to make changes. In this sense, loose timelines for reform clearly serve as facilitators of reform, or revisions to reforms, taking place over the long run.
However, finding time — both to design and to use — performance-based assessments remains a sizable barrier to assessment reform. In the words of an educator supporting the implementation of the Primary Learning Record at Park Elementary, "Time must be legislated into the school day [or week] to allow teachers to talk to one another, to share their experiences, and to work together to understand what's going on in the classroom." Some schools have addressed such issues by modifying school schedules to provide teachers with more time (through release time or through joint planning time) to think about and to develop assessment tasks and rubrics, as well as to plan instruction.
The resource of "time" has been provided on an ongoing basis in some form or another at five of the seven sites involved in school-initiated or national-initiated assessment reform. Common planning time (sometimes in the context of team teaching and sometimes in the context of teacher-initiated "study groups") has been introduced at Cooper, Park, Ann Chester, Sommerville, and Ni?os Bonitos. In addition, release time — to attend conferences, scoring sessions, or to develop assessment tasks and rubrics — has also been provided by schools to teachers at Sommerville, Ni?os Bonitos, and Ann Chester. Though teachers lament that they still lack the time to carry out reforms well, they do value the time that is allotted to help them better learn to use performance-based assessments.
At Noakes, teachers participate in district-wide professional development sessions every other Wednesday. However, teachers also invest a fair amount of their own time; as one teacher said, "It [performance assessments] puts stress on our free time, after school time, and weekends." At Thoreau, on the other hand, the assessments are so thoroughly institutionalized that no new schedules or structures have been developed to support the adoption of these assessments. New teachers are inducted into the school's education philosophy and approach by pairing them with veteran Thoreau teachers.
The three district-initiated assessments included in this study are so distinct from one another in terms of purposes of the assessments, format, scoring procedures, and the subject areas assessed that it is not possible to identify a set of common facilitators and barriers that pertain to district-initiated assessments. Rather, each of the three assessment reform efforts initiated at the district level shares some characteristics with assessment systems introduced at either the state or the school level. Therefore, in this section, we will:
Each of the three district-initiated performance assessment systems is considered separately.
Prince William County: Applications Assessments
As noted in Chapter 4 (See Exhibit 4-1), the Prince William County Public Schools' Applications Assessments were in the pilot and initial implementation phase at the time of this writing. The purpose of the assessments are to monitor student progress and to influence instruction. The major barrier thus far to both of these objectives is that the district's curriculum frameworks are not yet fully revised. Although all reforms are being guided by the district's Quality Management Plan and the six Standards of Quality, the concrete curriculum goals have not as yet been finalized. Furthermore, most teachers at the middle school included in this study were not aware of the substance of the new curriculum frameworks.
A second factor that might work against the goals of the assessment system is the lack of professional development in how to use the assessments for instructional purposes in the classroom. Putting assessment reform before revisions to curriculum frameworks has left teachers wondering about the future of the assessment and about just what they should be emphasizing in their classrooms. However, it must be emphasized that assessment development and administration is in the early stages in Prince William County. Hence, the potential barriers in this early stage may very well be hurdled down the road.
South Brunswick: Sixth Grade Research Performance Assessment
The South Brunswick Township Public Schools' Sixth Grade Research Performance Assessment is intended to assess 6th graders' research skills and to align instruction with curriculum to teach research skills. The district's specification of the research skills 6th graders should master, coupled with the perception that the schools were not graduating students with adequate research skills, served as a facilitator of the development of the assessment. In addition, technical help from the nearby Educational Testing Service facilitated the design of the assessment.
A potential barrier to the continual success of the assessments is that the assessment may be unfair to students with disabilities, as procedures for implementation and the scoring criteria are identical for all students, and no special arrangements are made for students with disabilities. Nonetheless, because of the low-stakes nature of the assessments, this potential barrier has not served to change the assessment purposes or the assessment format. As a result, district teachers fine-tune the assessments on an annual basis and continue to experiment with the scoring and implementation procedures of the assessment.
Harrison School District 2: Performance-Based Literacy Assessments
Harrison School District 2's Performance-based Literacy Assessments are intended to monitor student performance, to teach students to become self-assessors, and to serve accountability purposes. The impetus for reforming the district's language arts curriculum derived from the state's requirement that every district in the state develop standards-based education in reading and writing by 1994-95. (The state's requirement in other areas has a longer timeline.)
Given the state mandate, the district developed a detailed language arts curriculum that provided the basis for designing the aligned assessment system. Each assessment is keyed to one of the district's 13 significant student outcomes appropriate for the developmental level of students at different grade levels. In addition, the district involved every language arts teacher in the process of reforming the curriculum and also sought the help of experts in performance assessment in designing the coordinated system. Next, the district provided teachers with professional support (a peer coach model within schools) and development (through an "Assessment Academy") as well as assessment tasks to use in their classrooms.
In order to meet the purpose of teaching students to become self-assessors, the district devised scoring rubrics specifically for student use, and in order to meet the accountability purpose, the district devised scoring rubrics for teachers. Teachers use their scoring rubrics to assess student work and to report student scores to the district, and students use their scoring rubrics to assess themselves. The district, however, reports aggregate scores back to schools and does not use the results for high-stakes purposes.
The combination of aligned reforms, professional development, and use of assessment results for feedback purposes has resulted in a relatively smooth reform process. In addition, the state requirement for locally-determined, standards-based education provided the essential framework for the reform to be initiated.
The two barriers to effective implementation of the performance assessments teachers most often noted, however, were the lack of time and the lack of effective instructional models. An additional barrier both district and school administrators noted was that the parent community is not ready for narrative report cards because they continue to expect traditional grading. However, because the district is taking a long-term view of the reform process, this barrier is not a salient one for the Harrison education community.
Our study findings indicate that several factors function either as facilitators or barriers in the assessment reform process, depending upon their presence or absence. These facilitators and barriers either enhance or restrict a state's, school's, or district's capacity to design, develop, and implement assessment systems to meet certain purposes.
To reiterate, the factors that serve as facilitators or barriers at the state level are:
These factors, however, do not function independently. Our study indicates that the potency of any one factor depends, in part, on the presence and strength of other factors. For example, even if an assessment system is designed and based upon content standards, the perception that it is unfair may undermine its credibility (e.g., Arizona's performance assessment system). Thus, the system may not be used for accountability purposes.
At the school level, the factors that serve as facilitators and barriers in assessment reform are:
The presence of these factors serves as a facilitator in assessment reform while the absence of these same factors serves as a barrier. Assessment at the school level is intended primarily for monitoring student progress and for improving pedagogical practices. Hence, when teachers obtain time, support, and freedom from certain regulations that are antithetical to assessment reform, they are much more able to design systems that are tailored to meet their own classroom needs.
Our sample of district-level assessment initiatives indicates that the kinds of facilitators and barriers districts experience can be similar to those that states face, though on a smaller scale, or similar to those that schools face. In large part, the existence or absence of various facilitators and barriers is reflective of the district's purposes in introducing assessment reform.
In the next chapter, we extend further our analysis of the facilitators and barriers in assessment reform by focusing on the factors that support or impede the purposes of assessment reform at the school and classroom levels. Specifically, we explore the factors that affect teachers' abilities to use performance assessment technologies to guide meaningful changes in their teaching practices.
3Because district-initiated assessments encounter some of the same facilitators and barriers experienced by states or by schools, those performance assessment systems will be considered after an analysis of the facilitators and barriers that face schools initiating performance assessment systems.
-###-
[Chapter 5: Cross-Case Analysis2: Part 1 of 2] [Chapter 6: Cross-Case Analysis 3: Part 1 of 3]