Data & Research EVALUATION OF PROGRAMS
Executive Summary, Volume I: Final Report -- Field-Focused Study of the Comprehensive School Reform Demonstration Program (2003)
Downloadable File PDF (792 KB) | MS Word (710 KB)

The Comprehensive School Reform Demonstration Program. Comprehensive school reform calls for a "whole-school" and coordinated approach to improve schools. The strategy differs from piecemeal and fragmented efforts that also in the past have seemed only to lead to short-lived changes.

To stimulate whole-school reform across the country, Congress appropriated funds in FY1998 for the U.S. Department of Education (ED) to start the Comprehensive School Reform Demonstration (CSRD) Program. ED allocated the funds on a formula basis to states, who made awards to support 1,840 mostly Title I schools "in need of substantially improving" their performance. Subsequent rounds of annual awards to support additional schools have continued through FY2003.

In applying for and accepting these funds, schools were expected to implement 9 components, one of which is an effective, research-based method or strategy. Together, the 9 components comprise the comprehensive reform aimed at improving student achievement:

CSRD Components

No.Component
1.Effective, research-based methods and strategies
2.Comprehensive design with aligned components
3.Professional development
4.Measurable goals and benchmarks
5.Support within the school
6.Parental and community involvement
7.External technical support and assistance
8.Evaluation strategies
9.Coordination of resources

The awards are for three years and had to be a minimum of $50,000 per year. Schools can only receive a single CSRD award. Given these conditions, federal funds therefore are intended to serve as "seed money" for whole-school reform. Beyond this period of time, schools are to continue reform with their own resources.

The Field-Focused Study of CSRD. ED commissioned the Field-Focused Study of CSRD as one part of its national evaluation of the CSRD program. The study began by randomly selecting 18 schools from the first two annual rounds of CSRD awards (1998-99 to 2000-01, and 1999-00 to 2001-02). Field data were collected through four site visits to each school, conducted during 2000-02.

The objective of the Field-Focused Study was to provide initial feedback about the implementation of the CSRD program. The topics covered by the study and in this report include:

  1. The progress in implementing the 9 CSRD components by the 18 schools;
  2. The role of district and state influences in implementing the program;
  3. The apparent pathways or strategies for implementing comprehensive school reform; and
  4. The early signs regarding the potential sustainability of the program at the school level.

Conclusions on CSRD Implementation: A Mixed Pattern

The main priority of the Field-Focused Study was to determine the extent and nature of CSRD implementation at the 18 schools. Overall, the schools' implementation of CSRD at the 18 schools was uneven. Based on a 47-point instrument reflecting the 9 components and devised specifically for the Field-Focused Study, nine schools garnered 80 percent of the 47 points and could be labeled as "nearly fully" implementing CSRD. Another six schools garnered 65 percent of the 47 points and could be labeled as "partially" implementing CSRD. The remaining three schools all scored lower and were judged to be "poorly" implementing CSRD.

Implementation of the 9 CSRD Components. Implementation also was uneven across the 9 components (to streamline the discussion, several related components have been paired):

Component 1 (see Section 2.1 of the main report for more details):
The bulk of the schools' CSRD resources and attention has been devoted to the implementation of a research-based method. By the time of the final site visit, implementation of such a method was proceeding fully at 9 of the 18 schools, with partial or minimal implementation at the other 9;

Component 2 (see Section 2.2):
Less than half of the 18 schools had reforms with a comprehensive design, reflected by a comprehensive plan combined with staff awareness that CSRD should extend to a school's entire way of doing business and all its operations—and not simply adding a new function or project or activity;

Components 3 and 7 (see Section 2.3.1):
Both professional development and external technical support and assistance had largely been devoted to the implementation of the research-based method, not necessarily comprehensive reform more broadly;

Components 4 and 8 (see Section 2.3.2):
Both the measurable goals and benchmarks and evaluation strategies had been devoted to tracking student performance, not necessarily implementation progress;

Components 5 and 6 (see Section 2.3.3):
High turnover among staff and students had resulted in transient levels of either support within the school or parental and comunity involvement;

Component 9 (see Section 2.3.4):
Most schools were in a position to coordinate or converge resources, but resources for sustainability were still uncertain.

District and State Influences. The Field-Focused Study also collected data about district and state actions potentially affecting the schools' CSRD implementation. Some of these actions were part of the CSRD administrative procedures, because states implement CSRD by having districts apply competitively on behalf of some or all of their schools. In the process, both states and districts can support or monitor the schools' CSRD efforts. The study found varying degrees of such support (see Section 3.1.1).

More important than these procedures related directly to the administration of CSRD, the study uncovered other important state and district policies, not directly related to administering CSRD, that nevertheless influenced CSRD implementation (see Section 3.1.2). Some conditions, such as extremely limited professional resources, had a negative influence on CSRD. Other conditions, such as the direct alignment of CSRD designs with district improvement plans and state standards, had an extremely positive influence. Other conditions reflected the ongoing dynamics of school systems—e.g., districts reducing financial support for all external research-based methods, a district allowing a CSRD school to become a charter school, and a district deciding to merge two schools that happened to be CSRD schools.

Strong district or state influence, creating a "vertical" alignment to the school level, led alternatively to either complementarity or conflict with CSRD. As examples on the complementary side, CSRD provided two schools with resources and a compatible reform agenda to respond to their designation as underperforming schools in the state's accountability system. As examples on the conflicting side, the shifting content of state assessments led districts to use resources for alternative curricula and professional development that were contrary to those involved in a CSRD school's original plans or implementation. In general, these external state and district conditions appear to be highly relevant to CSRD implementation.

Conditions Associated with Successful Implementation and the Role of State and District Influences: Three Pathways to Reform

The study identified three different sets of conditions, or "pathways," that appeared to be associated with the successful implementation of CSRD (see Section 3.4). The first pathway is a component-driven pathway, whereby a school uses the 9 CSRD components to guide the development and implementation of a comprehensive reform. The second is a method-driven pathway, whereby the school adopts and implements a comprehensive research-based method that affects virtually all school operations and whose successful implementation substitutes for the need for any independent articulation of the 9 CSRD components. (However, many research-based methods focus on specific curricula and are not comprehensive.) The third is a vertical-driven pathway, whereby a school articulates and pursues the needed comprehensive strategies as a result of state and district requirements involving: the setting of standards, use of appropriate assessment tools, and required alignment of district- and school-based strategic planning and improvement plans to meet state performance standards.

No single pathway was considered the "best" or preferred pathway, and no pathway was necessarily more immune than the others to such disruptive conditions as: high principal turnover rates, limited professional development resources, or planned or unplanned school restructuring.

Sustainability of "Whole-School" Reform: Still Questionable Given Current Fiscal Climate

As a final topic, the study examined the prospects for sustaining school reform beyond the final year of CSRD funding (see Section 3.5).

Neither the original legislation nor ED defined the exact nature of a school's changes to be associated with sustaining a comprehensively reforming school beyond the three-year CSRD award period. As a result, the Field-Focused Study examined two different views of sustainability and judged the 18 schools according to both.

The first view, based mainly on the experiences of the New American Schools initiative, holds that the central changes to be sustained should be the practices associated with the originally-supported research-based method. The second view is that comprehensive reform, though embracing a research-based method, also transcends it. By this second view, successful sustainability would not necessarily be associated with the continued use of any particular method but could involve transitions from one researchbased method to another, over time. The transitions would have to reflect a progression toward continued school and student improvement rather than the "churning" of innovative practices.

Using the most lenient benchmark and accepting either of these two views as a criterion for assessing sustainability, 14 of the 18 schools were exhibiting a promising level of sustainability by the time their CSRD awards were ending. Accepting only the second view reduces the number to 11, and accepting the first view reduces it to 10.

The main barrier to sustainability continues to be the limited availability of sufficient resources, especially in light of states' and districts' revenue shortfalls in recent years. To sustain a reforming process, even when existing resources have been coordinated and targeted to reform, still requires discretionary funds to support such essential activities as: adequate professional development (including support for teacher substitutes), especially in situations of high teacher turnover; time for common planning periods or teachers' work on school leadership teams; and support for external technical assistance. Though such needs can be served with modest levels of funds, serving the needs is still a discretionary activity that may have to be ignored if core school operations are underbudgeted.

Volume II: Appendices -- Field-Focused Study of the Comprehensive School Reform Demonstration Program (2003)


 
Print this page Printable view Send this page Share this page
Last Modified: 05/27/2004