April - May 2002
Part II. Scientifically Based Research (SBR)
Few issues surrounding the CSR program have generated as much recent interest as the requirement that practitioners use scientifically based research (SBR) to make decisions about school programs. Because this requirement has also generated considerable anxiety, the symposium agenda devoted a significant portion of time to addressing ways in which SEAs can help school practitioners learn to use research to inform program development-particularly for instruction in the core subjects-and to cultivate a critical consumer mentality when considering methods, strategies, and models to include in a CSR program.
For all federal programs the reauthorized ESEA(with the exception of Reading First, which has its own definition), SBR is defined as "research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs." To meet the standards of SBR, research must:
- Employ systematic, empirical methods that draw on observation or experiment;
- Involve rigorous data analyses that are adequate to test the stated hypotheses and justify the general conclusions drawn;
- Rely on measurements or observational methods that provide reliable and valid data across evaluators and observers, across multiple measurements and observations, and across studies by the same or different investigators;
- Have been evaluated using experimental or quasi-experimental designs in which individuals, entities, programs, or activities are assigned to different conditions and with appropriate controls to evaluate the effects of the condition of interest, with a preference for random assignment experiments, or other designs to the extent that those designs contain within-condition or across-condition controls;
- Ensure that experimental studies are presented in sufficient detail and clarity to allow for replication or, at a minimum, offer the opportunity to build systematically on their findings; and
- Have been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective and scientific review.
There are several reasons why understanding SBR and research-based decision-making is so important in the CSR program:
(1) CSR demands a substantial investment of time and money
Stakes are high for building successful CSR programs because of the financial investment they reflect. Each school receives a minimum of $50,000 each year for a maximum of 3 years, and many schools also access other funds drawn from local, state, or other resources to design their comprehensive program. One estimate is that the investment can reach as high as $588,000 per school. CSR programs also require an enormous investment of time on the part of school staff and on the part of the system to invest in building staff skills and knowledge to successfully design and implement a truly comprehensive program.
(2) Implementation matters
CSR programs must be based on solid research because studies have consistently shown that full implementation of a school's comprehensive program is a key factor in whether it will succeed or not. According to studies of national school reform models, those schools that have good or at least adequate implementation have much stronger outcomes than schools that have poor or inadequate implementation.
(3) The law requires it
Finally, research-based decision-making is important to understand and apply because the CSR program legislation requires it. The mandate to use research appears in two of the eleven CSR components, which apply to both individual strategies and methods and to the school's CSR program as a whole.
Once it is clear why a critical consumer mentality is so important, practitioners need the tools and training to cultivate this perspective and learn how to apply research findings to the development of a comprehensive school reform program. Appendix C of the guidance, "Scientifically Based Research and the Comprehensive School Reform Program", contains several important tools to assist practitioners and a list of key resources for schools.
Although applying the criteria of scientifically based research to CSR programs is challenging, the CSR program staff encouraged participants to understand that one intent of this new requirement is to raise the standard so that reforms are carefully scrutinized before being adopted. They also discussed a Department project called the "What Works" Clearinghouse, which will provide information on practices that meet the gold standard of scientifically based research. The CSR Program and NCCSR will also be developing tools specifically to help practitioners at low-performing schools understand and work with research. The National Clearinghouse for Comprehensive School Reform (NCCSR)is also working with the Regional Education Labs (RELs)and several SEAs to develop a workshop to guide practitioners through the process of understanding and using research to make decisions.
One participant suggested that the real issue is how to build school-level capacity to meet the SBR requirement. Schools first need to be able to analyze and determine the value of their current practices and programs in order to make decisions about dropping, adding, or integrating old and new practices within their comprehensive school reform program.
One participant noted that, because the goal is not to turn practitioners into researchers, but for practitioners to become more critical consumers, SEAs and RELs must find a way to make SBR less intimidating at the school level. One way they can help is by providing schools with relevant research resources. SEAs may also need to provide support for smaller districts that might otherwise opt out of the CSR application process if using scientifically based research is deemed too costly and time-intensive.
|State education agencies and other State or regional entities can lead school practitioners through a process, modeled during the symposium, that gives participants a guided, hands-on experience in reading and evaluating research.|
Another participant expressed a concern about the availability of "gold standard" research that is applicable to diverse student populations, specifically research that addresses strategies for the instruction of limited English proficient students. Participants were also concerned about how to talk to school practitioners about SBR in a way that will help them develop a thoughtful and constructive approach to using SBR in their teaching and learning practices.
Participants had several questions about using the Significance of Effects table, which provides a series of steps by which to determine the theoretical base, evidence of effects on achievement, and implementation and replicability of research. Research that primarily answers questions about implementation and replicability should demonstrate evidence that the schools in which the research was conducted are similar to the schools considering the research with regard to school demographics and other contextual factors. The CSR staff also reiterated that, as a rule of thumb, only recent research-between 3-5 years old-should be considered.
Participants also wanted to know whether case studies and other types of qualitative research must be excluded from consideration. The CSR staff explained that while there is a clear preference for random assignment, control-group studies, other types of research can sometimes be considered. For example, a case study is useful if its findings can be confirmed by on-site observations, or the study involves multiple sites compared with selected comparison schools.
During discussion of the SBR requirements, the CSR staff and symposium participants agreed that it would be helpful if model developers and other technical assistance providers shared their research base on results produced by their recommended methods and strategies that address the eleven components. This information could include student achievement data as well as research relating to the theoretical base of their recommended model or program.
The Catalog of School Reform Models, produced by the NCCSR and Northwest Regional Educational Laboratory (NWREL), provoked some discussion in one symposium and generated ideas for revising the Catalog, such as identifying which of the CSR components each model addresses. The Catalog is not an "approved list" but all models included are vetted through a new, more rigorous peer review process. Participants were encouraged to look at and use the rubric used for the peer review, which is available on the Catalog website at http://www.csrclearinghouse.org/.
One participant suggested that model developers also need opportunities, like that which these CSR regional symposia offered, to meet and discuss issues related to the new federal legislation. In one state, coordinators, with the help of a REL, recently convened a meeting specifically for model developers to ensure they understood the state context in which they would be operating and the standards to which they would be held. Participants recommended that the CSR program office and NCCSR, with the help of RELs and state CSR coordinators, facilitate symposia for model developers and other technical assistance providers to discuss the federal requirements affecting SEAs, LEAs, and schools, and their roles within this new legislative context.