|PDF (3 MB)|
Appendix A: Research Methodology
This guide is based on a descriptive study of eight charter schools selected for their exemplary achievement and for geographic and programmatic variety. While the schools are successful, the descriptive methodology does not support causal claims about which factors, or combinations of factors, led to their success. Nor does this guide constitute an endorsement of any specific commercial program or instructional practice. It does provide a portrait of what several successful schools look like and an analysis of common elements across schools. A brief description of this project's methodology follows.
An informal, nationwide recommendation process resulted in over 250 schools from 31 states being suggested for consideration. Nominations came from the advisory panel (see the acknowledgments section), state departments of education staff, charter school associations, authorizers, charter school administrators, and parents. Requests for nominations went out through key contacts to these networks, as well as through the U.S. Charter Schools Web site. Many schools nominated themselves.
Site Selection Criteria
The first and major criterion for site selection was exemplary achievement. Following the advice of the advisory panel, the emphasis was on improvement in achievement, rather than absolute achievement level, and on improvement trends across several years, so as to identify schools that were reliably becoming stronger and more effective over time. More specifically, the school had to have been established as a charter school no later than fall 1999, and it had to have achievement data for three consecutive years on the same measure, in order to show gains from one year to the next in two consecutive years. A final achievement criterion was that the school had met its Adequate Yearly Progress (AYP) target in the most recent year for which AYP had been announced to schools, as of December 19, 2003.
To check achievement scores, the research staff looked at published data on state Web sites, at the database of achievement scores compiled by the Brown Center on Education Policy at the Brookings Institution, and at information supplied by schools. For some states, it was impossible to find interpretable data. In other states, all schools were too new to have enough achievement data available. Unfortunately, very small schools were generally eliminated, because the scores for small sample sizes are not reliable and therefore are not reported.
Twenty-nine schools had acceptable achievement data and moved to the next phase of screening. Following the advice of the advisory panel, information was collected through public data and brief interviews about the grade levels served by the school, demographics of the population served, location, authorizer, and educational program. The goal was to find a diverse set of schools, encompassing both elementary and secondary levels, serving mostly low socioeconomic status students but some serving the general population, having a range of authorizers, representing different ethnic configurations, and having locations around the country. Final factors in screening, based on interview data, were stable leadership, evidence of parent involvement and parent satisfaction, and a positive relationship with the authorizer.
Study Framework and Data Collection
A conceptual framework to guide the study was developed from an analysis of research on charter schools and organizational effectiveness. Charter school experts, recruited to serve on the advisory panel, provided feedback to refine this framework and prioritize issues to investigate. The resulting study scope guided all aspects of the study (see figure 2 on page 4).
Collecting detailed descriptive information from project participants was key to understanding each school's vision and practices, the outcomes or impact achieved, and lessons learned that others could benefit from. Each school hosted a two-day site visit that included interviews with site leaders, teachers, board members, parents, and students as well as observations of classes and school events. In addition, artifacts from the sites, such as letters to parents, schedules, and training agendas, were collected to provide concrete examples of school practices. Site visitors reviewed the information from each site and developed a case report.
From the case reports, artifacts, and transcripts of interviews, the project team identified common elements that contributed to success across the sites. This analysis built on the research literature and study scope but also reflected patterns in the data and significant features that emerged in this study's cross-case analysis.
This descriptive research process suggests promising practices—ways to do things that others have found helpful, lessons they have learned about what not to do, and practical, “how-to” guidance. This is not the kind of experimental research that can yield valid causal claims about what works. Readers should judge for themselves the merits of these practices, based on their understanding of why they should work, how they fit the local context, and what happens when they actually try them. Also, readers should understand that these descriptions do not constitute an endorsement of specific practices or products.
Reports and Dissemination
Two products resulted from this research: a report of the findings and this practitioner's guide. The report provides the detailed description of each site, sample artifacts, an analysis of key findings across sites, and key project documents. The practitioner's guide is a summary of the report intended for broad distribution through conference presentations, as well as through national associations and networks. The guide and report are also accessible online at http://www.ed.gov/admins/comm/choice/charter/.
Ultimately, readers of this guide will need to select, adapt, and implement practices that meet their individual needs and contexts. Schools coming together in learning communities may continue the study, using the ideas and practices from these sites as a springboard for their own action research. In this way, a pool of promising practices will grow, and schools can support each other in implementation and learning.