|PDF (1 MB)|
Appendix A: Research Methodology
The research approach used to develop this guide is a combination of case study methodology and benchmarking of "best practices." Used in businesses worldwide as they seek to continuously improve their operations, benchmarking has more recently been applied to education. Benchmarking is a structured, efficient process that targets key operations and identifies the following: promising practices in relationship to traditional practice, previous practice at the selected sites (lessons learned), and local outcome data. The methodology used here is further explained in a background document,19 which lays out the justification for identifying promising practices based on four sources of rigor in the approach:
Theory and research base;
Site evidence of effectiveness; and
Systematic field research and cross-site analysis.
The steps of the research process were: defining a study scope, seeking input from experts to refine the scope and inform site selection criteria, screening potential sites, selecting sites to study, conducting site visits, collecting and analyzing data to write case reports, and writing a user-friendly guide.
Study Framework and Data Collection
A conceptual framework was developed to guide the study of the selected sites. While there is a lot of daily activity at any school site, each case study needed to focus on those practices most likely to contribute to a school's success and sustainability. The framework for this study was an adaptation of that used in a previous guide on districtwide magnet schools in an earlier Innovations in Education series, incorporating additional research literature on magnet schools, organizational effectiveness, and sustainability of school reform. Whereas a previous guide, Creating Successful Magnet School Programs, focused on magnet program implementation from a district perspective, the research for this guide focused on the schools themselves. The dimensions of the conceptual framework for this guide were academic excellence through an innovative theme, cohesive and effective school operations, strategic partnerships, and adaptability to challenges. A site visit was conducted at each school to gather the information for this guide, each visit lasting for two days and including informal observations throughout the school, attendance at events, and interviews. The primary source of data was interviews with a variety of key groups, including parents, teachers, administrators, district leaders, and school partners. An interview protocol was developed based on the study framework and adapted to each role group. Key interviews were digitally recorded and later transcribed.
Documents from each school served as an additional source of information. Collected during the site visit, these documents included such items as school schedules, sample assessments, curriculum plans, newsletters, application forms, brochures, and report cards. Principals and district magnet office staff also completed a standard form designed for this project to facilitate consistent compiling of school demographic and outcome information.
Site Selection Process and Criteria
A cross-section of schools were selected to highlight K-8 magnet schools successfully meeting the desegregation and achievement needs of their particular districts. Considerations included districts with a high number of traditionally underserved populations (e.g., low income, special education, African-American and Hispanic students), variation in academic programming and magnet themes, a range of grade configurations (e.g., K-4, K-8, 4-8), and a range of geographic locations.
For this guide, a school had to be a dedicated magnet, not a school-within-a-school program, and could not use selective admission criteria (e.g., minimum grade point average, test scores, or audition) to enroll students. Each school had to show evidence of strong academic achievement as well as success in reducing minority group isolation for at least four consecutive years.
Based on state standardized test data, strong academic achievement meant that students of certain subgroups—including African-American and Hispanic students, those receiving special education services, English language learners, and students eligible for free or reduced-price lunch (an indicator of relatively low family income)—were outperforming local district public schools that served a similar population of students in math and reading. Schools considered for this guide also had to have met adequate yearly progress (AYP) for the most recent year for which data was available, with priority given to schools that met AYP for the last four consecutive years.
For interdistrict schools, which operate in extremely minority-group-isolated settings, success with desegregation meant that schools did not have any subgroup representing more than 80 percent of the student population. Within a given district, researchers defined for each subgroup a percentage that would be considered racial isolation in the context of the district based on demographics, and success with desegregation meant that a school had avoided isolation for any subgroup.
Because sustainability was the focus of the guide, one first step in selecting potential magnet school sites was to identify host districts with well-established magnet programs. A review of 198 school districts that had received federal Magnet Schools Assistance program (MSAP) funding at some point from 1985 to present, taking into account MSAP's changing criteria in that time period, narrowed the selection to 52 districts after eliminating those with emerging programs (less than four years old), declining programs (reduction in numbers of magnets, total magnet enrollment), underperforming programs (with regards to achievement or student enrollment data), or those districts that were undergoing reorganization. In each of the remaining 52 school districts, the SchoolMatters.com Web site was used as a standard reference to compare magnet schools within a district for achievement and success with desegregation.
Additionally, individual schools also were nominated by the magnet school researchers, practitioners, and board members of the Magnet Schools of America (MSA), who served as the guide's external advisory group. As a national organization, MSA identifies successful magnet schools through a rigorous application process that includes achievement, desegregation, and sustainability criteria; the award lists from the most recent five years were used to determine candidates.
Based on recommendations from the advisory group and a screening of schools based on achievement and student enrollment data (see below), the initial list was narrowed down to 22 schools. Additional information about specific program features and updated data were collected using phone interviews with school administrators to fill gaps in information.
From this group of 22 schools, six schools were ultimately chosen as case study sites, based on the compiled information and criteria ratings on a screening matrix. Demographic variation, a range of promising practices, geographic location, and achievement data were all considered in the final site selection in order to present a range of contexts for those considering magnet programs.
Schools selected met AYP targets for at least two consecutive years, including the most recent year for which data were available for each school. Researchers looked for schools that scored at least at the 50th percentile in math or reading on state standardized tests with demonstrated evidence of continued improvement for at least three years, or for schools that were consistently high achieving in the 90th percentile range annually. Data from Web sites of state departments of education and the Web site SchoolMatters.com provided achievement information. Researchers compared subgroup scores at the selected schools with a similar population of students in its local district public schools.
Student Enrollment Criteria
The profiled schools demonstrated success in maintaining stable patterns of enrollment over time. Compared to district demographics or interdistrict targets, these schools are maintaining diverse student populations with respect to race, ethnicity, and socioeconomic background.
Schools needed to demonstrate success with achievement and the ability to attract substantial numbers of students of different racial backgrounds for at least five years. Success was sustained over time at each of the selected schools regardless of MSAP status (two of the six schools were recipients of MSAP grant funding) or leadership transitions. In addition, these schools all serve as models of exemplary magnet programs at the district, state, or national level.
Analysis and Reporting
A case report was written about each site, and reviewed by site administrators for accuracy. From these case reports, artifacts, and transcripts of interviews, the project team identified common themes that contributed to success across the sites. This cross-site analysis consisted of using both the research literature as reflected in the study scope as well as emerging patterns in the data.
This descriptive research process suggests promising practices, including ways to do things that other educators have found helpful and lessons they have learned, as well as practical "how-to" guidance. This is not the kind of experimental research that can yield valid causal claims about what works. Readers should judge for themselves the merits of these practices, based on their understanding of why they should work, how they fit the local context, and what happens when they actually try them. Also, readers should understand that these descriptions do not constitute an endorsement of specific practices or products.
Using the Guide
Ultimately, readers of this guide will need to select, adapt, and implement practices that meet their individual needs and contexts. Schools and districts coming together in learning communities may continue to study, using the ideas and practices from these sites as a springboard for their own action research. In this way, a pool of promising practices will grow, and educators can support each other in implementation and learning.