Chapter 1: Overview of the Evaluation and The Year One Report
The Public Charter Schools Program (PCSP), established in 1994, represents the federal government's commitment to help charter schools meet planning, start-up, and early implementation costs. By helping charter schools overcome financial barriers, the PCSP is also designed to increase the number of charter schools nationwide. As of October 1999, 1,692 charter schools were in operation in 30 states, the District of Columbia, and Puerto Rico.1
Researchers at SRI International are conducting the first national evaluation of the PCSP. In addition to gathering systematic information about the program at the local, state, and federal levels, the 52-month study will continue to document the evolution of the charter school movement (a process begun under another study funded by the U.S. Department of Education and conducted by RPP International). Data collection and analysis for the SRI study are driven by a series of research questions on the operations and impacts of the PCSP, the characteristics of charter schools, charter school flexibility, and charter school accountability.
The Public Charter Schools Program
Federal interest in supporting the development of the charter school movement began in 1993, when President Clinton first proposed the Public Charter Schools Program (PCSP) and several Senators and Representatives proposed the Public Schools Redefinition Act. No action was taken, however, until the reauthorization of the Elementary and Secondary Education Act (ESEA) in 1994. At that time, the PCSP was enacted as Title X, Part C, of ESEA, with an initial appropriation of $6 million in FY 1995. The PCSP is a discretionary grant program, administered in the Office of School Improvement Programs in the Office of Elementary and Secondary Education. The PCSP is intended to support the planning, development, and/or initial implementation of charter schools, providing relatively unencumbered seed funding for states with charter school laws to distribute to charter school groups during the first 3 years of a charter school?s existence. The statute also makes provision for individual charter schools to apply directly to the Secretary of Education for a grant if their states choose not to participate or have been denied a grant.
Congress reauthorized the PCSP in 1998 by passing the Charter School Expansion Act of 1998 (P.L. 105-378). Eligibility for subgrants was expanded to include mature charter schools, which could apply for funds to disseminate promising school practices. The appropriation for FY 2000 was $145 million.
According to the authorizing legislation, the purpose of the Public Charter Schools Program is to "increase national understanding of the charter schools model by (1) providing financial assistance for the planning, program design and initial implementation of charter schools; (2) evaluating the effects of such schools, including the effects on students, student achievement, staff, and parents; and (3) expanding the number of high-quality charter schools available to students across the Nation" (Sec. 10301[b]).
These purposes are addressed by providing grants to states. States determine the process for putting 95 percent or more of the state grants into the hands of charter school planning groups and charter schools in the form of subgrants. These subgrants can be used for planning, implementation, and?for schools that have been in operation for 3 years or more?dissemination of promising charter school practices and programs. If a state has not applied for or received a PCSP grant, charter school planners may apply directly to the U.S. Department of Education (ED) for funds.
In addition, the Secretary of Education is authorized "[t]o provide for... evaluations or studies that include the evaluation of the impact of charter schools on student achievement" and "[t]o provide . . . for the collection of information regarding the financial resources available to charter schools. . . " (Sec. 10305[a]). The research reported in this document was supported by PCSP funds set aside for research and other national activities.
The Charter School Movement
Even though charter schools in the United States predate the PCSP, the movement is still less than 10 years old. Since the first charter school opened in 1992 in Minnesota, the number of these schools has increased steadily, and President Clinton has repeatedly challenged educators to increase the number to 3,000 by 2002. As the movement has grown, it has also struggled. Stories began to emerge early about the difficulties that charter schools?particularly those that were being created from scratch?faced in their first months and years (RPP International and the University of Minnesota, 1997). Depending on the specifications about state and local funding streams in a state?s charter school legislation, groups seeking to open charter schools were often obliged to capitalize the planning and early development of their schools out of their own pockets or by incurring debt. Finding, renting or buying, and renovating space were particular barriers, according to early surveys of the charter school field.
Exhibit 1-1 illustrates the growth of the charter school movement. The National Study of Charter Schools (Nelson et al., 2000) reports several interesting statistics about this growth, which the SRI data will supplement in this and later years. For an overview of charter school activity by state, see Appendix A.
SRI International?s Evaluation of the PCSP.
The PCSP legislation authorized an evaluation of the program, along with other national activities. Researchers at SRI International began this first evaluation of the PCSP in October 1998. The 52-month study has two purposes: (1) evaluation of the rapidly growing Public Charter Schools Program and (2) continued documentation of the evolution of the charter school movement begun under the National Study of Charter Schools, conducted by RPP International.2 The findings reported in this document are from the first year of the SRI evaluation, which included telephone surveys of (1) state charter school coordinators in 36 states, Washington, D.C., and Puerto Rico (N=38); (2) representatives of agencies that are authorized to award charters to charter schools under state charter school laws (charter school authorizers) (n=48); and (3) directors of charter schools that received PCSP grants directly from the U.S. Department of Education (ED) (n=14). Year 1 data collection also included the extraction of information from federal files maintained on PCSP grantees, analysis of state charter laws, interviews with federal staff, and focus groups with charter school planners, operators, and charter school authorizers.
Research Questions and Data Sources
The dual purposes of this evaluation are evident in the broad research question clusters that guide the data collection, analysis, and reporting requirements. Some questions are concerned specifically with the Public Charter Schools Program, and others are about the evolution of the charter school movement in a more general sense. Exhibit 1-2 displays the research questions under each cluster and summarizes the data collection methods that are being used to address them.
During the 4-year evaluation, answers to the research questions will be obtained through several sources of quantitative and qualitative data. The sources will differ for each of three planned reports. The data sources that were used this year are described below. The sources that will be used in later years are revisited in the last chapter of this report. Data sources for this report consisted of the following:
Telephone survey of state charter school coordinators. The charter coordinators (or equivalent) for all 38 states with charter school legislation were surveyed by phone in the summer and fall of 1999. The survey instrument incorporated a combination of open- and closed-ended items on charter school-related operations, requirements, and flexibility, and the state?s PCSP grant. Members of the SRI study team administered the surveys, which took approximately one hour per survey respondent. The response rate was 100 percent. (The basic characteristics of the state universe are displayed in Appendix A.)
Telephone survey of charter school authorizers. A sample of 50 authorizers that have awarded charters to charter schools was selected during the spring and summer of 1999. Among the states with charter school laws, many types of entities are authorized to issue charters. Because of uncertainties about the respondent pool for this part of the charter school story, the evaluation team and the study?s Technical Work Group favored a purposive sampling strategy for the first year of data collection to maximize the diversity of authorizers surveyed. Data collection took place at the same time as the state coordinator survey and involved the same researchers. The charter school authorizer survey also took about one hour to complete. Survey questions were designed to document charter school authorizers? experiences in assisting and monitoring charter schools; granting, renewing, and revoking charters; and financial issues (including the federal PCSP funds). Forty-eight charter school authorizer surveys were completed within the data collection window, for a response rate of 96 percent.
RESEARCH QUESTIONS AND DATA SOURCES (CONCLUDED)
* = Site visits include the written parent survey as a data source.
Note: Italicized research questions were added by the SRI evaluation team
In Exhibit 1-3, the charter school authorizer sample is displayed according to the type of agency. To facilitate the analysis of broader types of charter school authorizer, the local, state, institution of higher education, and other categories were created by collapsing similar types of agency.
There are limits to the conclusions that can be drawn from the purposive and therefore nongeneralizable sample of charter school authorizers. However, the sample supplies interesting information regarding the basic features of and variation across charter school authorizers. As is true across the country, most of the charter school authorizers in the Year 1 sample were local school districts (n=34). The other authorizers included in the sample were either state-level bodies (state boards of education, state educational agencies), institutions of higher education (IHEs), or independent chartering boards (n=14). The type of charter school authorizer appears to be closely linked to the number of schools chartered¾local agencies in our sample chartered fewer schools than other types of charter school authorizers (grouped as "nonlocal" agencies in most of the analyses for this report). Exhibit 1-4 displays the variation in average number of schools chartered by local and nonlocal authorizers and other descriptive statistics about them.
* Because of the small number of "state," "higher education," and "other" authorizers included in the sample, the team collapsed these data into one category "nonlocal" and the data by type of authorizer are presented by "local" and "nonlocal" categories for most of the remainder of this report.
It is important to note some details that do not show up in Exhibit 1-4. First, approximately one-third of local charter school authorizers chartered only one school (none of the other types of agencies had chartered only one school). Second, only 9 percent of local charter school authorizers chartered 16 or more schools, compared with 64 percent of nonlocal authorizers. The Year 1 surveys did not explore why the number of schools chartered is so closely linked to type of authorizer, but from past charter school work the team has learned that local charter school authorizers tend to charter in their local attendance area, whereas nonlocal agencies charter schools across the state or region.
Telephone survey of charter schools receiving PCSP grants directly from ED. In states that have not applied for or do not receive PCSP grants, charter schools are eligible to apply for start-up and implementation grants directly from the federal government. A total of 23 charter schools either currently receive these grants or have in the past. A survey of PCSP-related questions was administered to representatives from these schools by a member of the study team. Fourteen charter schools responded, for a response rate of 61 percent.3
Extraction of information from federal files on PCSP grantees. With the cooperation and assistance of PCSP staff at ED, members of the study team retrieved information for the states, charter schools, and school districts that received PCSP grants before 1999. The federal files for 25 states, 21 charter schools, and 2 school districts were reviewed before Year 1 data collection occurred. The federal files provided background information on grantees that team members used for "precoding" several items prior to telephone surveys. In addition to federal files, team members also gathered information on states from various other (principally Web-based) sources, including state laws, regulations and guidelines, and charter school profiles. By supplying information on the survey from these other sources, the precoding step saved time for the survey respondents. As a result, especially in the case of PCSP-related items, respondents often simply confirmed or corrected the information gleaned from the federal files.
Interviews with PCSP staff and other government respondents. In an effort to place the PCSP in the context of other school reform and public school choice issues in the federal government, members of the study team conducted eight semistructured individual and group interviews with 12 Department of Education employees. Interviews were conducted with members of the PCSP program staff, staff of the Planning and Evaluation Service at ED, and those individuals from other ED offices involved in the Departmentwide school choice team. Although interviews were informally tailored to interviewees, team members sought to elicit general information about the nature and involvement of the respondents in the choice team, ED?s monitoring of the PCSP law, and the nature and types of technical assistance provided to the field. More specific information on the information gathered from these interviews is incorporated anecdotally in Chapter 2.
Focus groups with charter school operators, charter school planners, and charter school authorizers. Through an addition to the original evaluation design and in consultation with the U.S. Department of Education, members of the SRI study team designed and conducted a series of 18 charter school-related focus groups in January and February 2000. The focus groups were conducted to help the Department determine the format and content of future technical assistance activities. The groups consisted of individuals planning charter schools (six focus groups), persons who were already operating charter schools (nine focus groups), and representatives of authorizers that had awarded charters (three focus groups). Logistical support (scheduling, recruiting participants) was provided by the Charter Friends National Network and their local contacts. Exhibit 1-5 indicates where the focus groups of each type were conducted.
Data Collection and Analysis
Telephone surveys were administered in the summer and fall of 1999 by members of the SRI evaluation team. The surveys of PCSP grantees were relatively straightforward since the team had names, contact information, and job titles for state charter school coordinators and the leaders of charter schools that received PCSP funding directly from the federal government. Even where staff turnover had occurred, it was usually easy to identify the new respondent, schedule an appointment, and administer the survey. In a few cases, a second respondent was contacted to obtain information that was not provided by the first (typically because he or she was new to the job).
Identifying the appropriate respondents for the charter school authorizer survey was a larger and more time-consuming challenge. Team members usually had to call more than one office in a charter school authorizer to determine the right person to take the survey. The variety of types of charter school authorizer contributed to this situation. Even within local educational agencies (LEAs), the largest type of charter school authorizer in our sample and among the universe of agencies that have awarded charters, different kinds of personnel responded, ranging from the district superintendent to the district?s general counsel. Other respondents included the vice president of a state board of education and a district administrator who served part-time as the principal of a charter school.
Once data were collected and tabulated, a variety of data sorts and cross-tabulations were performed. Formal statistical estimates of the universe were not conducted on the Year 1 data, either because the entire universe was sampled (in the case of state coordinators and charter schools receiving PCSP funds directly from ED) or because the sample was not generalizable to the relevant universe (in the case of charter school authorizers). Hence, most of the information reported in the following sections is based on simple, descriptive statistics.
Organization of the Report
The Year 1 data in this report paint a comprehensive picture of a number of issues: the development of the Public Charter Schools Program, state and charter school authorizer perspectives on charter school flexibility and accountability, and the charter school activities of states and a sample of charter school authorizers. This picture, however, is also a "snapshot" of a rapidly evolving movement during a narrow time interval (summer and fall 1999).
Readers who would like more detail about future data collection activities for this evaluation are referred to Appendix B.
The report is organized according to the evaluation themes and findings from the first year of data collection and analysis. The broad themes are identified in Exhibit 1-6.
As this list of themes indicates, the key issues explored in the Year 1 report are the Public Charter Schools Program, charter school flexibility, and charter school accountability, mostly from the perspective of respondents to the state coordinator and charter school authorizer surveys. Information from federal interviews and the focus groups is incorporated anecdotally.
The report concludes with a chapter that summarizes the study findings as of Year 1 and provides information on the evaluation team?s plans for future rounds of data collection, analysis, and reporting.
Two additional reports are planned under this contract. They will include charter school-level data from multiple sources: (1) the National Study of Charter Schools conducted by RPP International; (2) the National Center for Education Statistics? Schools and Staffing Survey (SASS), which included a charter school supplement in its 1999-2000 data collection effort; (3) the SRI survey of charter schools in 2000-2001 and 2001-2002, and (4) site visits to charter schools in 2000-2001 and 2001-2002. In addition, the SRI evaluation team will conduct a substudy of student performance in charter schools compared with student performance in regular public schools in a limited number of states where such comparisons can be made under rigorous design considerations.
1 Although 36 states, the District of Columbia and Puerto Rico have enacted charter school laws, charter schools are not yet in operation in all of these states. In the remainder of this report, the District of Columbia and Puerto Rico are usually grouped with the other states.
2The National Study of Charter Schools tracked the development of charter schools nationwide from 1995 to 1999. This study produced four annual reports and a number of topical reports that are or will be available on-line. The fourth-year report is available at http://ed.gov/PDFDocs/4yrrpt.pdf.
3 Although the response rate is low, the respondents did not appear to have any characteristics that distinguish them from the nonrespondents (e.g., state, status of PCSP grant, etc.).