Knowing which programs are effective and ineffective provides concrete guidance for school superintendents, principals, and teachers who need information to make sound decisions that will improve instruction and raise student achievement. The National Center for Education Evaluation (NCEE) designs evaluation studies to produce rigorous scientific evidence on the effectiveness of education programs and practices. In particular, the Center focuses on conducting rigorous impact studies of promising educational programs and practices that are supported through federal funds.

The evaluation studies that are supported by the Center use methodologies that can provide scientific evidence to answer questions of effectiveness. Evaluations funded by the Center answers one of three questions:

• What is the impact of the federal program on the intended outcomes? This question examines whether the federal funds have had a significant impact on key outcomes such as student achievement. Evaluation studies that address this question compare the outcomes of participants in school programs receiving federal funds versus those who are not in federally funded school programs.

• Is the program model effective? Several of the Department's programs support specific program models such as charter schools, family literacy, or after-school programs. For this question, the evaluation study addresses the impact of this program model in the diverse array of conditions and settings under which the model is implemented.

• Is a specific intervention (or class of interventions) effective? For some of the Department's largest programs in elementary and secondary education, state and local education agencies have the flexibility to use federal funds to support the educational intervention(s) that they believe will improve student achievement. In many cases, several states and districts have adopted a specific intervention (e.g., Success for All). For this question, the evaluation study assesses the effectiveness of a particular intervention (or class of interventions) that has been widely adopted or is viewed as promising.

In the fall of 2003, NCEE launched twelve studies using evaluation designs that will produce unbiased estimates of program impact. Six additional evaluation studies were awarded in the fall of 2004. The evaluations cover a wide range of topics and populations including: early childhood education, early reading, reading interventions for struggling readers at all grade levels, education technology in math and reading, after-school academic remediation, family literacy, charter schools, English language acquisition, technology, professional development, teacher education, teacher induction, adult literacy, school choice, and school-based violence prevention. Listed below are descriptions of new NCEE studies and NCEE studies that are in progress -

New NCEE Studies

The following NCEE evaluation studies were awarded in September 2004 -

Evaluation of Reading Comprehension Programs

This study will assess the effectiveness of reading comprehension programs with direct instruction of comprehension skills within social studies and sciences content areas.

• Four reading comprehension programs will be selected by spring of 2005 for piloting in the 2005-06 school year.
• Elementary schools will be randomly assigned to use the various reading programs or to a control group during the 2006-07 school year.
• Contractors for this four-year evaluation are Mathematica Policy Research, RG Research Group, and RMC Research.

Teacher Induction Impact Evaluation

This study will examine the effectiveness of a teacher induction program in increasing retention rates and affecting teacher practices among novice elementary school teachers.

• An induction program will be selected through an open competition held in the fall of 2004 and the selected program will be implemented in 20 high poverty districts during the 2005-2006 school year.
• Elementary schools within the districts will be randomly assigned to the selected induction program or the induction program normally offered by the district.
• Longitudinal data on teacher practices and teacher retention will be collected annually through the fall of 2008.
• Contractors for this five-year study are Mathematica Policy Research, the Center for Educational Leadership, and Learning Point Associates.

Adult ESL Literacy Impact Evaluation

This study will examine the effectiveness of an enhanced ESL curriculum (one that includes explicit instruction in the basic components of literacy) in improving the English reading, writing, and speaking skills of adult ESL learners who have low levels of literacy in their native language.

• Approximately 1,800 adult ESL learners from ten adult education programs will be randomly assigned to a classroom in which the enhanced ESL curriculum is delivered or to a classroom in which the typical instruction is delivered.
• Data collection will include learner background interviews, classroom observations, attendance logs, and immediate post-treatment assessments of learners' English reading, writing, and speaking abilities.
• The evaluation also includes the option to assess learners' English reading, writing, and speaking abilities twelve months post-treatment.
• Contractors for this five-year study are the American Institutes for Research, the Lewin Group, Mathematica Policy Research, and Berkeley Policy Associates.

An Evaluation of the Impact of Supplemental Literacy Interventions in Freshman Academies

This study will assess the impact of two supplemental literacy programs for striving readers in ninth grade smaller learning communities. The evaluation contractor and an expert panel will select the two literacy programs in the fall of 2004.

• Approximately 32 schools in eight to twelve districts will be randomly assigned to implement one of the two programs in the 2005-2006 and 2006-2007 school years.
• Two cohorts of eligible ninth grade students (those reading two to four grade levels below) in each school will be randomly assigned to participate in the supplemental literacy program or to participate in their regular study hall or elective.
• Contractors (Manpower Demonstration Research Corporation and the American Institutes for Research) for this five-year evaluation will collect and analyze data on students' literacy skills, scores on state assessments, course taking, and retention.

DC Choice

The Evaluation of the Impact of the DC Choice Program (a federally-funded scholarship program for low-income residents of Washington, DC) will be based largely on a comparison of outcomes of approximately 2,000 student applicants randomly assigned by a lottery to either receive a scholarship or not receive a scholarship.

• The lottery for fall 2004 student enrollment took place in spring 2004. The next lottery for fall 2005 student enrollment will take place in spring 2005, and may be conducted each year if slots at private schools become available.
• Data will be collected for four years, for students in both the scholarship and non-scholarship groups.
• Contractors (Westat and Georgetown University) will administer academic assessments, collect school records, and conduct student, parent, and principal surveys each spring.

Violence Prevention Programs

This study will evaluate the effectiveness of a school-based violence prevention program for middle schools. Violence is a particular concern for middle schools because incidents of school-related violence are more likely to occur in these grades, however there is some question about the effectiveness of the violence-prevention programs available for adoption in middle schools.

• A competition was announced in October 2004 for developers of school-based violence prevention programs to submit proposals to implement a program in middle schools.
• The violence prevention program will be selected in December 2004 and will be evaluated in selected middle school beginning in the 2005-2006 school year and for two subsequent school years.
• In order to determine the impacts of the selected violence prevention program, disruptive, aggressive, and violent incidents in 20 schools randomly assigned to participate in the program will be compared to 20 other schools randomly assigned to not participate.
• Contractors for this five-year study are RTI International, Pacific Institute for Research and Evaluation, and Tanglewood Research, Inc.

NCEE Studies in Progress

Early Reading First National Evaluation

Early Reading First (ERF) National Evaluation, a Congressionally mandated study, will assess the impact of ERF grants on children's language and literacy outcomes and on preschools' language and literacy instruction and practice. The study will use a quasi-experimental design (regression discontinuity) that will provide unbiased estimates of program impact.

• The treatment group will consist of children attending preschools in 28 ERF grantee sites; the comparison group will be children attending preschools in 40 sites that applied for, but did not receive, ERF funding.
• The study will test whether the funding and technical assistance received by ERF preschools improves children's language and literacy outcomes and classroom language and literacy instruction and practice relative to other preschool children and classrooms.
• Data collection will occur in 2004-2005 and will include child assessments, classroom observations, student records, and surveys of preschool directors, preschool teachers, and parents.
• The sample of 2,211 four-year old children will be assessed at the fall and spring of their pre-kindergarten year.
• Contractors for this three-year study (July 2003 - September 2006) study are Decision Information Resources, Mathematica Policy Research, and University of Texas-Houston Health Sciences Center.

Reading First Impact Study

The Reading First Impact Study is a mandated evaluation that will evaluate the impact of Reading First on student reading achievement. The study will use a quasi-experimental design (regression discontinuity) that will provide unbiased estimates of program impact.

• In districts that used a quantifiable rating/ranking process to choose schools for Reading First funding, the study will compare reading outcomes in schools that received Reading First funding with reading outcomes in comparison schools that scored below the cutoff point for Reading First funds.
• The evaluation will collect data over a three-year period (2004-2005 through 2006-2007)
• Data collection will include direct assessments of student reading achievement and information on classroom reading instruction.
• Contractors for this five-year study (October 2003 - September 2008) are Abt Associates, Manpower Demonstration Research Corporation, RMC Research, Westat, and Rosenblum-Brigham Associates.

Development, Implementation, and Impact Evaluation of Academic Instruction for After-School Programs

The Department held an open competition and had experts in reading and math select two developers, Success for All in reading and Harcourt Publishers in math, to adapt their research-based school day curricula for use in the after-school setting. These new intensive after-school academic curricula are being pilot-tested in eight after-school centers during the 2004- 2005 school year.

• The full evaluation will begin in 2005-2006 and consist of two parallel studies (one testing the math curriculum and one testing the reading curriculum) with 25 centers in each study.
• After-school program participants at each center will be randomly assigned to either the intensive academic curriculum or to the academic activities (usually homework help) that the center usually provides.
• Each of the two studies will include approximately 1,000 students who receive the treatment and 1,000 who receive homework help only.
• The contractors for this four-year study (October 2003 - August 2007) are Manpower Demonstration Research Corporation; Public/Private Ventures; Bloom Associates and Survey Research Management.

Remedial Reading Programs -Closing the Gap

Closing the Gap project is an evaluation of intensive remedial reading programs for 3rd and 5th graders who have not yet acquired the reading skills necessary to succeed in school.

• The reading programs selected for the evaluation each offer 100 hours of pullout instruction in small groups.
• This three-year study will assess the impact of these interventions on students during the 2003-2004 school year, with follow-up data collection on student achievement planned for spring 2005.
• The contractors for this study are Florida State University and Mathematica Policy Research, Inc.

Impact Evaluation of Teacher Preparation Models

The Impact Evaluation of Teacher Preparation Models is designed to assess the effects of different types and amounts of teacher training on student achievement, taking advantage of the existing variation in teacher training across different routes to certification-both alternative and traditional.

• A pair of new teachers in the same grade will be formed at each of 80 schools included in the study, with one teacher having taken the traditional route and one the alternative route to certification.
• Pairs that include teachers in low coursework alternative programs and teachers in high course work traditional certification programs provide information about the impact of differing amounts of teacher training; pairs that include teachers in high coursework alternative programs and teachers in high coursework traditional certification programs provide information about the impact of training prior to hiring (traditional routes) versus training on-the-job (alternative routes).
• Students will be randomly assigned to teachers in the pair, and student test scores will be compared.
• The study will also statistically control for the types of teachers who choose to participate in each route and whom the selected school hires to assess the characteristics of teacher preparation that may matter most in improving test scores.
• Data collection began in the fall of 2004 in the first cohort of schools. Additional schools to complete the sample will be added in the fall of 2005.
• Data collection includes student records, pre- and post-test scores, and measures of teacher practice during a one-year follow-up.
• The contractor for this four-year study (October 2003 - September 2007) is Mathematica Policy Research, Inc.

The Impact of Professional Development Models and Strategies on Teacher Practice and Student Achievement

The Impact of Professional Development Models and Strategies on Teacher Practice and Student Achievement is designed to evaluate the effects of two professional development approaches on improving early reading. The approaches are: (1) a 5-day summer institute with 3 days of follow-up through the school year, and (2) institutes and follow-up days plus coaching by an in-school reading specialist trained in a particular coaching approach.

• The professional development models will be piloted in a small set of schools during 2004-2005 while districts will be recruited for the full evaluation.
• Random assignment of schools to receive the different professional development models will take place in spring 2005.
• Full implementation and data collection will begin in Summer 2005, including measures of professional development implementation quality, teacher knowledge and practice, and student achievement.
• The contractors for this five-year study (September 2003 - September 2008) are the American Institutes for Research and Manpower Demonstration Research Corporation.

Evaluation of the Effectiveness of Educational Technology Intervention

This mandated evaluation assesses the impact of using selected technologies that are intended to improve student academic achievement in reading and/or in mathematics. Seventeen educational technology products were selected competitively in the fall of 2003 for implementation in the 2004-05 school year.

• The question of the impact of educational technology on reading or mathematics achievement is being addressed through an experimental methodology in which teachers were randomly assigned to an educational technology or to a control group.
• The study will also examine the association between school and classroom conditions and teacher practice and student achievement.
• The contractors for this evaluation are Mathematica Policy Research and SRI International. The three-year study began September 30, 2003, and will conclude in 2006.

Optimizing Educational Outcomes for English Language Learners

Optimizing Educational Outcomes for English Language Learners is an evaluation of enhanced versions of structured English immersion and transitional bilingual education programs that differ in the amount of Spanish language used for instruction, compared to typical, existing programs.

• Sixteen schools have been selected to participate.
• Teachers and Kindergarten students within the schools will be randomly assigned either to the enhanced program or to the same type of program that is currently in place.
• Teachers will be randomly assigned to conditions while students will be randomly assigned to teachers.
• In grades K to 3, the study will compare enhanced versions of either model to the ELL program already in use.
• Main study outcomes are the acquisition of English skills and content knowledge.
• The grantee for this five-year study (October 2003 - June 2008) is the University of Houston in collaboration with the Center for Applied Linguistics and the University of Texas, Austin.

Effects of Transitional Bilingual Education, Two-Way Bilingual, and Structured English Immersion Programs and the Literacy and Oracy of Spanish-Dominant Children

This evaluation will test enhanced versions of the Success for All structured English immersion, transitional bilingual and two-way bilingual programs in 15 high-poverty schools.

• Students will be randomly assigned to alternative models.
• One set of 10 schools will compare the structured English immersion model to the transitional bilingual education model.
• A second set of 5 schools will compare the structured English immersion model to the two-way bilingual model in which students learn literacy and oracy in English and Spanish.
• The study will provide estimates of the impact of the three major alternative approaches to the education of English language learners, as well as evidence on the conditions under which each approach is maximally effective.
• The grantee for this five-year study (October 2003 - June 2008) is Johns Hopkins University.

Project English Language Literacy Acquisition

Project English Language Literacy Acquisition is an evaluation of enhanced versions of structured English immersion and transitional bilingual education programs compared to existing programs that are already in place. The enhanced versions include the use of additional strategies in intensive English, professional development and parent training.

• Study participants include 1152 native Spanish-speaking students in 48 kindergarten classrooms (24 structured English immersion and 24 transitional bilingual) among 20 elementary schools.
• Students begin the study in kindergarten and will be followed through the third grade.
• The grantee for this five-year study (October 2003 - June 2008) is Texas A&M Research Foundation, in collaboration with Sam Houston State University and Southern Methodist University.

Evaluation of the Impact of Charter School Strategies

The purpose of this study is to evaluate achievement gains of middle school entrants in charter schools. The study will work with the selected charter schools to implement lotteries in order to randomly assign applicants to either receive (treatment group) or not receive (control group) admission to the schools. Enrollment lotteries are required in most state charter school laws and federal grantee regulations require charter schools with waiting lists to select students for enrollment by lottery.

• About 50 middle schools will be purposively selected to meet the study's data needs and to reflect variation in the policy environment.
• Recruitment of schools will take place in fall 2004 and early 2005.
• Random assignment and data collection will begin in 2005, continuing for at least two years, and will include student records, annual test scores, and surveys of students, principals, and parents.
• The contractor for this five-year study (October 2003 - October 2008) is Mathematica Policy Research, Inc.

Even Start Classroom Literacy Interventions and Outcomes Study

The Even Start Classroom Literacy Interventions and Outcomes Study is testing four intervention models within Even Start family literacy projects. The study design includes random assignment of 120 Even Start projects to receive one of the four intervention models or to participate in an "as is" control group.

• All projects assigned to an intervention group will receive one of two early childhood education curricula that provide enhanced instruction in early literacy.
• Two of the intervention groups will also receive enhanced parenting education and parent-child activities curricula that are integrally linked with the early childhood education instruction.
• The other two intervention groups will continue to provide parenting education and parent-child activities as they are already being provided.
• The control group of projects will not receive any of the enhanced curricula, but will continue to provide their existing services.
• The study will assess children's early literacy skills and parent's literacy and parenting behavior.
• The study will assess children's early literacy skills and parent's literacy and parenting behavior.
• Contractors for this five-year study (September 2001 - September 2006) are Westat and Abt Associates.

The National Evaluation of the 21st Century Community Learning Centers Program

Federal legislation to support after-school programs, the 21st Century Community Learning Centers Program, was originally funded in 1998 in response to the need for children to have access to safe and supervised after-school activities that can help develop academic, personal, and social skills.

• In 1999 the Department awarded a contract to Mathematica Policy Research, Inc. to conduct an independent evaluation of this program.
• The evaluation includes an elementary-school study and a middle-school study.
• The elementary-school study was based on a sample of 14 typical, although not statistically representative, program grantees.
• The evaluators randomly assigned approximately 2600 students either to participate or not in the program (1400 treatment and 1200 controls).
• The middle-school study used a matched comparison group design that had a representative sample of 34 grantees. There were 4500 students (1900 treatment and 2600 comparison) in the middle school baseline sample.
• The first report based on one year of follow-up data from this ongoing evaluation was released in February 2003. The second report was released in October 2004 and the final report is expected by March 2005.