Data & Research EVALUATION OF PROGRAMS
National Evaluation of Title III Implementation Supplemental Report: Exploring Approaches to Setting English Language Proficiency Performance Criteria and Monitoring English Learner Progress
Report Summary
Downloadable File MS Word (54 KB)

More Resources
Complete Report
download files PDF (3.1 M)

Background

  • Purpose: This report offers several empirical methods that state policy-making authorities can use as one part of a deliberative, judgmental process to set English Language Proficiency (ELP) performance standards and operationalize ELP assessment and accountability criteria.

  • Central questions of report: This report addresses three key questions:

    1. What analytical methods can be used to determine a meaningful and empirically based ELP performance standard?
    2. What analytical methods can be used to establish a realistic, empirically anchored time frame for attaining a given ELP performance standard?
    3. How can an English learner's ELP level be taken into account when setting academic progress and proficiency expectations?

  • Design: This report explains analytical methods that can be applied to longitudinal student-level achievement data in order to identify empirically based ELP and academic achievement goals for English learners (ELs). In so doing, the authors describe the methods and then use longitudinal student-level achievement data from several states in order to illustrate each of the methods with "worked examples."

Results

Key findings organized by key questions:

  1. What analytical methods can be used to determine a meaningful and empirically based ELP performance standard?

    • The report describes three methods for analyzing empirical data in order to assist policymakers in determining an empirically based ELP standard for ELs, applies these methods to data from three states, and discusses how the results might be interpreted and used to support decision making. The approaches are:

      1. Decision consistency analysis, which analyzes linguistic and academic proficiency-level categorizations and seeks to optimize consistent categorization of ELP students at the state's pre-established academic proficient cut score.
      2. Logistic regression analysis, which estimates the probability of being proficient on academic-content assessments for each ELP score. This approach could identify ELP scores for which students have a probability of equal to or greater than 50/50 (0.5) of being proficient on the content assessment.
      3. Descriptive box plot analysis, which identifies the ELP level at which at least half of the EL students are scoring above the academic-content proficient cut score. At this point, students are equally distributed above and below the state's proficient performance standard in academic content. This may suggest that, above this point, more than just language proficiency is contributing to observed scores.

    • Taken together, these three approaches provide multiple sources of evidence to investigate and corroborate the point at which an ELP performance standard might be set. The report recommends that all three approaches be used, when feasible, in order to provide policymakers with more complete, "triangulated" empirical evidence for delimiting a range of performance and defining options to establish an ELP performance standard for ELs.

  2. What analytical methods can be used to establish a realistic, empirically anchored time frame for attaining a given ELP performance standard?

    • The report outlines two analytical approaches for establishing a target time frame for ELs to attain a pre-identified ELP performance standard:

      1. Descriptive analysis, which follows over time EL students who start at a pre-specified date at varying English proficiency levels. The proportions of EL students who annually attain the ELP criterion are then shown in a bar chart. The goal of this approach is to get a sense of percentages attaining language proficiency, by time, initial ELP level and grade span.
      2. Event history analysis, which is also known as survival analysis and is used extensively in the fields of engineering and medicine to estimate the time required for an event of interest to occur (Klein and Moeschberger, 1997). For analyses in this report, the event is an EL student's attaining the given ELP performance standard. The goal of this approach is to calculate a time frame that incorporates students for whom the event of interest does not occur.

  3. How can an EL's ELP level be taken into account when setting academic progress and proficiency expectations?

    • The report outlines three analytical approaches:

      1. Progressive benchmarking. This family of methods adjust either (a) EL students' content achievement scale scores or (b) EL students' weight (their individual "count"), based on each student's ELP level relative to his or her initial ELP level and time in the state school system. In this method, there is an expectation that 1) students will increase in English language proficiency annually from their level of initial English proficiency and that 2) students will increase in content achievement annually. Thus, while recognizing the effect of limited English proficiency on ELs' academic performance on tests given in English, scale score or calculative weight adjustments lessen as students increase in ELP level, as expected, or, if they do not, as they continue in EL status over time. In essence, expected performance benchmarks progressively increase (and corresponding adjustments progressively decrease) to the point at which no adjustments are made at all.
      2. Indexed progress method. This method uses an EL's ELP growth as a proxy for academic content performance on a weighted, time-sensitive basis for more newly arrived ELs who enter the state's school system at lower initial ELP levels. These weights and time frames are empirically derived for each subject matter and grade tested because "the impact of limited English proficiency on academic performance varies by subject matter and grade [e.g., ELs with lower levels of language proficiency have more difficulty demonstrating content knowledge in English language arts compared with mathematics, and this difficulty increases at higher grade levels]" (Working Group on ELL Policy 2010, p. 5).
      3. Status and growth accountability matrix (SGAM). This method acknowledges student attainment of academic proficiency (i.e., the AYP performance standard) and/or a predetermined, acceptable level of student growth toward academic proficiency (e.g., a level of academic progress to be considered "on track" to proficiency in a reasonable time frame), without considering an EL's ELP level.

    • The report identifies numerous caveats to keep in mind in using these methods. For example, the methods are exploratory and are meant to foster discussion and more research. Second, the outcomes generated by these methods may vary quite a bit depending on the grade level and the assessments used.

Previous Reports from this Study

ED released three evaluation briefs from this study in May of 2010:

  • Title III Policy: State of the States;
  • Title III Accountability: Behind the Numbers; and
  • Title III Accountability and District Improvement Efforts: A Closer Look.

These three briefs are located at: http://www2.ed.gov/about/offices/list/opepd/ppss/reports.html#titleiii

Contractor

The study is being conducted by the American Institutes for Research (AIR).


 
Print this page Printable view Bookmark  and Share
Last Modified: 03/29/2012