A key component of Title I is a high-quality assessment system aligned with the States content and performance standards that helps to ensure that the performance expected of children in Title I schools is the same as that expected of all children. Such an assessment system would monitor progress toward achievement goals for accountability and would stimulate and support superior instruction, functions that are not well served by many current assessment systems. Allowing States to develop or adopt different assessment systems provides for flexibility to address the needs of children in each State and its LEAs and schools.
Title I requires each State to develop or adopt a set of high-quality yearly student assessments that measure performance in at least mathematics and reading/language arts, and to use these assessments as the primary means of determining the yearly performance of each school and LEA served under Part A in enabling all participating children to meet the States student performance standards.
This section focuses on the assessments in State assessment systems which are intended for monitoring the progress of schools and districts in helping all students attain the States standardsthe assessments which will constitute the primary basis for judging the "adequate yearly progress" of Title I schools and districts. Assessment requirements during the "transitional" period while States are developing these new systems are discussed in a later section of this guidance. "Transitional" assessments are allowed until the year 2000-2001, so States will have time to create an aligned system of challenging content and students performance standards with appropriate assessments. This section first discusses general assessment requirements, then examines assessment issues for special populations of students, and ends with a consideration of various technical quality issues.
Statute and Regulations
(Section 1111(b)(3); § §200.1(b)(2) and 200.4)
Each State shall develop or adopt a set of high-quality yearly student assessments, including assessments that measure performance in at least mathematics and reading/language arts, that will be used as the primary means of determining the yearly performance of each school and LEA served under Part A in enabling all participating children to meet the States student performance standards.
A State may satisfy this requirement if it has developed or adopted a set of high-quality yearly student assessments in other academic subjects that measures performance in mathematics and reading/language arts.
Assessments must meet the following requirements:
- Be the same assessments used to measure the performance of all children, if the State measures the performance of all children.
- Be aligned with the States challenging content and student performance standards, and provide coherent information about student attainment of the States standards.
- Be used for purposes for which the assessments are valid and reliable, and be consistent with relevant, nationally recognized professional and technical standards for those assessments. (Assessment measures that do not meet these requirements may be included as one of the multiple measures if the State includes in its State plan sufficient information regarding the States efforts to validate the measures and to report the results of those validation studies.)
- Measure the proficiency of students in academic subjects in which the State has adopted challenging content and student performance standards.
- Be administered at some time during
- grades 3 through 5;
- grades 6 through 9; and
- grades 10 through 12.
- Involve multiple approaches within an assessment system with up-to-date measures of student performance, including measures that assess complex thinking skills and understanding of challenging content.
- Provide for
- participation in the assessments of all students in the grades being assessed;
- reasonable adaptations and accommodations for students with diverse learning needs necessary to measure the achievement of those students relative to the States standards;
- inclusion of LEP students, who shall be assessed, to the extent practicable, in the language and form most likely to yield accurate and reliable information on what they know and can do to determine their mastery of skills in subjects other than English. (To meet this requirement, the State shall make every effort to use or develop linguistically accessible assessment measures, and may request assistance from the Secretary if those measures are needed.)
- Include, for determining the progress of the LEA only, students who have attended schools in the LEA for a full academic year, but who have not attended a single school in the LEA for a full academic year.
- Provide individual student interpretive and descriptive reports that include individual scores or other information on the attainment of student performance standards.
- Enable results to be disaggregated within each State, LEA, and school by gender, and by major racial and ethnic group, English proficiency status, migrant status, students with disabilities as compared to students without disabilities, and economically disadvantaged students as compared to students who are not economically disadvantaged.
If a State has developed or adopted assessments for all students that measure performance in mathematics and reading/language arts under Title III of the Goals 2000: Educate America Act or under another process, the State shall use those assessments, modified, if necessary, to conform with these requirements.
The requirement to assess in mathematics and reading/language arts does not relieve a State from including students served under Title I in assessments in any other subjects the State has developed or adopted for all children.
If a State has not currently developed or adopted assessments that measure performance in at least mathematics and reading/language arts, the State shall
- by the beginning of the 2000-01 school year, have developed those assessments and field-tested them for one year; and
- develop a timetable and benchmarks of quality, including reports of validity studies, for completing the development and field testing of those assessments.
The State may request a one-year extension from the Secretary to test its new assessments if the State submits a strategy to correct problems identified in the field-testing of its assessments.
If a State has not developed assessments that measure performance in at least mathematics and reading/language arts by the beginning of the 2000-01 school year and is denied an extension, the State shall adopt a set of assessments in those subjects such as assessments contained in the plans of other States the Secretary has approved.
Questions and Answers
14. Must State assessments used for Title I be uniform statewide?
Title I requires that participating States develop and implement State assessments that are aligned with the States content standards. There are three approaches that States are implementing: a single Statewide assessment system under which all students being assessed take the same assessments; a Statewide system with local assessments added with the approval of the State; and local assessment systems approved by the State on the basis of models or criteria to ensure high quality assessments.
15. Must a State develop or adopt assessments in all subjects?
Title I requires that assessments be developed or adopted in at least mathematics and reading/language arts. Assessments in mathematics and reading/language arts are the minimum required. If a State has developed assessments for all children in any other subjects, children participating in Title I must be included in those assessments as well. In fact, because the purpose of Title I is to help children achieve in all academic areas, not just math and reading,States are strongly encouraged to use assessments in all academic areas to determine how well children in Title I schools are doing. States and LEAs should thus establish assessments systems that adequately inform teachers, parents, and the community of a schools or LEAs progress toward attainment of State performance standards and provide information to improve instruction.
16. Must all participating children in Title I be assessed?
No. In the grades selected for assessment all children participating in Title I must be assessed including students with disabilities and LEP students. For example, if a State assesses in grades 4, 7, and 11 all Title I participating students in these grades must be included. Title I students in other grades need not be assessed.
17. May children participating in Title I be assessed with different assessments than those used by the State for all other children?
If a State and/or LEA measures the performance of all children, such assessments must be used to measure the performance of children participating in Title I. Moreover, children participating in Title I must be part of any accountability system the State establishes for all children. Such assessments may be modified to accommodate LEP students and students with disabilities provided they meet standards of comparability.
18. Must a State use only mathematics and reading/language arts assessments to satisfy Title s requirements that assessments be in "at least mathematics and reading/language arts "?
No. The Title I regulations permit a State to assess students in academic subjects other than mathematics and reading/language arts if those assessments also measure performance in mathematics and reading/language arts.
19. When a State submits its plan to the Secretary, must it also submit a copy of its assessments?
No. A State need not submit its assessments to the Secretary. Rather, a State plan must demonstrate that the State will develop or adopt or has developed or adopted a set of high-quality, yearly student assessments that will be used by the State, its LEAs, and its schools to carry out the Title I requirements.
20. If a State does not have to submit copies of its assessments with its State plan, how may a State demonstrate in its plan that the State has developed or adopted a set of high-quality, yearly student assessments?
A State plan can demonstrate that the State has high-quality assessments in some or all the following ways:
- The plan demonstrates that assessments are aligned or will be aligned with challenging standards.
- The plan describes how high-quality assessments will influence instruction and help support continuous improvement of schools and districts.
- The plan describes how quality benchmarks were chosen and the plans for their validation.
- The plan describes validity designs and implementation plans for individual measures.
- The plan describes rules for combining results across performance standards and a validation plan.
- The plan describes the technical quality of accommodated or adapted versions of assessments.
- The plan contains a reasonable schedule for developing assessments including field testing.
- The plan describes the process through which the State will move from transition assessments, where used, to its final assessment.
- The plan explains and provides examples of reporting procedures for various audiences.
21. What are some ways to judge the alignment of local standards and assessments with State standards and assessments?
If the State allows the use of local standards and/or assessments, the State has the responsibility of ensuring that the local standards and assessments are aligned with the States system. Areas of alignment that the State may consider include--
- range of coverage;
- depth of coverage;
- degree of emphasis on topics or areas; and
- degree of rigor.
In reviewing local assessments, States can look at the alignment of the assessments with the State assessments and with the State and local content standards. The same areas reviewed for determining alignment of content standards can be useful in reviewing assessments.
Empirical evidence of alignment between State and local standards and assessments may also be gathered. For example, a State may wish to administer its assessment to a sample of students in a district and compare the results to the results of the local assessment.
Question 4 addresses issues related to judging the quality of content standards and is accompanied by sample criteria for evaluating standards.
22. In a targeted assistance school, if Title I children receive services in only one subject, must they be assessed in at least mathematics and reading/language arts.
Yes. Even if participating children only receive services in one subject, such as mathematics, they still must be assessed in at least mathematics and reading/language arts for Title I purposes.
23. What is meant by assessments that involve multiple up-to-date measures of student performance?
This provision requires that different approaches and formats be used in the assessment system. Examples include criterion-referenced tests, multiple choice tests, writing samples, completion of graphic representations, standardized tests, observation checklists, performance of exemplary tasks, performance events, and portfolios of student work. The assessments must include measures of complex skills and understanding of challenging content in at least mathematics and reading/language arts. Data derived from indicators such as attendance and graduation may be used to augment these approaches but not to replace them.
24. Do multiple approaches within an assessment system mean that a State must have multiple approaches within each subject area assessed?
Multiple approaches, such as a criterion-referenced test and a performance event, for a single subject are not required. However, multiple approaches may provide more complete information on school progress.
25. Must migratory students be included in a State assessment system?
Including migratory students in State assessments is important to achieving ESEAs goal that all children achieve to high content and performance standards. It is important that States coordinate with each other so that migratory students are neither subject to assessment programs in several districts or States nor are excluded altogether from these assessment systems. States may wish to consider including migratory students in a State assessment against the standards of their home-based State (i.e., the State in which the child intends to graduate). Each State which receives a migratory student for a period during the year should coordinate with other States in which the student has been enrolled to provide continuity in the educational services the studentreceives, and with the students home-base State regarding the administration of assessments and the reporting of results.
26. If a State has not developed or adopted assessments in at least mathematics and reading/language arts, by when must they be developed or adopted?
These assessments must be developed and field tested by the beginning of the 2000-01 school year. A State may request a one-year extension from the Secretary to test its new assessments, if the State submits a strategy to correct problems identified in the field testing of its assessments.
27. Must all students served by Title I in a district or school have the same data reported in their individual student interpretive and descriptive reports?
No. The individual student reports required by law need not be composed of comparable elements. Childrens profiles on a number of measuresfor instance, their scores on matrix-sampled performance tasks, or their actual workmay be included without meeting strict standards of comparability among each of the sampled items. The goal of individual reporting is to provide the best possible information to parents about their own childrens progress and to teachers about students in their classes.
Assessment of Participating Private School Students
In consultation with private school officials, an LEA must provide participating private school children an equitable opportunity to meet the States content and student performance standards. In some instances, however, it may not be appropriate to expect private school children to meet the States standardsfor example, if those standards are not aligned with the curriculum of the private school. If the LEA, in consultation with private school officials, determines that it would beinappropriate to measure the achievement of participating private school children in relation to the States content and performance standards, the LEA must develop alternative program goals that provide reasonable promise of those children achieving the high levels called for by the States student performance standards.
An LEA must assess annually the progress of the Title I program toward enabling private school Title I participants to meet the States challenging student performance standards (or the LEAs alternative program goals). Generally, an LEA must assess the progress of the Title I program using the States definition of adequate yearly progress. However, the LEA may need to modify that definition, in consultation with private school officials, to better measure the progress of participating private school children. In measuring adequate yearly progress, the LEA has the flexibility to group children in a manner that will provide the most accurate information of this progress. For example, the LEA may decide to group children by type of instructional method, grade level, school, or other appropriate basis.
In general, an LEA must use the State assessment system (i.e., the final assessment required under section 1111(b)(3) of Title I or the transitional assessment under section 1111(b)(7)) as well as any additional measures or indicators the LEA deems necessary in order to measure how well the Title I program is enabling the private school students to meet the States challenging student performance standards (or the LEAs alternative program goals). In some instances, however, it may not be appropriate for the LEA to use the State assessment system. If the LEA, in consultation with private school officials, determines that the State assessment would not provide accurate information about the progress of participating private school children, the LEA may use other assessment measures that more accurately reflect the progress of those children toward meeting the States standards.
If an LEA determines that a Title I program serving private school children has not made adequate progress (or met the criteria established by the State for transitional assessments) for two consecutive school years, the LEA, in consultation with the private school, must develop a Title I program improvement plan that has the greatest likelihood of improving the performance of participating private school children in meeting the States student performance standards.
28. May Title I funds be used to assess private school children?
Title I funds may be used to assess private school children if the assessment is being used only for Title I purposes. To the extent, however, that an assessment is conducted for other purposes, it may not be paid for from Title I funds. For example, if private school children, in general, are included in the State assessment, Title I funds may not be used to pay for the assessment of those private school children participating in Title I.
Early Learning Assessment
Almost half of the students served by Title I are in grades below third grade. However, Title I does not require assessment until the third, fourth, or fifth grade. This leaves a major part of the Title I population being served in grades for which accountability information may be indirect. States may wish to collect performance data for children in grades K-2 for early diagnosis of childrens needs, to evaluate program effectiveness, guide curriculum and instruction, or provide information that can be used to measure school and LEA progress. As a result, it is important to consider several factors carefully in developing an assessment program for these students.
29. Is it necessary to assess students below grade 3 for Title I?
Title I does not require pre-grade 3 assessment for determining school and district progress; however, there is a need for both formative evaluation information and information on the success of Title I in the early grades. Although "formal" assessment below grade 3 may not be part of the States assessment system, districts may wish to develop measures of success of their early childhood programs. There are four reasons that States may wish to assess students below grade 3:
- to improve classroom instruction and learning;
- to assess the impact of Title I in grades K-3, since about half of all Title I students are in those grades;
- to gather information about adequate yearly progress and accountability; and
- to gather information on a K-2 building.
When States request information on primary school achievement it raises awareness and has an impact on programs. Asking for data does not imply depending on paper and pencil, large-scale assessments, which are not allowed below grade 3 in some States. Data might consist of teachers reporting what percent of their students are achieving at appropriate benchmark levels for their grade. The reports can be based on teachers observations and samples of students work.
30. May States include student assessment information from students below Grade 3 to judge adequate yearly progress?
Yes. Although Title I requirements for grade-level coverage in the State assessment system begin with grades 3-5, States may use assessment information from earlier grades for accountability purposes.
31. What do teachers need to know to make judgments of student progress in relation to content and performance standards?
In order to produce judgments of student progress, teachers need to be proficient in sound classroom assessment and knowledgeable in content and performance standards. With a move toward more complex assessments both in classrooms and at district and State levels, teachers need to develop their professional capacity to use and interpret newer assessments. Title I calls for informative, diagnostic reporting on individuals classroom performance and for program accountability that aligns with the accountability used for all students. The classroom assessment component often will require teachers understanding of how to judge classroom portfolios or how to make valid, reliable observations of students performance. The accountability demands mean that teachers must understand and be able to teach to the more complex accountability assessments that are being implemented in an increasing number of States and districts.
32. What are appropriate assessment strategies and considerations for assessing early learners?
Some States have legislation that prohibits the testing of young children with standardized, paper and pencil, large group tests. This legislation is based on research on the needs and development of young children and reflects the guidelines for assessing young children published by such groups as the National Association for the Education of Young Children. Some States, such as South Carolina, have worked with districts to develop their own sets of guidelines for assessing young children. While it is important to consider developmental differences in young children, it is also essential to avoid using those initial differences as excuses for students not making adequate progress. Keeping content and performance standards in mind, assessment approaches need to be designed so that teachers can determine and communicate student achievement with respect to the standards rather than in relation to where students start. The latter focus may result in students falling further behind with each year rather than catching up to other students.
Assessments and Limited English Proficient Students
For limited English proficient students appropriate assessment instruments and other measures should take into account content area and the language of instruction. Limited English proficient students should be assessed to determine their mastery of skills in subjects other than English. The decision about appropriate accommodations, if needed, should be made on an individualized basis.
33. How does a State address the needs of LEP students when developing challenging content standards?
Standards should be developed with communities that include teachers, parents and others who work with and understand the languages and cultures of LEP students. State and locally developed standards should incorporate the cultural background and life experiences of linguistically diverse children and reflect the diversity of the State or district. Similarly, the curriculum and assessments should include elements of the local languages and cultures and be linked directly to the standards.
34. Must LEP students be required to meet the same content and performance standards as other students?
Yes. Title I requires that content and performance standards apply to all participating students, including limited English proficient students. Initially, however, LEP students may take more time to meet the standards because they must also develop English language proficiency. As such, additional benchmarks towards meeting content standards might be developed to assess LEP student progress. In addition, States might develop supplemental benchmarks in English language arts that address the acquisition of English language skills.
35. How do we support more effectively LEP students achievement of content and performance standards?
This will require effective instruction, assessment of students to determine where they are relative to the State standards, good teacher professional development, appropriate materials development, family involvement, knowledge of the second language acquisition process, and knowledge of the cultural and communications patterns that bear on the academic performance of LEP students.
36. What factors should be considered when deciding how to assess LEP students?
Assessments for LEP students are to be rigorous yet appropriate for the needs of these students. Because of the diversity of this population (in terms of languages spoken and the level of English proficiency), no single method may address the needs of all LEP students. A variety of approaches is recommended. One is to assess content knowledge in the primary language. This approach may be appropriate for students who receive instruction in the primary language or those who are more proficient in their primary language than in English. The primary language assessment should cover the same standards being assessed for all students.
37. If assessment in the primary language is impracticable or inappropriate, what other assessment practices can be used in order to meet the needs of LEP students?
Other modifications currently in use at the State and local levels include extra time, small group administration, flexible scheduling, simplified directions, and allowing the use of dictionaries. Other modifications might entail providing audio-taped instructions in the primary language, allowing students to respond in either their primary language or English using audiotape, providing additional clarifying information at the end of the test booklet or throughout the test (e.g., synonyms for difficult words or phrases), and decreasing the English language demands of the assessment.
38. How does a State or LEA determine whether to test in English or use a modified assessment?
Decisions regarding when to test LEP students in English should be based on assessment of English language skills that measure four domains of language (speaking, listening, reading, writing), and on the number of years a student has received academic instruction.
39. How does a State or LEA determine whether to test in English or in the primary language?
States should develop guidelines to help decision-makers choose the most appropriate assessment option. The most appropriate option should enable a child to demonstrate knowledge and skills in the content area.
40. How should LEP student assessment scores be disaggregated?
When the final assessment system is in place, student assessment results must be disaggregated by LEP status. Further disaggregation by economic status would help determine whether differences between LEP students and non-LEP students are a result of socioeconomic status or language proficiency.
41. How might States go about developing assessments for LEP students?
States and local school districts might consider borrowing (from other States or entities such as large school districts with substantial numbers of LEP students) content area assessments in languages other than English if such assessments conform with their content standards. This process might also involve cooperative efforts among two or more States, or the development of multi-state item banks and should include persons knowledgeable about the assessment of LEP students and systems serving them. The newly formed Regional Comprehensive Assistance Centers will be a source of help in locating and developing assessments in languages other than English. Finally, States might request assistance from the Office of Bilingual Education and Minority Languages Affairs (OBEMLA). OBEMLA is required to assist States that request such assistance and identify appropriate assessment measures in languages other than English that are present in their Title I student populations.
Assessment and Students with Disabilities
Students with disabilities must be included in State and local assessments. The term "students with disabilities", as it is used in this guidance, refers to students who are eligible for services under the Individuals with Disabilities Education Act (IDEA) as well as students who are covered under Section 504 of the Rehabilitation Act of 1973 (Section 504) and Title II of the Americans with Disabilities Act (ADA). Under IDEA, a student is eligible for services if the student has one of the covered impairments and because of that impairment needs special education and related services. Under Section 504 and Title II, the student is covered if the student has a physical or mental impairment which substantially limits one or more major life activities such as learning.
42. Students with disabilities are tested extensively. Why should they be included in the assessment system that other children are expected to take?
Currently, much of the individualized testing of students with disabilities is designed to decide whether they are eligible (or remain eligible) for special education services. Assessments for Title I purposes focus on whether the yearly performance of each LEA and school served under Title I is enabling all children served to meet the States student performance standards. It is important that students with disabilities be included in these assessments because they are expected to meet the same standards as other students.
43. What if assessments being used to measure the performance of all students do not measure what students with disabilities are currently learning in their programs?
While some students with disabilities may need modified instructional approaches, generally all students need to be working toward the same challenging standards.
44. Must students with disabilities be included in Title I assessments?
Yes. Title I and Federal civil rights laws require the inclusion in assessments of all children, including children with disabilities, in the grade being assessed. For a small number of students with disabilities, the severity of their physical or cognitive limitations prevents them from participating meaningfully in exactly the same assessments as other students, even with the availability of appropriate accommodations. For this small population of students, appropriate alternatives should be used to assess their educational progress.
45. What assessment accommodations may be provided for students with disabilities?
Students with disabilities must be provided with appropriate accommodations when necessary to enable participation in the assessments. Assessment accommodations include changes in the way assessment items are presented, changes in the way a student may respond, changes in the timing or scheduling of an assessment, and changes in the setting that are used to provide an equal footing for students with disabilities who need the accommodations. Assessment accommodations help students show what they know without being placed at a disadvantage by their disability.
46. Who decides which students with disabilities get assessment accommodations, and how is this decided?
There are no specific Title I regulations that indicate who is to make the decision about which students with disabilities require assessment accommodations during assessments, nor about how the decision is to be made. Current practice in most States with statewide assessments is to have the students Individualized Education Program (IEP) team make the decision about the assessment accommodations that the student needs during assessments. If a students IEP states the assessment accommodations that the student is to receive, then these accommodations must be provided. Some States, however, have other mechanisms in place for deciding about the use of accommodations.
For best testing practice, it is generally recommended that the decision about the use of accommodations be made by a person or a group of people who knows the student. Typically, this is assumed to be the IEP team or a group that knows the child as is the case of a child covered by Section 504 that must receive individualized appropriate education but does not have an IEP. These decisions are to be made on the basis of individual student characteristics and needs, not on the basis of labels (such as category of disability). In determining which students require accommodations during Title I assessments, it is important to be aware of the role ofaccommodations in instruction and in assessment. Although there are exceptions, it is generally held that the accommodations students receive during instruction are the accommodations that they should receive during assessment.
47. How do I know which accommodations may affect the validity of assessments?
There are many questions about which accommodations preserve the reliability of test scores and the validity of inferences (whether the test measures the same content standards in the accommodated and non-accommodated versions) based on these scores. These questions have not yet been fully answered. Specific test protocols provided by commercial achievement test publishers may or may not provide adequate information on which accommodations produce comparable scores between accommodated and non-accommodated versions of the tests.
In some cases, decisions on allowable accommodations can be based on some logical assumptions about the purpose of the test. For example, a measure of silent reading comprehension could be invalidated by reading the test items to the student. On the other hand, providing a quiet setting for highly distractible students could hardly be seen as compromising the test's validity.
Technical Issues of Assessment
A key component of State and local reform efforts and Title I is a high-quality assessment system aligned with the States content and performance standards that helps to ensure that the performance expected of children in Title I schools is the same as that expected of all children. Title I assessment requirements pose several new challenges. First, State and local assessments must demonstrate how well schools enable students to meet challenging standards developed by the State, rather than focusing on how well students perform against "the norm." Second, the assessments must focus on specifically defined performance standards rather than the generic skills measured by most norm-referenced tests. Finally, Title I requires a shift in accountability to the performance of the school and LEA in educating students to meet rigorous content standards.
Several important technical issues involved in the use of assessments for determining progress are discussed below: the use of multiple measures and approaches, matrix sampling, and technical quality.
48. What are multiple measures and approaches?
Title I requires that the assessment system include multiple approaches to assessing student achievement in at least reading/language arts and mathematics. A measurement approach is a procedure for measuring student achievement. Measurement approaches encompass a wide variety of strategies, including traditional tests that use items and tasks such as multiple-choice items, short-answer items, and extended-response tasks. Other approaches emphasize student performance and include the use of portfolios and projects designed to be assessment tools, observational checklists, interviews, running records, and many other techniques.
49. Why are multiple approaches desirable?
A single measure or approach is unlikely to adequately measure the knowledge, skills, and complex procedures covered by rigorous content standards. Multiple measures and approaches can be used to capitalize on the strengths of each measurement technique, enhancing the utility of the assessment system and strengthening the validity of decisions made based on assessment results.
Multiple measures and approaches can enhance the validity, reliability, equity, and utility of the inferences made about student achievement by increasing content coverage and matching the assessment approach to the knowledge, skills, and complex procedures contained in rigorous academic content standards. It is important that information about student achievement in a content area cover the full range of content and use appropriate assessment techniques matched to the content standards.
Multiple measures are useful when the assessment system is used for multiple purposes. For example, a broad test used to sample the content area may provide information about student performance that is a useful component in determining progress at the school level. Results of extended open-ended assessments may supplement the multiple-choice results at the school level, adding information for evaluation purposes, and provide more useful information to teachers for designing specific instructional approaches for their students.
Multiple approaches are also important because of the relationship between assessment and curriculum. Research suggests that the format of "formal" assessment often affects curriculum, so that the curriculum reflects only what is assessed and the manner in which it is assessed. A mixture of measurement approaches can counteract the tendency to narrow the curriculum.
50. Must the results of multiple measures be combined for individual student reports?
No. Title I requires that the assessment system provide individual student reports, but it does not require that a single score be reported for each content area. Depending on the nature of the measures used and the purpose(s) for using multiple measures, it may or may not be desirable to combine scores for student reports. Validity of the inferences that will be made based on the scores are key considerations in decisions about reporting combined scores. Even though scores may be combined for one purpose, the original, non-combined scores can still be useful in clarifying what an individual student knows and can do in the content area assessed.
51. Should scores on multiple measures in a content area be combined to describe school and district performance?
Title I requires that the measure of yearly progress of schools and districts in helping students served by Title I to meet challenging content standards be based primarily on the States system of assessments. For example , a school result might be the proportion of students who scored at each of the States proficiency levels. If multiple measures are used as part of the assessment system, it may be useful to combine the information from the measures in reporting school and district-level results.
For example, the assessment system used to measure yearly progress may include three mathematics assessments. School or district results could be reported based on student performance on each assessment, producing three mathematics results for the school/district, or the data from the three assessments could be combined to produce a single mathematics result for the school/district.
The decision of whether to combine scores to produce school or district results, and the method used to combine scores, should be based on the need to produce reliable and valid information for the purpose(s) and use(s) intended. If school/district results are reported for each assessment separately, overall judgments of performance in a content area can be based on the pattern of results, using either a conjunctive approach (requiring a particular level of performance on each assessment) or a compensatory approach (allowing performance on the various assessments to counterbalance each other).
52. May a sampling design be used to collect information for measuring school and district progress?
If a school receives Title I funds, assessment data about student performance in at least reading/language arts and mathematics must be used in determining whether the school is making adequate yearly progress. Therefore, any sampling design used must provide results that can be reported at the school level in at least these two content areas.
53. May State assessments employ matrix sampling in Title I schools?
Matrix sampling is a technique in which the total set of assessment items is divided among several test forms, usually of comparable difficulty. Within a school (or other unit of interest),results are available based on the whole set of items; however, each student responds to only a portion of the items. For example, three different versions of a test in a content area may be developed, with each form containing a different set of items and tasks. Because each student receives only one of the test forms, each student responds to only one-third of the items covered by the entire set of test forms.
For States considering using matrix sampling, special concerns are targeted assistance schools and schools with small enrollments. Enough students need to respond to each test form in order to be able to make valid inferences based on school-level scores. In targeted assistance schools that use assessment data only from students served by Title I in measuring progress, matrix sampling designs may not provide adequate information. In small schools, each student may have to receive more than one test form in order to provide enough information at the school level.
54. What are some advantages and disadvantages of matrix sampling?
Matrix sampling can increase the content coverage assessed at the school, district, and State levels. It can reduce the amount of time spent on assessment and allow the use of a wider variety of assessment approaches than might be possible in a design that requires every student to respond to every assessment exercise. Research indicates that matrix sampling can increase the reliability and stability of school-level scores used for measuring progress, even in small schools.
Although matrix sampling is efficient, it reduces the information available from the assessment for individual students and may have to be supplemented by other assessment measures. A matrix sampling approach might make it difficult to provide data that can be disaggregated.
55. What are the technical issues involved in matrix sampling?
The validity of student-level scores based on a content sampling technique is a key consideration. Matrix sampling may result in scores that are not useful for individual students.
At the school and district levels, the number of students taking each form needs to be large enough to provide reliable and valid results. Assessment forms should be carefully designed so that the targeted unit (e.g., the school) receives appropriate content coverage for the purpose(s) of the assessment. Issues of reliability, validity, and equity that apply to individual student scores can be applied at the group level as well (see section on technical quality).
Technical Quality of Assessments
In determining how to apply technical standards to assessments used in the State assessment system, a primary consideration is the use of assessment information, that is, the inferences made based on assessment results. When assessment results are used to provide information at the school or district level, the quality of decisions made based on the entire system of information is important. The quality of the inferences made about individual students is critical when these same assessments are used to make student-level decisions.
56. What technical standards should be applied to the assessment system?
Title I requires that the assessment system be used for purposes that are valid and reliable and that it meet nationally recognized professional and technical standards. The law further allows the use of some measures that do not yet meet these requirements, if these measures are part of a system of multiple measures, and the State is in the process of collecting evidence of the validity of these measures. If the State assessment system used for measuring school and district progress includes locally developed or selected assessments, the State is responsible for ensuring that the local assessments meet the technical standards required by the law. The State may require that localities collect evidence of technical quality, or the State may collect such evidence.
57. How do technical standards apply to uses of assessment data at the school and district levels?
Many of the same technical standards that are applied to uses of assessment data to make inferences about individual students can also be applied to make inferences about aggregate performance. For example, the data should adequately cover the knowledge, skills, and complex procedures contained in the content standards assessed; if schools or districts include some local measures in the aggregated data, comparability of decisions made across schools and districts should be investigated and ensured. Equity issues that apply to individual student scores are equally important at the aggregate level. Intended effects of the assessment system, such as encouraging changes in curriculum and instruction, as well as potential unintended effects, such as narrowing the curriculum, are important issues to explore.
For example, when a combination of a variety of sources of information is the basis for deciding whether a school should be put in program improvement, the stability of the information and the validity of the decision become key issues. Similarly, when school performance is described based on the proportion of students meeting each of the States performance standards, the technical quality of the aggregate classifications is important.
58. What are some sources of information regarding technical quality of assessments?
The primary reference for technical quality of educational assessments is Standards for Educational and Psychological Testing (1985), developed by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. This document contains descriptions of standards for test development, use, administration, and interpretation. With the exception of one chapter on program evaluation, the standards are oriented towards the use of assessment to provide individual student scores. The principles embodied in the standards are applicable to any educational assessment; however, the language and content are more readily interpreted in the context of traditional standardized assessments.
The Standards are currently being revised The revisions will more clearly apply to a variety of measurement approaches, including performance-based assessments.
Other sources of information for evaluating the technical quality of educational measurements are available. A number of articles discussing applications of technical standards to performance based assessments, with specific recommendations, have been published in recent editions of professional journals (e.g., Journal of Educational Measurement, Educational Evaluation and Policy Analysis, Educational Researcher, Educational Measurement: Issues and Practices). Organizations such as the National Center for Educational Statistics, the National Center for Research on Education, Standards, and Student Testing, and the new Standards Project have also published reports devoted to technical issues in performance assessments. The Regional Comprehensive Assistance Centers can provide States with resources and assistance in designing and evaluating their assessment systems.
States with performance assessment programs are working with independent evaluators and conducting their own research to enhance the technical quality of their programs (e.g., Vermont, Maryland, Kentucky). In addition, States have banded together to develop technically-sound performance-based assessments and to conduct research on technical issues, through the Council of Chief State School Officers State Collaborative on Assessment and Student Standards.