Idaho State Department of Education
Alternate Assessment Design—Reading (AAD-R)
A consortium of states including Idaho, Kansas, and Utah are applying for EAG funding for the proposed Alternate Assessment Design—Reading project. With technical support from SRI International and an impressive group of nationally known experts already committed to supporting this project, states will collaborate to (a) extend the conceptual framework of evidence-centered design (ECD) to alternate assessment based on alternate achievement standards (AA-AAS) in reading using the Principled Assessment Designs for Inquiry (PADI) model and (b) develop AA-AAS testing designs and performance tasks that will address the states’ priority academic standards in reading.
Each of the collaborating states have completed one NCLB peer review cycle with their current AA-AAS systems. They are revising or improving their assessment systems to improve technical quality and reliability. Until now, few states have documented AA-AAS test development or the rationale used to set priorities for determining test content. The proposed project will employ ECD and the PADI model, a systematic process to guide the selection of content and the design of assessment tasks. The collaborating states have experience with ECD and PADI that they will extend to a new content area not previously addressed with this approach. AAD-R will build on the foundation of earlier work in other content areas and benefit from the lessons learned from their previous experience.
The project objectives include (a) developing design patterns—frameworks or schema used to design assessments, (b) describing the conditions required to effectively present tasks and evaluate student performance, and (c) producing examples of assessment tasks and scoring systems. Design patterns and tasks will be housed in an online bank available to all the states. Each of the states will pilot test a set of tasks and employ common instruments to collect data. The project will produce Procedural Guidelines for designing assessments and assessment tasks, conduct an informational webinar, and make presentations at national conferences. The proposed project will expand and strengthen the representation of grade-level reading content in AA-AAS. This project offers an opportunity both to extend a contemporary approach to AA-AAS test design through a novel application and to evaluate this extension across multiple state contexts.
Minesota Department of Education
The Accessible Portable Item Project
Computer-based test delivery holds promise to increase the efficiency with which tests are administered and the speed with which results are returned to schools. Two challenges to computer-based delivery, however, are the provision of test accommodations and the ability to easily deliver test items across different delivery systems. The Accessible Portable Item Protocol (APIP) Project brings together a consortium of states (MN, FL, MD, MT, NH, SC, UT, & VT) to develop the capacity of all states to use a standard item mark up language for accessible computer-based test items. As a result of this project, the APIP will allow all states to ensure that our test items are accessible for students with a variety of needs and that our items are portable across computer-based delivery systems that apply the APIP standards.
The APIP will build on recently released Question and Test Interoperability standards to define standard methods for tagging test content so that it is presented in a consistent manner within any computer-based test delivery system that is developed to interpret the APIP standards. The APIP standards will significantly decrease costs associated with transferring items between the systems used by different vendors when a state testing program changes test vendors. The standards will also allow states to more easily use released items originally develop for an NCLB summative test for other purposes, such as formative or benchmark assessments.The APIP project will result in the following products:
- A primer designed to help all states develop an understanding of Question and Test Interoperability (QTI) standards and an associated Accessibility meta-tag system.
- A reference table that maps the universe of accommodations currently allowed by state testing programs to the Accessibility meta-tags.
- Sample RFP language that specifies the use of the QTI and Accessibility tags by test vendors.
- A sample of "best practice items" that apply the QTI and Accessibility Meta-Tags and which are delivered by a computer-based test delivery prototype to demonstrate the feasibility of employing the standards to deliver accessibility of test items in a computer-based environment.
Pennsylvania Department of Education
Modified Alternate Assessment Participation Screening (MAAPS) Consortium
The Modified Alternate Assessment Participation Screening (MAAPS) Consortium includes the departments of education from Arizona, Pennsylvania, and South Carolina, along with researchers from Vanderbilt University and the University of Pittsburgh, and test developers from EduWomen and Discovery Education Assessment, with the shared purpose of creating a multi-part screening system for identifying students who would be eligible for an alternate assessment based on modified academic achievement standards (AA-MAS). The MAAPS System will include electronic screening tests to predict proficiency in reading and mathematics, as well as a measure of opportunity to learn (OTL) essential academic objectives. The primary goals/objectives of the MAAPS Consortium are to (1) develop tools to facilitate educators' accurate assessment participation decisions for students with disabilities, (2) evaluate the validity and consequences of the participation decision-making tools, (3) apply the MAAPS system for students with disabilities to determine its utility and likely conseqences, and (4) disseminate knowledge learned from the development and implementation of the MAAPS System. Activities to accomplish these objectives include meetings to develop and refine measurement tools, several validity evidence studies, and ultimately a training conference for professional development. The primary outcome of this project is that IEP teams will be able to make reliable AA-MAS participation decisions. The MAAPS System will be designed for implementation at the 8th grade level in reading and mathematics, providing screening data in the form of repeated measures, to help educators make decisions with confidence. Secondary outcomes include examining the relationship between OTL and disability status, sharing information about methods for developing altered items for AA-MAS, and learning about a development process that can be extended to other grade levels. The MAAPS Consortium will draw from the successful work completed in the Consortium for Alternate Assessment Validity and Experimental Studies project (Elliott & Compton, 2006-2009), as well as on investigators’ experience in development and validation of alternate assessments and other educational assessment tools.
Utah State Office of Education
Description-Enhanced Assessments for Students with Visual & Print Disabilities
Overview: Audio description provides access to complex images and graphics for children with visual and print disabilities and plays an increasingly important role in multi-media classrooms. As an accommodation, however, description has not been approved by any state for use in state assessments, in spite of its potential to (a) control standardized test administration, (b) increase independent access to visual content, and (c) reduce costs in test construction.
Project goal and partners: The Utah, Colorado, and Kansas state education agencies seek to examine the use of description as an accommodation for students with visual and print disabilities by investigating student comprehension under multiple conditions and documenting meaningful and effective practices for access to visual and complex images within state assessments .Partners include WGBH National Center on Accessible Media, the National Center on Severe and Sensory Disabilities, and a panel of national advisors .
Project objectives and activities include (a) training partners in research-based descriptive practices; (b) analyzing, developing and field-testing descriptions using “retired” test items from the Utah Performance Assessment System for Students; (c) conducting two rounds of assessment with 450 students to measure comprehension and evaluate efficiency, clarity, and comprehension; and (d) producing guidelines for best practices in description of test items for national dissemination.
Project outcomes: (1) Student comprehension data that contributes to the research base on accessibility of test items to meet the diverse needs of students with visual and print disabilities; (2) capacity-building within partner states to provide consistent, efficient, meaningful, and cost-effective methods of providing access to complex images in test items through descriptions; and (3) guidelines for widespread dissemination to assist other states in developing description accommodation for their statewide assessments.
Virginia Department of Education
Assessing REAL Science on a Large-Scale Assessment: The Promise of Computer-Interactive Items for High School Students with Language Challenges
The Virginia Department of Education, in partnership with the New Jersey and North Dakota Departments of Education, the University of Wisconsin, Center for Applied Linguistics, and Pacific Metrics Corporation, requests $1,961,563 to complete the proposed project. Our goal is to improve the assessment of complex science knowledge and skills in two end-of-semester benchmark tests for all high school students, and especially those with language challenges (i.e. less English-proficient English language learners, students with learning disabilities in reading, and students with hearing impairments). We will do this by developing and studying computer-based interactive item prototypes, and by considering when these kinds of items are comparable to traditional item approaches.
The dynamic items will use the computer’s capabilities to replace large amounts of language by using animation and interactive techniques to present items, and allowing students to demonstrate their skills by interacting with stimuli, assembling, modeling, and drawing. Some of the cognitively complex interactive items will also use programmed algorithms to present sequenced items where students’ responses to a first set of questions condition how they will move through the item to a common final screen. Comparability of the interactive items with language-intensive traditional items will be investigated by studying how the students with language challenges and native English speakers with no IEPs will perform on pairs of traditional and interactive items which measure the same target content at the same grain size.
Because of the complex comparability issues that arise when different kinds of items, forms and tests are used in a state’s academic testing system, the project will convene a cogntive panel to develop a defensible codification system that will define comparability arguments. This codification system will delineate the benefits and limits of different types of observations and explicate the kinds of evidence needed to defend common score inferences when the skills of different students are measured with different instruments, or when item types in the assessment system change over time. A comprehensive dissemination plan is also proposed.
Washington Office of Superintendent of Public Instruction
Evaluating the Validity of English Language Proficiency Assessments (EVEA)
Under Titles I and III of the No Child Left Behind Act of 2001, all states must establish English language development standards and English language proficiency (ELP) assessments that are aligned with these standards and yield scores for Title III accountability purposes. To date, all states have implemented these standards and assessments; some have done so as part of consortia and other have adopted existing assessments or developed independent assessments of their own. This project will create a consortium of several states that have developed their own ELP assessments to build a joint validity argument and design a series of studies that could address specific components of that argument. Each state will identify its own validity evaluation priorities and the consortium will determine a set of group priorities for instrument development and pilot studies to be conducted as part of this project. In addition to specific benefits for participating states, this project will yield an approach to validity evaluation of these ELP assessments as well as two or more instruments that would be available to all states after the project’s completion. This project has been designed to alleviate as much as possible the burden on state staff by holding only two in-person meetings during the 18-month project, developing an on-line project workspace to support networking and interactions among states and researchers on their own schedules, and providing each state with a dedicated research partner. Washington will serve as the lead state for the five-state consortium, which also includes Idaho, Montana, Oregon, and South Dakota. Other partners in this work include edCount, LLC, the National Center for the Improvement of Educational Assessment (NCIEA), the Department of Education at the University of California at Los Angeles, Synergy Enterprises, Incorporated, and the Pacific Institute for Research and Evaluation (PIRE).