NOTE: Conversion to ASCII may have changed formatting, removed fottnotes, affected a graphic chart. Use Word Perfect 6.1 version or hardcopy for proposal preparation. PROPOSAL REQUIREMENTS A. Organization and Content of Proposal A.1 Technical Proposal The Technical Proposal should be organized into the following sections: Table of Contents, Introduction, Statement of Work, Project Management, Proposed Staff Related Experience, and Related Corporate Experience. Specific requirements for each section are discussed in greater detail in the following paragraphs. A.1.1 Table of Contents The Table of Contents should provide an easy means to identify major points of discussion. A.1.2 Introduction The Introduction should demonstrate the offeror's understanding of the purposes, methodology, and products of the project, and should include a short summary of the offeror's qualifications and unique strengths related the work outlined in the RFP. A.1.3 Statement of Work The Statement of Work section should correspond to the Statement of Work section in this document. The offeror's Statement of Work expands upon each of the tasks and subtasks outlined in the corresponding section of the RFP and discusses procedural issues related to completing each task in according to timelines. The offeror's plan for carrying out each task, including the naming of staff members who will play a major role in completing the task, must be clearly stated. Offeror should address how they will accomplish the optional tasks. Any variations to the tasks and subtasks defined herein are to be accompanied by a rationale. If the variation affects the overall cost of the study, this should be indicated in the technical proposal and alternative designs should be separately costed in the cost proposal. Offerors are encouraged to propose alternatives that, if adopted, would improve upon the study or reduce the costs. However, for evaluation purposes, the offeror shall submit a proposal to perform the work as specified in this document. Offerors are to thoroughly address all tasks and subtasks listed in the Statement of Work section of this RFP. Specific instructions and additional information for some of the tasks are listed below. Task 1. Initiate Project 1. Submit general management and monthly progress reports The offeror should describe in detail and submit a detailed tentative master schedule for all activities to be performed during the life of the contract. Included on the file will be a task-by-task description of all events to be performed in the project, including starting dates, milestones, and completion dates of the various components of each task. The Department of Education is currently considering moving the administration of the NTR/NTM from spring to fall, in which case the first operational administration of the NTR/NTM would shift from spring 1999 to fall (October or November) 1999. The timing of the pilot and field tests would shift accordingly, so that the field test would move from spring 1998 to fall 1998, with the pilot occurring some time before the field test. The offeror is to write the proposal assuming a spring 1999 administration cycle; however, the offeror should propose an alternate schedule that would result from shifting from a spring to fall testing cycle. The offeror should also discuss any impact of the alternate schedule on procedures, staffing and/or cost. 2. Establish subject-specific advisory panel The offeror should submit a preliminary list of potential advisory panel members with justifications. The offeror should identify the institutional affiliation of each potential panel member, along with a brief biographical sketch highlighting expertise and prior work related to the substantive and methodological needs of this endeavor. The proposal should include a draft agenda for the first advisory panel meeting. The offeror should recommend a schedule of panel meetings in keeping with the overall plans presented in its proposal. It is preferable that the panels meet in Washington DC, unless the offeror provides reasons to do otherwise. 3. Design pilot and field tests The offeror should describe in detail, a design for the pilot test and a design for the field test to result in the necessary forms for the 1999 operational administration of the NTM/NTR. Each design proposal should include the anticipated number of items and test forms, and the sampling plan. The contractor should describe in detail the possibility for opersampling in the field test for race/ethnicity, students with disabilites (including the different types of disabilities and/or different types of a accomodations), and LEP's. The offeror should describe in detail an efficient and non-intrusive method for collecting student background information to enable the DIF analyses, and the degree to which such analyses would be feasible given sample sizes. This should describe parameters for Department use. Task 2 Develop 1999 Items Write test items The offeror should describe in detail the development, security, and maintenance of item test bank(s). The offeror should demonstrate that items will be developed by individuals who possess the required level of content knowledge (mathematics for the NTM and reading for the NTR) and testing expertise. The process for the content advisory panel review of all items should also be described. The offeror should also to address methods for accomplishing this task in the short time frame. Conduct Cognitive Labs The offeror should describe in detail cost-effective means to identify the cognitive properties of test items using think-aloud cognitive lab pilots. This section should describe the sample of students, the cognitive lab procedures that will be used, qualifications of staff who will conduct the labs and interpret the findings, how the information will be used for item development, and the offeror's prior experience conducting think-aloud cognitive lab sessions. Develop Scoring Criteria The offeror should recommend and provide a detailed rationale for the type of scoring rubrics to be used for the constructed response items (e.g., holistic vs. analytic, general vs. item-specific). This section should include a description of how the rubrics will be developed, and other necessary scoring materials that will be needed to accompany the rubrics (e.g., anchor papers, training papers). The offeror should also describe procedures for developing, field testing, and revising scoring criteria. Develop Appropriate Accommodations for Students with Disabilities and Limited English Proficiency It is important that the NTM/NTR be maximally inclusive of all student populations. Methods for instructing schools to include all student populations in the pilot test should be discussed in detail in this section. The offeror should propose in detail methods for determining appropriate accommodations for students with disabilities and LEP students for the NTR/NTM, allowing for national input. The offeror should anticipate accommodations that will be offered for the operational test, and discuss in detail procedures for providing those accommodations for the pilot test. The contractor shall not be required to produce non-standard test materials for the pilot test (i.e., Braille & large print for the NTM/NTR; audio & bilingual Spanish/English for the NTM) though these non-standard test materials will be required for the field test and operational test. Prepare Informational Materials The offeror should describe in detail a design for the informational materials, including a description of the appearance and content. It should be assumed that the Government Printing Office (GPO) will do the printing; the contractor will be responsible for distributing the printed materials to the education agencies. Establish School Participation This section should describe in detail the number of schools that will be needed for the pilot test and specific plans for securing the cooperation of the necessary number of schools within the time frame of a few months. The offeror should propose a detailed plan for securing the cooperation of local test coordinators to make local arrangements for pilot testing. Pilot Test Booklets The contractor should describe in detail the features of the scannable, consumable pilot test booklets. Printing of the pilot test booklets will be done by GPO; the contractor will be responsible for distributing the pilot test booklets to participating education agencies. Administration Procedures and Training The offeror should describe in detail the training procedures for pilot test administrators and the logistics of how that training will be accomplished. The contractor will be responsible for developing the administration training manual; GPO will be responsible for printing; and the contractor for distributing. Implement Receipt Control System The offeror should describe in detail a receipt control system at its facilities. The offeror should describe in detail the events and activities that will be monitored by this system and describe how the system's files will be maintained. Evaluate and Revise Test Items In this section, the offeror should describe in detail the appropriate item analyses to evaluate the quality and appropriateness of pilot test items. The offeror is encouraged to propose t additional analyses to those listed in this document that would be useful for identifying items with the best psychometric properties. Task 3 1999 Field Test Preparation Appropriate Test Accommodations for Students with Disabilities and Limited English Proficiency The offeror should describe in detail procedures for ensuring that appropriate accommodations are available during the field test. The offeror should address in detail the procedures for preparing the Braille and large print versions of all of the field test forms for the NTM/NTR. In addition, for the NTM, procedures for creating the bilingual (English/Spanish) and audio cassette versions of all of the field test forms should be described in detail. Task 4- Conduct 1999 Field Test Operations Establish Reporting Plans and Metric The offeror should describe in detail and suggest one or more reporting metrics for the NTR/NTM that are intuitively understandable to the public, along with justification for using the particular metric. The goal is to avoid difficult-to-interpret scale scores. The offeror should discuss in detail a process for conducting parent focus groups (including a description of the parent sample, staff who will conduct the focus groups, background experience of the corporation in conducting focus groups) and how the focus group findings will be used to help determine the best reporting metric. The contractor should discuss in detail and recommend in the proposal an appropriate plan for reporting field test results back to the individual students who participated. This recommendation should include a detailed discussion of the kinds of scores/information that could be reported back to students/parents, the time frame for producing those reports, and any limitations associated with interpretation of the individual field test scores. Score Field Test Booklets The offeror should describe in details the procedures for machine-scoring the multiple choice items. The offeror should also to describe, in detail, the procedures for scoring the constructed response items, including methods for ensuring high inter-rater reliability. This section should include a detailed discussion of how scorers will be trained and monitored to ensure reliability. Safeguard Test Instruments and Records The offeror should describe in details the facilities and procedures for ensuring the security of the returned test booklets. Prepare and Document Data Files The offeror should describe in detail the procedures for creating and maintaining the field test data files. The offeror should also to describe the system of statistical quality control. Analyze Field Test Data The offeror should propose all of the item/test statistics that will be produced, including but not limited to those specified in this SOW. The offeror is encouraged to expand on the initial suggestions for psychometric procedures specified under Task 4.6, and to recommend alternative methodologies. Coordinate with Linking Contractor The offeror should suggest in detail procedures for coordinating efforts with the linking contractor in order to provide the necessary linking data. Equate Test Forms The offeror should propose in detail the psychometric methods for scaling and equating equivalent test forms. The offeror should describe in detail the design, procedures, and analyses that will be required to maintain a stable scale across years. The offeror should propose a detailed plan for building the necessary links for equating 1999, 2000, and 2001 test booklets, and propose a system for checking on scale drift. Task 5 - 1999 Final Operations For each of the subtasks under Task 5, the offeror should describe in detail procedures for creating products for the operational test in camera ready copy to be provided to the licensing contractor. The licensing contractor shall coordinate the printing, distributing, scoring, analyzing, reporting and administering of the operational NTR/NTM. For each of the subtasks under Task 5, the offeror should describe in detail the procedures for determining the content and design of the following documents for the operational test: informational packet (Task 5.1); sample test (Task 5.2); operational test booklets (5.3); training manual for test administrators (Task 5.4); scoring training guide, rubrics and calibration papers (Task 5.5); test reporting guidelines (Task 5.6); test utilization guide (Task 5.7); and test accommodations guidelines (Task 5.7). Several of these subtasks are described as requiring a consensus process, in which case the offeror should describe procedures for achieving this. The offeror should also to discuss in detail procedures for coordinating with the licensing contractor. Tasks 6-14 The offeror should propose a detailed plan, detailed time line, and detailed estimated costs for selecting and/or developing enough items for development of operational forms after spring, 1999. The plan should specify in detail how the offeror proposes to pilot new items and equate subsequent forms. This plan for additional item selection and/or development shall meet the same requirements for selection and/or development as the initial items. Task 15 - Provide Research and Validation Support The offeror should propose in detail an ongoing research agenda for the life of the contract, so that two studies are completed each year. The first two research studies to be conducted the first year of the contract are to address: 1) comparability of tests administered under non-standard conditions, and 2) validation of test uses. The offeror should propose in detail a specific study or several alternative studies to address each of these issues that could feasibly be conducted and completed during the first year of the contract. The offeror should also to provide in the proposal an detailed outline of proposed studies to be conducted each subsequent year of the contract. The offeror is encouraged to propose additional or alternative research topics to those listed under Task 15. In proposing this research agenda, the offeror should consider which issues are most crucial for directing test development activities. Justification for the proposed studies should be provided in detail.. A.1.4 Proposed staff related experiences The Proposed Staff Related Experience section should contain a summary for each proposed key staff member that indicates how his/her prior work experiences and educational background relates to his/her proposed role on the NTR/NTM. Offerors are encouraged to submit current vitae for each staff member proposed for the project. The vitae should document only directly related experience, educational background, publications, and professional activities. Each vitae must list (1) each degree earned (along with the date when each was confirmed, the confirming institution, and the major field of study); 2) no more than four prior/current work or project experiences; and 3) no more than 10 publications and professional presentations. Offerors should not submitting general vitae or vitae that do not clearly identify the information identified above. For proposed staff that are not currently employed by the corporation, a letter of commitment should be included. Any changes or substitutions of the key personnel will require advance approval by the Contracting Officer in consultation with the OERI COTR. A.1.5 Related corporate experience The Related Corporate Experience section should describe in detail the offeror's capability and experience in all areas related to conducting a large-scale, national testing program, including expertise in: creating test items and forms, designing and conducting national pilot and field tests, developing and using computer technology and software systems in support of large-scale data collection, scoring, designing test booklets, content area expertise (mathematics for the NTM, reading for the NTR), convening advisory groups, and creating informational materials. Offerors should include detailed evidence of its prior and ongoing corporate projects are demonstrative of its expertise in the tasks required for this project. One page abstracts of related work should be included which clearly identify the relevance to the NTR/NTM project. The abstracts should include the name, current affiliation, and current telephone number of the sponsor's COTR. These COTRs may be asked to report their experience on the project (e.g., responsiveness, flexibility, problems, product quality.) The corporate experience section should contains a description of the approach that the offeror will use to provide corporate oversight for the NTR/NTM. This section should include detailed information on how project expenditures will be monitored and controlled. The specific steps that the offeror will take to insure the quality of the work and products of its staff, and to identify problems in performance should be clearly defined. A. Proposal Evaluation Criteria The following criteria and point values will be used to evaluate proposals submitted in response to this RFP: (Please note that the total point distribution totals 500.) 1. Quality and credibility of the General Approach and Statement of Work (225 points) Demonstrates an understanding of the psychometric, logistical, and security challenges specific to this project and proposes creative, innovative, and affordable solutions (100 points) Approach to completing each task, and the tasks as a whole, is complete and feasible within the proposed time frames (75 points) Demonstrates a familiarity with issues in psychometric theory related to education assessment (including item response theory) (50 points) 2. Qualifications, technical and managerial, of the proposed project director (75 points) Expertise in the design and conduct of large-scale educational assessment (20 points) Expertise in the development of tests of reading for the NTR or mathematics skills for the NTM of school children (20 points) Demonstrated expertise supervising the work of many professionals and monitoring the overall quality of their work on many tasks simultaneously (20 points) Demonstrated ability to adhere to project schedules, keep costs within budget, and manage subcontractors (15 points) 3. Soundness of the management plan and task/time commitments of staff (75 points) Proposed project organization demonstrates an efficient use of staff with clearly defined lines of authority and responsibilities (25 points) Demonstrates effective control of tasks and resource allocations that will ensure the timely completion of tasks/deliverables within the proposed budget (25 points) Proposes a sound and feasible approach to controlling the quality of project work and products (25 points) 4. Qualifications of the project staff (75 points) Expertise of staff matches proposed assignments (25 points) Collectively possess the technical expertise, knowledge, and experience in all areas related to the study, including: field operations, field data collection, field staff data collection training, data processing and computer applications research methodology, statistics and sampling, child assessment, reading or mathematics test development (congruent with the test for which proposal is submitted), and psychometrics (25 points) Subject matter expertise in reading that is inclusive of the range of instructional philosophies for the NTR, or subject matter expertise in mathematics that reflects an understanding of the NCTM standards for the NTM (25 points) 5. Corporate expertise, capability, and oversight (50 points) Demonstrated corporate expertise in the following areas: conduct of large-scale national studies; conduct of studies of school children; design and selection of national samples; development and use of computer technology and software systems in support of large-scale data collection; design and development of large data bases; design and use of systems for processing, documenting and maintaining large scale data bases; developing weights for complex sample designs; design of survey instruments; and the development of assessment batteries for measuring reading or mathematics ability in children (25 points) Evidence of strong corporate commitment to this project as evidenced by placement within the corporate structure, allocation and availability of appropriate corporate resources as needed (25 points),