Attachment A U.S. Department of Education Planning and Evaluation Service "National Evaluation of the Even Start Family Literacy Program" Statement of Work Contents 1. Introduction and Background. . . . . . . . . . . . . . . . 1 a. Purpose . . . . . . . . . . . . . . . . . . . . . . . 1 b. Legislative Requirement for Evaluation. . . . . . . . 1 c. Performance Indicators. . . . . . . . . . . . . . . . .1 d. The Even Start Program. . . . . . . . . . . . . . . . 2 e. Prior Data Collection and Information Systems . . . . 4 2. Objectives . . . . . . . . . . . . . . . . . . . . . . . . 6 3. Requirements . . . . . . . . . . . . . . . . . . . . . . . 10 A. Tasks to Be Performed . . . . . . . . . . . . . . . . 10 PART A: PROJECT MANAGEMENT Section 1: Project Management . . . . . . . . . . . . . . 10 Task 1. Update baseline management plan . . . . . . . . . 10 Task 2. Provide technical assistance on the evaluation through telephone calls and written communications. 11 Task 3. Establish Performance Measurement System. . . 11 Section 2: Project Design and Improvement12 Task 4. Develop overview booklet on this project. . . . . 12 Task 5. Establish an expert work group . . . . . . . . . . 12 PART B: EVEN START PERFORMANCE INFORMATION REPORTING SYSTEM (ESPIRS) Section 3: Revise and Implement Universe Data Entry System (ESPIRS) for States and Projects. . . . . . . . . . . . . . . . . . . 15 Task 6. Review data collection forms and evaluation design for clarification 15 Task 7. Adapt computer system for local project and State coordinator data entry of universe data . . . . . . . . . . . . . . . . . . 16 Task 8. Update the data entry system user guide and prepare a State version 17 Section 4: Implement Universe Data Entry System (ESPIRS) . 17 Task 9. Conduct an annual evaluation conference and training program for local Even Start project directors and evaluators and State Even Start coordinators 17 Task 10. Receive and enter State coordinator data to develop and maintain tracking system of projects21 . . . . . Task 11. Provide projects with new grantees package . 22 Task 12. Receive, edit, and enter July data submitted by local projects 23 PART C: EVALUATION OF EVEN START PROGRAM Section 5: Develop and Implement Experimental Design Study Component 24 Task 13. Develop materials to implement experimental design study on program effectiveness . . . . . . . . . . . . . . . . . . 24 Task 14. Select Even Start projects for experimental design study and develop data collection schedule . . . . . . . . . . . . . . . 25 Section 6: Implement National Evaluation Framework and Prepare Evaluation Reports 26 Task 15. Update data analysis and evaluation plan . . 26 Task 16. Prepare interim reports and summary database . . 27 Task 17. Prepare final report and summary database. . . . 28 Task 18. Develop and produce individual project profiles 28 Task 19. Produce Special Analysis of Migrant Projects . . 29 Task 20. Produce Special Analysis of Tribal Projects . . . 30 PART D: UPDATE NATIONAL EVALUATION DESIGN Section 7: Update National Evaluation Design 30 Task 21. Review and update Even Start program performance indicators 30 Task 22. Recommend changes to the Even Start national evaluation design 31 PART E: ADAPT NATIONAL EVALUATION DESIGN (OPTIONS) Section 8: Prepare Revised National Evaluation Design (options) 31 Task 23. Edit and test forms (option) 31 Task 24. Prepare and support IMCD/OMB forms clearance request (option)32 Task 25. Update the data collection forms, computerized system and user's manual (option) . . . . . . . . . . . . . . . . . 32 B. Reporting Requirements. . . . . . . . . . . . . . . . . 33 C. Timeline. . . . . . . . . . . . . . . . . . . . . . . . 34 D. Deliverables. . . . . . . . . . . . . . . . . . . . . . 44 1. Introduction and Background a. Purpose The purpose of this four-year project is to evaluate the Even Start Family Literacy Program at the national level during the school years 1997-98 though 2000-01. The national evaluation shall involve collection of evaluation and performance data and analysis of that data from all local Even Start projects--estimated to number 576 projects in 1996-97 and 635 projects in 1997-98. In addition to the collection of a set of common data for the universe of projects, this evaluation will also include an experimental design data collection on a smaller number of projects to assess program effectiveness. The evaluation will culminate in three annual interim reports, a final report, individual project and State profiles, and special analyses of Migrant Education projects and tribal projects. b. Legislative Requirement for Evaluation The Even Start legislation (Title 1, Part B, Section 1209 of the Elementary and Secondary Education Act) requires an independent national evaluation of the projects funded under Even Start. This section reads as follows: "SEC. 1209. EVALUATION. "From funds reserved under section 1202(b)(1), the Secretary shall provide for an independent evaluation of programs assisted under this part (1) to determine the performance and effectiveness of programs assisted under this part; and (2) to identify effective Even Start programs assisted under this part that can be duplicated and used in providing technical assistance to Federal, State, and local programs. c. Performance Indicators Title I, Part B of ESEA requires the Department to report to Congress on the effectiveness of programs assisted by the Even Start Family Literacy Program. In addition to the legislative requirement to evaluate Even Start, the Government Performance and Results Act (GPRA) of 1993 (P.L. 103-62) established broad requirements to hold federal agencies accountable for program results, improve program management and effectiveness, and provide objective information for Congressional decision making. Federal agencies are required to begin submitting to the Office of Management and Budget (OMB) strategic plans for their program activities and annual performance plans with performance indicators to measure outputs, service levels, and outcomes. Agencies are required to submit annual reports to Congress on actual performance achieved and explain why any of the goals were not met. OMB Circular A-11 (1994), addresses GPRA requirements: Agencies are encouraged to begin the work necessary to include more program performance indicators and performance goals in the budget decision-making process and budget document. In this regard, the focus should be toward developing quantitative and qualitative measures of outputs and outcomes...It is recognized that implementation of GPRA and the development of output and outcome measures is a difficult and complex process. The efforts started this year are expected to be a first step toward development of policies, procedures, and practices which will evolve over the course of the GPRA timetable...(p. 25). Narrative statements [in budget requests] describe the performance goal, outputs, and outcomes...These statements should discuss how the budget year request supports these performance goals, outputs, and outcomes while addressing significant increases or decreases in performance or financing...The statements should discuss both the short- range and long-range performance goals of the program, and include data on significant performance indicators where appropriate...Generally, performance indicators which focus on outputs or outcomes should be used instead of workload and other process measures. d. The Even Start Program Authorization. Even Start was authorized by the Elementary and Secondary Education Act of 1965 as amended by the Hawkins-Stafford Elementary and Secondary School Improvement Amendments of 1988, Part B of Chapter 1 of Title I (P.L. 100-297). The Even Start legislation was amended in July 1991, when Congress passed the National Literacy Act (P.L. 102-73). In 1994, the Even Start program was reauthorized by the Improving America's Schools Act as Part B of Title I of the Elementary and Secondary Education Act. According to the law, the goal of Even Start is to: "...help break the cycle of poverty and illiteracy by improving the educational opportunities of the Nation's low-income families by integrating early childhood education, adult literacy or adult basic education, and parenting education into a unified family literacy program....The program shall (1) be implemented through cooperative projects that build on existing community resources to create a new range of services; (2) promote achievement of the National Education Goals; and (3) assist children and adults from low-income families to achieve to challenging State content standards and challenging State student performance standards." (P.L. 100-382, Sec. 1201). Services. The Even Start program represents an innovative combination of programs for adult literacy or basic education, parent education and early childhood education. This federal initiative offers promise of addressing the literacy crisis in the nation through an integrative approach to adult and early childhood education. Focusing on parents and children as a unit, Even Start projects have three interrelated goals: þ To help parents become full partners in the education of their children; þ To assist children in reaching their full potential as learners; and þ To provide literacy training for their parents. To be eligible for Even Start under the reauthorized law, a family must have an adult who is either eligible for adult basic education programs under the Adult Education Act or who is within the State's compulsory school attendance age range, and must also have a child less than eight years of age. Even Start is family-focused rather than parent- or child-focused. That is, Even Start projects must provide participating families with an integrated program of early childhood education, adult literacy or basic skills training, and parenting training. The theory is that these components build on each other and therefore families need to receive all three services, not just one or two, in order to effect lasting change and improve children's school success. Services can be grouped into two areas: (1) core educational services; and (2) support services. The three core services are: þ Adult education: high-quality intensive instructional programs for adults to promote adult literacy including adult basic education (ABE), adult secondary education (ASE), English-as-a-second-language ( ESL), and preparation for the General Education Development (GED) certificate; þ Parent education: high-quality intensive instructional programs to empower parents to support the educational growth of their children; and þ Early childhood education: developmentally appropriate educational services for children, and programs designed to prepare children for success in regular school programs. Even Start projects are required to provide some core services to parents and children jointly and to provide some home-based services. In addition to core services, Even Start projects typically provide a range of support services. Examples include transportation, custodial child care, health care, meals, nutrition assistance, mental health referrals, referrals for employment, advocacy assistance with governmental agencies, counseling, child protective services, referrals for screening or treatment for chemical dependency, referrals for services to battered women, special care for a handicapped family member, or translators. The Even Start legislation requires that support services be obtained from existing providers whenever possible, to ensure that Even Start projects avoid duplication of services. Finally, Even Start projects often provide special events for participants. These may be one- time events such as a pot-luck supper, or they may include occasional activities or demonstrations on subjects of interest to the participating families. e. Prior Data Collection and Information Systems National Evaluation Information System (1989-90 to 1992-93) This project (contract number LC 90062001) involved collecting unit record data on participating families (parents and children). For the maintenance of the first four years of the information system, local Even Start project directors, staff, and/or locally hired evaluators collected the data, filled out data collection forms (paper forms in the first two years and a computerized system for the last two years), and submitted them to a contractor for the National Evaluation Information System (NEIS) (Abt Associates, Inc., with RMC Research Corporation as subcontractor). The data were submitted twice a year, in December and in June. The contractor edited and entered the data, analyzed them, and prepared periodic interim reports and a final report. (see Government-provided materials list in appendix). The focus of the NEIS was the question: "Does Even Start work?" This study collected project implementation data from all projects plus limited outcome data from the first two cohorts (1989-90 and 1990-91) of projects. In addition, the study used experimental design (random assignment) in some in-depth study sites. This study did not definitively answer the question of Even Start's effectiveness, however. Even Start Information System (1993-94 to 1996-97) The Even Start Information System (ESIS, contract number EA 94068001) was the system in place during the second four-year cycle of evaluation. Like the NEIS, the ESIS was designed to collect a common set of data from each Even Start project and from all Even Start participants. The contractor for this four-year project was Fu Associates, Ltd. with Abt Associates, Inc. as subcontractor. The data were submitted once a year, in July. The contractor edited and entered the data, analyzed them, and prepared annual interim reports and a final report. (see Government-provided materials list in appendix). This study did not use experimental design. The focus of this study was on identifying promising and effective practices. The study's focus question was: "What works and under what circumstances?" The universe study of all projects focused on implementation data. This study's in-depth component was a sample survey of 60 projects that collected limited outcome data. Unfortunately, the number of projects in the sample study dwindled throughout the contract. Even Start Projects and Participants Fiscal Year of Funds Federal Funding School Year of Operation Number of Projects Number of Families Number of Children Number of Adults Total Participants FY 1989 $14,820,000 1989-90 76 2,450 2,760 2,900 5,660 FY 1990 $24,201,000 1990-91 122 6,600 8,580 7,560 16,140 FY 1991 $49,770,000 1991-92 239 14,900 20,890 17,920 38,810 FY 1992 $70,000,000 1992-93 340 16,518 22,429 18,586 41,015 FY 1993 $89,123,000 1993-94 439 29,400 estimated 38,220 estimated 28,224 estimated 66,444 estimated FY 1994 $91,373,000 1994-95 476 28,500 37,619 27,227 64,846 FY 1995 $102,024,000 1995-96 560 30,001 45,103 34,440 79,543 FY 1996 estimated $101,997,000 1996-97 635 36,195 50,673 39,815 90,488 FY 1997 estimated $101,997,000 1997-98 696 39,672 55,541 43,639 99,180 Sources: ED program data for funding and number of local grants; evaluation contractors (Abt Associates, Inc., Pelavin Associates, Inc., and Fu Associates, Ltd.) for participants. Note: As no family data were collected in FY 1993, participant data for that year are estimated based on established patterns. All participant and project data for FY 1996-1997 are estimated based on prior patterns. 2. Objectives The National Evaluation of the Even Start Family Literacy program shall: a. Enable ED to fulfill the requirement in the Even Start legislation for evaluation of the effectiveness of the Even Start program. b. Enable ED to respond to the requirements of the Government Performance and Results Act (GPRA) to report out annually on indicators of program performance. c. Enable ED to meet the evaluation requirements of the Education Department's General Administrative Regulations (EDGAR) which stipulate that all grantees conduct annual evaluations. d. Provide performance data useful to local, State, and Federal administrators, legislators, and policy-makers for program improvement and policy development. e. Improve on performance indicators that can be used to review the program's implementation and impact. f. Add to the knowledge base on the effects of family literacy programs by investigating the relationships between program processes and outcomes. g. Provide evaluation data that can serve as a base on which additional evaluation studies that might be funded separately could build. h. Track the progress of the Even Start program through linkages with the previous two national evaluations. The focus of this project shall be on the evaluation of Even Start's effectiveness and the identification of effective implementation rather than the maintenance of an information system. The key to this project is analyzing and evaluating data from the information system as well as other sources to improve the program's effectiveness. The methodology of this evaluation shall be rigorous and address the following research questions: 1. Who is served by the program and what services do they receive? Is the program reaching the appropriate target population? 2. How is the federal funding spent on the program? 3. How well does Even Start work? How many of the projects are well implemented? 4. What are the outcomes of Even Start? Do participants perform better on key measures than similar persons who do not participate? 5. What are effective practices and programs? More specifically, the research questions will encompass the following subquestions: 1. Who is served by the program and what services do they receive? Is the program reaching the appropriate target population?  What are the characteristics of family units participating in Even Start?  What are the background characteristics of adults who participate in Even Start (e.g., gender, age, race/ethnicity, employment status, annual income, primary language)?  What is the educational history of adults who participate in Even Start? What percentage of adults enter with a high school diploma or a GED and what are their characteristics?  What are the background characteristics of children who participate in Even Start (e.g., age, race/ethnicity, presence of handicapping condition, primary language)?  What are the program participation histories of children who participate in Even Start?  How do the characteristics of Even Start adults compare with the characteristics of participants in adult education programs nationally?  How do the characteristics of Even Start children compare with the characteristics of participants in Title I and Head Start programs? To what extent is there overlap in program participation?  How do background and literacy characteristics of adults and children vary across projects? Is it possible to identify sets of projects that appear to be using very different targeting criteria? Do some projects appear to target "high need" populations while others target "low need" populations?  How many families participate in Even Start? In each of Even Start's core services?  What percentage of families participates in each of Even Start's three core services? In all core services?  How long do families participate in Even Start (how many months are services received)?  To what extent do Even Start families participate in the services offered? How many hours do participants participate in each service?  What is the geographic and urban/rural distribution of Even Start projects? 2. How is the federal funding spent on the program?  What is the federal cost for Even Start projects?  What services are purchased with Even Start funds?  What is the local share ? 3. How well does Even Start work? How many of the projects are well implemented?  How are families recruited into Even Start? How do projects target recruitment efforts?  What local evaluation activities are being undertaken by Even Start projects?  To what extent do projects respond to the findings of their local evaluations?  What is the nature of services in early childhood education, adult education, and parenting education offered by the project? What activities does the project offer for parents and children together? What is the nature of home visits offered by the project? What types of support services are provided to participants? Are the instructional techniques that projects use research based?  What types of collaborations exist between Even Start and other agencies? What types of agencies are collaborators?  What barriers exist to successful program implementation?  Are the differences in program implementation associated with factors such as project cohort, length of time that the project has had a grant, area of the country?  Are local projects able to obtain the technical assistance they need from the States?  What are the technical assistance needs of local projects?  What are the characteristics and variations of State operations and support for local projects? 4. What are the outcomes of Even Start? Do participants perform better on key measures than similar persons who do not participate?  What gains are observed for Even Start children on measures of school readiness and vocabulary?  What gains are observed for Even Start adults on measures of functional literacy, GED attainment, employment status, annual income, parent expectations, and parenting skills?  With what literacy skills do Even Start participants leave the program? Do Even Start children enter school ready to learn?  With what degree of confidence can any observed gains be attributed to participation in Even Start rather than to other factors? Do control group families have similar gains? What can be said of Even Start participants' progress versus progress made by participants in other programs? 5. What are effective practices and programs?  What participant characteristics (e.g., educational level of adults at entry, age of parent, size of family) are associated with positive outcomes?  What project characteristics (e.g., urban/rural location, degree of implementation, year- round versus school-year operation schedule, high federal cost per family versus low federal cost per family) are associated with positive outcomes?  What staff characteristics are associated with positive outcomes?  What service characteristics (e.g., amount of core service received, percentage of core services delivered at home versus in a center, percentage of core services delivered to parents and children together, adherence to a model) are associated with positive outcomes?  What supports from external sources (e.g., state assistance, technical assistance centers) are associated with positive outcomes?  Is it possible to define subgroups of families based on need criteria (e.g., income, entry education level), and determine whether outcomes vary across the subgroups? Data Collection Data shall be collected from all projects and State coordinators through the Even Start Performance Information Reporting System that includes data items such as:  Characteristics of participants  Project implementation information (curriculum and instruction, coordination among project services, recruitment and retention strategies)  Project goals and objectives  Hours of services offered by projects  The extent to which participants participated in the services offered  Progress indicators and outcomes for participating families Information shall also be collected through an experimental design study to be designed in year one of this contract. The contractor shall develop a data collection for this component that shall include information on the progress and services received by a control group. Evaluation Components The plan described in this statement of work will be able to address the above questions in ways that the previous information system could not. The use of new methodologies, including the individual analysis and comparisons involved in individual project and State profiles, an experimental design component, and a new instrument for data collection from the universe of projects (Even Start Performance Information Reporting System component) shall answer the research questions more efficiently. The contractor shall develop a design for the experimental design component that is able to address research question number 3, which was essentially ignored in the Even Start Information System of years 1994-1997. The National Evaluation of the Even Start Family Literacy Program shall consist of five components: (1) Universe study component to be called the Even Start Performance Information Reporting System (ESPIRS), which shall collect data from all projects; (2) The Experimental Design Study component to collect more detailed data from a selected subset of projects; (3) Individual Project and State profiles based on data collected in the ESPIRS (4) Special Analysis of ESPIRS data on Migrant Education Even Start projects; and (5) Special Analysis of ESPIRS data on Even Start projects for Indian tribes and tribal organizations. Data from the ESPIRS on all projects shall be analyzed in combination with data collected from the experimental design study (during the years of data collection for that particular component). The analysis results from the ESPIRS and experimental design component shall be reported in annual interim reports that shall address the research questions discussed above using appropriate methodology developed over the course of the project. The analysis results from the last year of this project shall be reported and also the whole four years of the project shall be summarized (and compared with information from the first two information systems where possible) in a final report. In addition to these four reports, data collected in the ESPIRS and (if applicable) the experimental design component shall be reported out in individual project profiles that show various indicators for each particular project in comparison to that project's state and also the national data. Finally, data from the ESPIRS shall be pulled for the Migrant Education projects and the projects for Indian tribes and tribal organizations and reported in two special analyses that compare each subgroups data to the national data. 3. Requirements A. Tasks to Be Performed PART A: PROJECT MANAGEMENT Section 1: Project Management Task 1. Update baseline management plan. 1.1 Meeting with the Department of Education. The contractor's project director and key project staff shall meet with the Department of Education's (ED) Contracting Officer's Technical Representative (COTR), Contract Specialist, and other appropriate ED staff and fax a memo to the COTR summarizing the meeting within five working days after the award of this contract to discuss details regarding the tasks outlined in this statement of work. 1.2 Revise Baseline Management Plan. The contractor shall revise the proposed Baseline Management Plan submitted with the proposal for accomplishment of all tasks, subtasks, key events, milestones, deliverables, and delivery of their associated products. The Revised Baseline Management Plan shall be submitted to Department of Education Contracting Officer's Technical Representative (COTR) within 2 weeks after the award of this contract. 1.3 Approval of Revised Baseline Management Plan. COTR will provide notification of approval or required modification within 1 week after receipt. The contractor shall submit the final plan to the COTR within 4 weeks after the award of this contract. 1.4 Provide quarterly management updates. The project director and one additional key staff member shall meet with the COTR and other ED staff (designated by the COTR) at the U.S. Department of Education in Washington, D.C. once each quarter over the duration of the contract. The purpose of the meetings is to brief ED on the progress of the study and discuss issues as they arise. If the contractor is not located in the D.C. area, the trips shall be combined with other travel where possible. Task 2. Provide technical assistance on the evaluation to State coordinators and local projects through telephone calls and written communications. Local projects and State coordinators will probably call regarding the status of plans for the conference and, after the conference, about the data collection forms and procedures. Contractor staff shall answer questions on use of the instruments or software, as needed, usually by telephone. The contractor shall record the caller and purpose of telephone calls and any other correspondence and include a summary report on the number and main purpose of requests for assistance in the monthly report to the COTR due within the first ten days of each month of this contract. The contractor shall answer technical assistance requests starting immediately after the date of award of this contract. Task 3. Establish Performance Measurement System. The Contractor shall develop a performance measurement system that shall: (1) identify problem areas by order of importance; (2) identify anticipated schedule slippage and cost overruns; and (3) provide a means of determining where the program manager can withdraw resources to assist more critical tasks. The Contractor shall submit reports from the performance measurement system to the COTR within the first 10 days of every calendar month. Each report shall include:  A summary of major activities and accomplishments during the reporting period.  A description and discussion of any problems encountered in the past month or expected in the future, along with a description of the plan for resolving problems that have occurred and for preventing future problems.  Information for each task on significant findings and events, problems encountered, and problems resolved.  A brief description of activities planned for the next month.  A statement of the extent to which the project is on schedule.  A discussion of exceptions to the management plan, either already encountered or expected. If there are no exceptions, the report shall so state.  The number of hours expended by each staff member, by name and by task, for the reporting period and cumulatively for the entire contract period, and a projection of staff assignments for the next month. Section 2: Project Design and Improvement Task 4. Develop overview booklet on this project. The contractor shall prepare and submit to the COTR a booklet of no more than 25 pages on the national evaluation design. In the booklet, the contractor shall describe the national evaluation plan and research questions and how the universe data system or ESPIRS and the experimental design component shall be used to answer those questions and evaluate Even Start. The contractor shall also describe the project profiles and special analyses of Migrant Education and Tribal projects in the booklet. The purpose of the booklet is to serve as a resource for the COTR and project director when asked about the project. The contractor shall submit the draft booklet to the COTR 5 weeks after the award of this contract. The COTR will review within 2 weeks. The contractor shall provide 5 copies of the final overview booklet to the COTR within 9 weeks of the award of this contract. Task 5. Establish an expert work group. 5.1Selection of work group members. The Contractor shall propose twelve to fifteen work group members who are researchers in the field of family literacy (or one or more of its components), local or state practitioners, or evaluators of local Even Start projects to attend four one-day meetings in Washington D.C. The contractor shall provide the list of proposed members, along with suggestions for how to divide the group into technical and stakeholder subgroups, to the COTR within 2 weeks of the award of this contract. The COTR will provide comments within 2 weeks of receipt. The Contractor shall coordinate the selection process by performing such tasks as making phone calls and obtaining dates of availability. The Contractor shall make all travel and honoraria arrangements for group members. The Contractor shall deliver to the COTR a list of members no later than 6 weeks after the award of this contract. 5.2 Prepare materials for and conduct Fall 1997 work group meeting. The Contractor shall prepare a presentation and agenda for the Fall 1997 meeting including the ESPIRS design, data and findings of the previous two evaluations, data analysis plans and a draft report outline for the 1998 Interim Report. [As discussed in Task 13, the Contractor shall also prepare an outline of a design for the experimental design study component of the national evaluation]. The Contractor shall deliver the draft presentation materials to the COTR no later than 7 weeks after the award of this contract. COTR will provide comments no later than 2 weeks after receipt. The Contractor shall send the approved materials to all work group members and to the COTR within 11 weeks of the award of this contract. The Contractor shall arrange and conduct the 1997 expert work group meeting no later than 12 weeks after the award of this contract. The Contractor shall present the prepared materials and facilitate group discussion. The Contractor shall deliver a summary of the meeting to the COTR no later than 3 weeks after the meeting. 5.3 Prepare materials for and conduct Winter 1998 work group meeting. The Contractor shall prepare a presentation and agenda for the Winter 1998 meeting including the design of the experimental study component, the draft OMB package for the data collection instrument and any updates for the group. The Contractor shall deliver the draft presentation materials to the COTR no later than 19 weeks after the award of this contract. COTR will provide comments no later than 2 weeks after receipt. The Contractor shall send the approved materials to all work group members and to the COTR within 23 weeks of the award of this contract. The Contractor shall arrange and conduct the Winter 1998 expert work group meeting no later than 24 weeks after the award of this contract. The Contractor shall present the prepared materials and facilitate group discussion. The Contractor shall deliver a summary of the meeting to the COTR no later than 3 weeks after the meeting. 5.4 Coordinate independent review of draft 1998 Interim Report. The Contractor shall send copies of the first draft of the 1998 Interim Report to the members of the work group for review no later than 60 weeks after the award of this contract. The Contractor shall deliver the comments from this review to the COTR no later than 5 weeks after the report is sent to reviewers. 5.5 Prepare materials for and conduct Fall 1998 work group meeting. The Contractor shall prepare a presentation and agenda for the Fall 1998 meeting including discussion of the draft 1998 Interim Report and any other updates. The Contractor shall deliver the draft presentation materials to the COTR no later than 59 weeks after the award of this contract. The COTR will provide comments no later than 2 weeks after receipt. The Contractor shall send the approved materials to all work group members and the COTR within 63 weeks of the award of this contract. The Contractor shall arrange and conduct the Fall 1998 expert work group meeting no later than 64 weeks after the award of this contract. The Contractor shall present the prepared materials and facilitate group discussion. The Contractor shall deliver a summary of the meeting to the COTR no later than 2 weeks after the meeting. 5.6 Prepare materials for and conduct Spring 1999 work group meeting. The Contractor shall prepare a presentation and agenda for the Spring 1999 meeting including the findings from the 1998 report and the outline and data analysis plans for the 1999 Interim Report. The Contractor shall deliver the draft presentation materials to the COTR no later than 84 weeks after the award of this contract. The COTR will provide comments no later than 2 weeks after receipt. The Contractor shall send the approved materials to all work group members and the COTR within 88 weeks of the award of this contract. The Contractor shall arrange and conduct the 1999 expert work group meeting no later than 89 weeks after the award of this contract. The Contractor shall present the prepared materials and facilitate group discussion. The Contractor shall deliver a summary of the meeting to the COTR no later than 2 weeks after the meeting. 5.7 Coordinate independent review of draft 1999 Interim Report. The Contractor shall send copies of the first draft of the 1999 Interim Report to the members of the work group for review no later than 114 weeks after the award of this contract The Contractor shall deliver the comments from this review to the COTR no later than 5 weeks after the report is sent to reviewers. 5.8 Prepare materials for and conduct Fall 1999 work group meeting. The Contractor shall prepare a presentation and agenda for the Fall 1999 meeting. The Contractor shall prepare a brief report describing the lessons learned at the time from working through this evaluation and recommended changes in the study design and instrument for the next evaluation. The Contractor shall deliver the draft presentation materials to the COTR no later than 111 weeks after the award of this contract. The COTR will provide comments no later than 2 weeks after receipt. The Contractor shall send the approved materials to all work group members and the COTR within 115 weeks of the award of this contract. The Contractor shall arrange and conduct the Fall 1999 expert work group meeting no later than 116 weeks after the award of this contract. The Contractor shall present the prepared materials and facilitate group discussion. The Contractor shall deliver a summary of the meeting to the COTR no later than 2 weeks after the meeting. 5.9 Prepare materials for and conduct Spring 2000 work group meeting. The Contractor shall prepare a presentation and agenda for the Spring 2000 meeting including the findings from the 1999 report and the outline and data analysis plans for the 2000 Interim Report. The Contractor shall also prepare an update report on the recommended changes in the study design and instrument for the next evaluation. The Contractor shall deliver the draft presentation materials to the COTR no later than 136 weeks after the award of this contract. The COTR will provide comments no later than 2 weeks after receipt. The Contractor shall send the approved materials to all work group members and the COTR within 140 weeks of the award of this contract. The Contractor shall arrange and conduct the 1999 expert work group meeting no later than 141 weeks after the award of this contract. The Contractor shall present the prepared materials and facilitate group discussion. The Contractor shall deliver a summary of the meeting to the COTR no later than 2 weeks after the meeting. 5.10 Coordinate independent review of draft 2000 Interim Report. The Contractor shall send copies of the first draft of the 2000 Interim Report to the members of the work group for review no later than 166 weeks after the award of this contract. The Contractor shall deliver the comments from this review to the COTR no later than 5 weeks after the report is sent to reviewers. 5.11 Prepare materials for and conduct Spring 2001 work group meeting. The Contractor shall prepare a presentation and agenda for the Spring 2001 meeting including the findings from the evaluation thus far and the outline and data analysis plans for the final report. The Contractor shall deliver the draft presentation materials to the COTR no later than 189 weeks after the award of this contract. The COTR will provide comments no later than 2 weeks after receipt. The Contractor shall send the approved materials to all work group members and to the COTR within 193 weeks of the award of this contract. The Contractor shall arrange and conduct the 1999 expert work group meeting no later than 194 weeks after the award of this contract. The Contractor shall present the prepared materials and facilitate group discussion. The Contractor shall deliver a summary of the meeting to the COTR no later than 2 weeks after the meeting. 5.12 Coordinate independent review of Final Report. The Contractor shall send copies of the first draft of the final report to the members of the work group for review no later than 218 weeks after the award of this contract. The Contractor shall deliver the comments from this review to the COTR no later than 5 weeks after the report is sent to reviewers. PART B: EVEN START PERFORMANCE INFORMATION REPORTING SYSTEM (ESPIRS) Section 3: Revise and Implement Universe Data Entry System (ESPIRS) for States and Projects Task 6. Review data collection forms and evaluation design for clarification The COTR will provide the contractor with the information collection package within one week of the award of this contract to enable the contractor to become familiar with the proposed forms and their differences from the previous information systems. (See Government-provided materials list in appendix). The contractor shall review the data collection forms. The contractor shall identify differences in the ESPIRS forms from the previous information system's forms to determine different analytic techniques and new ways to use all of the data to be obtained in this project to evaluate Even Start. The contractor shall consult the COTR as needed to clarify the forms and identify any problems anticipated in translating the forms to a computerized data entry system. The contractor shall provide questions and comments and any additions to the analysis plan in the form of a memo to be faxed to the COTR within 2 weeks after the award of this contract. The COTR will answer these questions and address comments by telephone or meeting within 2 weeks of receipt. Task 7. Adapt computer system for local project and State coordinator data entry of universe data. The computerized entry system is designed to allow for easy data entry on the part of Even Start grantees, to reduce data entry costs to the Government and the contractor, and to result in cleaner, more accurate data. The COTR shall provide the contractor with the data entry system coding for DOS and for Macintosh within one week of the award of this contract. The contractor shall model revisions to the system on the previous system. In addition to providing the DOS and Macintosh versions, the contractor shall produce a Windows version. The contractor shall make the following revisions to the system: 7.1 Update computerized forms system. The contractor shall revise the current system to include the changes made to the forms by ED and OMB. The contractor shall complete initial development of the universe system by approximately 10 weeks of the award of this contract. The contractor shall then proceed to test the system on no more than 9 local and State project directors and shall complete the system (described in Tasks 7.2 and 7.3 below). 7.2 Test the system. After completing initial development of the universe system, the contractor shall test the draft data entry system on local Even Start projects and State coordinators. This activity shall not involve travel (outside of the local area) by the contractor. The COTR will provide names of local project directors and State coordinators who are willing to test the DOS software, the Windows software, and the Mac software. 7.3 Demonstrate and deliver data entry system. The contractor shall present a demonstration of the data entry system to the COTR within 14 weeks of the award of this contract. The COTR will comment within 1 week. The contractor shall revise the final package and submit it to the COTR on diskette, within 18 weeks of the award of this contract. Task 8. Update the data entry system user guide and prepare a State version. The contractor shall review and adapt the current user guide (see Government- provided materials list in appendix) to reflect changes in the system and shall develop a guide for State coordinators to use in implementing the system. The local and State guides shall be in addition to any guidance provided in the software system itself. The contractor shall provide a draft of these two guides to the COTR within 18 weeks of the award of this contract. The COTR will review within 1 week. The contractor shall submit final versions of the guides to the COTR within 21 weeks after the award of this contract. Section 4: Implement Universe Data Entry System (ESPIRS) Task 9. Conduct annual evaluation conference and training program for local Even Start project directors and evaluators and State Even Start coordinators To keep all the State and local grantees current on the requirements of the national evaluation as well as provide training necessary to implement the system, the contractor shall convene regular meetings on the evaluation requirements in the Even Start legislation and in the data collection system designed and implemented under Task 7. The Contractor shall provide this training to all projects and State coordinators at least once during the life of this contract. Trainings shall be held in October of each year of the contract (except in year one, see Tasks 9.1-9.4), and if new grants are awarded after October an additional conference shall be held in February. The number of participants at each conference shall not number more than 250 (except in year one, see Task 9.1). Participants in the conferences will be expected to pay for their travel costs. Specific procedures the contractor shall follow are: 9.1 Plan first evaluation conferences. Due to the timing of the award of this contract, the Contractor shall hold the first evaluation conferences in March and April of 1998. Since all current grantees will need training at the beginning of this contract, the contractor shall expect to train about 1200 participants in the first year of this contract. The contractor shall hold three regional conferences to number no more than 400 participants at each conference. The Contractor shall hold the regional conferences in three areas: east coast, west coast, mid-region of the U.S. The contractor shall hold additional regional conferences in the event that there are more than 400 participants scheduled for each conference. The contractor shall coordinate this training with the Comprehensive Technical Assistance Centers in the regions. To the extent that coordination with other relevant program development conferences is feasible in time frame, location, and cost, the contractor shall arrange such coordination. 9.2 Submit agenda, invitation letter, and proposed site. By week 10 after the award of this contract, the contractor shall propose sites for the first set of conferences. By 14 weeks after the award of this contract, the contractor shall submit to the COTR a draft agenda, draft conference evaluation survey, and letter of invitation. The COTR will furnish copies of the agendas, transmittal letters, and conference evaluation forms for a previous conference for use as a model, within 1 week after the award of this contract. In the registration materials, the Contractor shall ask projects and State coordinators which version of the software they would like to use. After review and approval of the agenda and invitation letter by the COTR (within 1 week), the contractor shall mail out the final letter of invitation and agenda to all State Even Start coordinators and local Even Start projects and the COTR within 21 weeks after the award of this contract. 9.3 Hold first evaluation conferences. In March-April 1998, (conferences held no later than week 34) the contractor shall organize at least three three-day training and evaluation assistance conferences for the estimated 650 local Even Start grantees plus 52 State coordinators of Even Start. The number of participants expected are: 2 persons per local project and 52 State participants, for a total of 1352 Even Start participants. As stated in Task 9.1, in order to achieve a group of no more than 500 participants, the first evaluation conference shall be a set of at least three meetings. The following table specifies the content, responsible party, and expected session length. The contractor shall expect that some of the sessions will run concurrently. In the registration forms, the Contractor shall ask participants what version of the ESPIRS they will need. Session Topic Overview of Even Start Program and Regulations (Mandatory) Overview of the National Evaluation (Mandatory) Automated ESPIRS Demonstration /Hands on Training Findings from Previous Evaluations (Mandatory) Test administration Purpose Large general session; presentation of program model, law General session; presentation of evaluation components, schedule, procedures, and data collection forms Small group session; demonstration of automated data entry software, using a computer Large general session; presentation of findings from previous evaluations General session; Provide information on administering the adult and child instruments to be used in Form E. Length of Sessions 1-1.5 hours 1.5 hours 1.5 hours, cannot be concurrent 50 minutes about 1.5 hours Presenters ED--Pat McKee Contractor Contractor Contractor Contractor Number of Participants Max. 250 Max. 250 Max. 40 per session Max. 250 Max. 100 Meeting Room Format Large room, classroom style, head table Large room, classroom style, with head table Breakout room, either classroom style or theater style, with head table Large room, either classroom style or theater style, with head table Large room, classroom style, head table Audiovisual Equipment 1 large screen 1 overhead projector 1 microphone 1 large screen 1 overhead projector 1 microphone 1 screen 1 data projector 1 computer 1 computer stand at least 20 computers for the hands on 1 large screen 1 overhead projector 1 microphone 1 large screen 1 overhead projector 1 microphone Scheduling Sequence Ideally, before any other session listed Must be before ESPIRS demo Must be after the evaluation overview Should be after the evaluation overview, before or after demo Must be after the evaluation overview In addition to the training sessions described in the table, the contractor shall provide additional ideas and comments for other sessions. The contractor shall provide all State coordinators and all local project directors (who have not previously received or need another set of materials) with a hard copy of the data forms and instructions for completing them, a copy of the software in the version they requested on a diskette, the user's guide, and a schedule for delivery of data to the contractor at each of the conferences. This package of materials shall be referred to as the new grantees package. 9.4 Memo reporting on conference. The contractor shall prepare a brief report on the events of the meeting, including a list of attendees and identification of projects or State coordinators that did not attend the training, synthesis of questions asked and comments made, and recommendations for any future evaluation training. The contractor shall submit the report to the COTR within 3 weeks after the date of the conference. 9.5 Future evaluation conferences. The contractor shall hold similar conferences in October 1998, October 1999, and October 2000 for new grantees (two participants per project) and a limited number of continuing projects that may need more instruction on the evaluation. The contractor shall include the Comprehensive Technical Assistance Centers in the regions in these conferences. The contractor shall ensure that each conference has no more than 250-300 participants. In the event that a conference has more than 300 scheduled participants, the contractor shall consult with the COTR about holding regional conferences. The contractor shall inform the COTR of any new grants that are awarded by the States after the yearly October conferences. In the event that the States award new grants after October, the contractor shall hold similar conferences in February of 1999, 2000, and 2001. [Task specifications for the February conferences will be in brackets below]. The contractor shall submit a memo with proposed dates for the conference/s and location/s by the April [August] prior to each conference for review and approval by the COTR. The COTR will review and comment on the memo within 2 weeks of receipt. The contractor shall submit a draft agenda and transmittal letter to the COTR for review by the May [September] prior to each conference. The COTR will review and comment on the memo within 2 weeks of receipt. (The Contractor shall ask projects and State coordinators which version of the software they would like to use in the registration materials.) The contractor shall send the approved registration materials to projects and State coordinators and the COTR at least two full months prior to each conference, (roughly equivalent to the August [December] prior to each conference). The contractor shall hold the conferences in October 1998 [February 1999], October 1999 [February 2000], and October 2000 [February 2001]. The contractor shall report to the COTR in a memo on the conference, providing the same types of information described in task 9.4 within 3 weeks of the conference. Task 10. Receive and enter State coordinator data to develop and maintain tracking system of projects. 10.1 Contact State Coordinators. Within 1 week of the award of this contract, the COTR will provide a list of State Coordinators' contact information and the previous tracking system information. Within 2 weeks of the award of this contract, the contractor shall mail out to all State Coordinators (and the U.S. Department of Education's Office of Elementary and Secondary Education for Migrant and Tribal projects) the ESIS-2's state form with a letter requesting contact information on the continuing and new projects in their state. The contractor shall include in the mailing a list of all projects from the previous tracking system from each state to ensure that the information contained in the previous tracking system on continuing projects is current. In the package, the contractor shall also explain to State coordinators that the state form shall be computerized and the software given out at a training on the system in spring 1998. Within 4 weeks of the award of this contract, the contractor shall follow-up with any State Coordinator who has not responded. 10.2 Develop tracking system. Starting with the information provided by State Coordinators in task 10.1, the contractor shall put together a tracking system of projects. In addition to the information provided by the State coordinators, the contractor shall place any other useful information about projects into the tracking system. Examples of other information the contractor shall track are data collection information (has the project submitted data and if so in what format), number of years the project has been in operation, whether or not the current staff of the project have gone through training, whether or not the project has received the new grantees package. Within 6 weeks of the award of this contract, the contractor shall deliver the updated tracking system to the COTR on disk or electronically in software that is accessible to the COTR. After this delivery, the contractor shall provide future updates of the tracking system at the COTR's request. 10.3 Update tracking system. The contractor shall ensure that the tracking system contains the most up to date information on all grantees. To ensure that new grantees are included in the tracking system, the contractor shall mail out a reminder letter to all State Coordinators along with another paper copy of the state form of the ESPIRS or copy of the software if necessary (including ED's Office of Elementary and Secondary Education in the mailing for Migrant and Tribal projects) requesting that they update the state form for projects in their state that are continuing (mailing shall include the contractor's most recent list of projects) as well as complete a state form for new projects. The contractor shall send reminders to State coordinators at least twice a year in late June of 1998, 1999, 2000 and in late October of 1998, 1999, and 2000. Within 4 weeks of sending the letter, the contractor shall follow-up with any State Coordinator who has not responded and shall obtain the information in time to send projects invitations to the evaluation conferences by August [December] of 1998, 1999, and 2000 and to assign new projects a project code. The contractor shall use the information obtained to update the project tracking system each year. Task 11. Provide projects with new grantees package 11.1 Prepare initial evaluation instructional mailing packages. To ensure that projects begin intake data collection for the beginning of the 1997-98 school year, the contractor shall send a hard copy of the data collection forms to all projects with instructions on how to collect intake data using the paper forms. The contractor shall use this mailing to obtain information from each project on the version of the computerized system software they would like to receive at the spring 1998 conference. The contractor shall make clear to projects that they will receive training in the spring of 1998 on how to use the computerized system to transfer the information taken on the paper forms to the computerized system. The contractor shall provide a draft copy of the materials to the COTR no later than 4 weeks after the award of this contract. Upon COTR approval (within 1 week), the contractor shall send the evaluation package out to all grantees and the COTR no later than 6 weeks after the award of this contract. 11.2 Ensure all projects (who did not already receive a new grantees package at conferences) receive a new grantees package. The contractor shall prepare the materials necessary for projects to complete yearly data collection. This package of materials shall include a hard copy of the data forms and instructions for completing them, a copy of the software in the version projects have requested on a diskette, the user's guide for that version of the software, a schedule for delivery of data to the contractor, and other useful tips learned by the contractor throughout the contract. This package shall be called the new grantees package (see Government-provided materials list in appendix). The contractor shall provide each project with this package either at the evaluation conferences (see Task 9.3), or by mail for those projects who do not attend a conference. The contractor shall send a package to projects not attending the annual conference by October of each year (or as soon as possible for new grantees upon the contractor finding out that the particular grantee will not attend the conference). Task 12. Receive, edit, and enter July data submitted by local projects. The contractor shall receive evaluation data from the grantees by July 15 of each year and enter them into a database for analysis. The contractor shall review the data submitted on diskette and, if clean, copy the data into the main database. If not, the contractor shall edit the data, contacting the local project as appropriate. Because not every local project will be able to use the computerized software package for data submission, the contractor shall provide alternative methods of data submission. In July 1996, 521 projects submitted data using the computerized system (70 percent submitted DOS diskettes and 30 percent submitted Mac diskettes). Thirty-nine projects submitted paper forms. A local project will have the option to submit data in hard copy. The contractor's staff shall enter the data from these local projects into the database, contacting the local project if needed for clarification or to add missing data. Hand entry of data submitted through hard copy forms has been time consuming in the past. The contractor shall complete data entry and deliver reports by the date specified in tasks 16 through 20 regardless of possible data entry burden created by projects who use paper forms. Thus the contractor shall strongly consider creating scannable versions of the forms for projects indicating that they are not able to use the computerized system. The contractor shall mail to each local project a summary table of the data submitted and inputted for that project--to provide the project an opportunity to review the data and to identify problems. The contractor shall also mail to each State coordinator a summary of the data submitted by projects in their state--to provide for project monitoring. Upon request from State coordinators and local projects, the contractor shall send the raw data in a format compatible with the needs of the requester. The contractor shall combine data from all grantees into an analytic file that includes all information received (such as information on participant characteristics; core program services; support services and other special activities; the processes of recruitment, screening, and assessment; results of testing; staffing; cooperative arrangements with other agencies; and factors that influenced implementation of the project). The contractor shall update the COTR by memo on a bi-weekly basis during the period of data collection as to the response rate and provide a list of projects that have not yet sent data. PART C: EVALUATION OF THE EVEN START PROGRAM Section 5: Develop and Implement Experimental Design Study Component Task 13. Develop materials to implement experimental design study on program effectiveness (See Task 5.2 and 5.3) 13.1 Develop design. The contractor shall submit to the COTR no later than 7 weeks after the award of this contract a description of proposed procedures for implementing a study to be broadly representative of the program that shall randomly place families into treatment or control groups and collect data on progress, outcomes and program intensity In the study, the contractor shall provide outcomes data from these two groups of families to analyze the impact and effectiveness of the program. The contractor shall plan this study in accordance with the July 15th deadline for the universe study data provided by the ESPIRS for coordinating both sets of data for the national evaluation reports. 13.2 Prepare draft design. The contractor shall gain ideas on the design of the experimental study component of the national evaluation through an expert work group meeting held in November of 1997 [See task 5.2]. Within 3 weeks of the work group meeting (approximately December 12, 1997), the contractor shall deliver to the COTR a draft design for the experimental study that is refined by the proceedings of the expert work group. In the design, the contractor shall include the planned methodology, data collection instrument, variables to be measured, data analysis plan, site selection plan, and plan to incorporate the analysis results into the national evaluation interim and final and other reports. The contractor shall design the data collection plan to coordinate with the July 15th universe data collection deadline. The COTR will review this design within 3 weeks after receipt. The contractor shall submit a revised design to the COTR within 19 weeks after the award of this contract. 13.3 Prepare final design. The contractor shall again gather input on the revised study design through an expert work group meeting held in February of 1998 [See task 5.3]. Within 3 weeks of the meeting (approximately March 6, 1998), the contractor shall submit the final study design to the COTR. The contractor shall incorporate both the COTR's comments and the input of the work group in the final design. 13.4 Prepare and support OMB clearance package. The contractor shall prepare an OMB review package for the approved experimental design study component to include data collection forms and a full supporting statement to the request for OMB review of collection of information (pursuant to the Paperwork Reduction Act and 5 CFR 1320). The COTR will be responsible for obtaining approval from the appropriate offices. The contractor shall make available an appropriate staff member to assist in responding to questions raised in the Federal review, and to clarify and amend the supporting statement if needed. The contractor shall include in the forms clearance package brief, concise statements of (a) the Even Start mandate for evaluation; (b) types of information to be requested; (c) steps taken to minimize respondent burden; (d) the sampling plan; (e) plans for tabulating data and statistical tests for analyzing it; (f) the data collection schedule and estimated time lapse before results are to be made available; (g) the steps taken to have plans reviewed by outside persons; (h) estimates of burden for each type of respondent and how such figures were estimated; (i) discussion of "sensitive questions," if any; and (j) estimates of the cost of the activity. The package brief shall include a copy of the instruments, marked to show the data collection's mandate and the voluntary nature of the respondent's participation, and shall include table shells indicating how data shall be tabulated and analyzed. The contractor shall submit a draft of the OMB forms clearance package to the COTR within 19 weeks of the award of this contract. The COTR will review and comment within 2 weeks. The contractor shall gather input into the draft package at the expert work group meeting to be held in February of 1998. The contractor shall submit the final version of the OMB forms clearance package (incorporating COTR's and work group's comments) to the COTR within 28 weeks after the award of this contract, in hard copy and on disk or electronically using software accessible to the COTR. Task 14. Select Even Start projects for experimental design study and develop data collection schedule. 14.1 Select projects. Project participation in this study is voluntary. The contractor shall solicit recommendations on possible participant projects as well as volunteers and determine in conjunction with the COTR the extent to which projects are feasible candidates (for example, willing to implement an experimental design). The contractor shall notify State coordinators about the selection of projects in their state. The contractor shall propose a list of possible sites using the site selection plan designed in task 13.2. The contractor shall begin to conduct negotiations of random assignments and site visit schedules with the possible sites within 28 weeks of the award of this contract. The contractor shall submit the list of projects who have agreed to and are capable of random assignment to the COTR for review and approval, along with a draft letter of notification to the projects from the contractor, within 43 weeks of the award of this contract. After COTR review (within 2 weeks of receipt of list), the contractor shall submit the final list and transmittal letter with tentative data collection schedules to the COTR and send this information to the participating projects within 47 weeks of the award of this contract. 14.2 Implement data collection. The contractor shall arrange plans to collect the data specified in the approved forms and study design from local projects. The contractor shall make these arrangements with individual projects based on the date at which they expect to start providing services to families. The contractor shall begin data collection for the randomized study by fall 1998 (no later than 30 days after families in each participating project begin receiving services). The contractor shall work with individual projects to randomly place families into an Even Start treatment group or a control group of equal size. The contractor shall also work with participating projects on making sure that the control group receives no Even Start services while participating in this study. The contractor shall send the notification letter for participation in the study along with confirmation of the first period of data collection scheduled to projects and the COTR at 50 weeks after the award of this contract. The contractor shall complete data collection by July 15th of each year of the study. 14.3 Prepare a paper on a study design for the experimental design component. Information on past Even Start children's achievement when they go on to school and adult's success in the workforce or further education has the potential to be important in showing the long-term effectiveness of Even Start. The Contractor shall prepare a detailed design paper for a study that could examine participants in the experimental design component in the 3 to 5 years following Even Start. The contractor shall deliver this paper to the COTR within 156 weeks of the award of this contract. 14.4 Maintain a database of participants in the experimental design component. The contractor shall maintain a database of participating families (with their agreement) in the experimental design component for the purposes of possible future follow-up studies. The Contractor shall deliver the updated database to the COTR upon completion of this contract. Section 6: Implement National Evaluation Framework and Prepare Evaluation Reports Task 15. Update data analysis and evaluation plan The contractor shall update the data analysis plan submitted in the proposal. In the revised plan, the contractor shall describe the specific statistical methods that will be used to analyze the data from the ESPIRS and experimental study components. The contractor shall also describe the tables and graphs that summarize the data concerning participant characteristics; core program services; support services and other special activities; the processes of recruitment, screening, and assessment; staffing; cooperative arrangements with other agencies; factors that influenced implementation of the project; and technical assistance needs identified by the projects. The contractor shall describe planned tables and graphs based on the achievement and program intensity data. The contractor shall submit the draft analysis plan to the COTR within 10 weeks of the award of this contract. The COTR will review and comment within 2 weeks. The contractor shall submit the final analysis plan to the COTR within 13 weeks of the award of this contract. Task 16. Prepare interim reports and summary database 16.1 Analyze data and prepare interim reports. The contractor shall prepare and submit three annual interim reports. These reports shall include narrative and descriptive information and analytic charts, graphs, and tables to summarize the data concerning participant characteristics; core program services; support services and other special activities; the processes of recruitment, screening, and assessment; results of testing and other outcome measures; staff characteristics and staff development activities; cooperative arrangements with other agencies; and factors that influenced implementation of the project. In the reports, the contractor shall synthesize the universe data from the ESPIRS and the data from the experimental design study component. The reports should be cumulative and compare the most recent results with results from the previous two evaluations. The contractor shall include a detailed technical appendix of the methodology used in the data analysis. The contractor shall prepare a draft outline of each report and submit it to the COTR in June of 1998, 1999, and 2000. The COTR will comment within 2 weeks. The contractor shall submit the final outline to the COTR within June of 1998, 1999, and 2000. The contractor shall submit five copies of the draft interim report to the COTR (estimated size--200 pages) on October 15, 1998, October 15, 1999, and October 15, 2000. The COTR shall review the draft and provide comments within 3 weeks. The contractor shall revise the report and submit 10 copies of a second draft report to the COTR by November 20 of 1998, 1999, and 2000 that the COTR will send to appropriate offices within ED for review. The COTR will provide comments on the second review within 4 weeks. The contractor shall incorporate the second round of comments and submit 10 copies of a final draft interim report to the COTR by January 8 of 1999, 2000, and 2001. The COTR will provide comments from a third review within 4 weeks. The contractor shall incorporate the third round of comments and submit 100 copies of the final interim reports to the COTR by February 19 of 1999, 2000, 2001. 16.2 Prepare summary database. The contractor shall annually prepare and submit a database containing summary data submitted by all Even Start grantees. The database shall contain measures of participant characteristics; core and support services; recruitment, screening, and assessment; test scores; staffing; and cooperative arrangements. The contractor shall deliver the database to the COTR on diskette in a format that is accessible to the COTR at the same time as each final interim report. Task 17. Prepare final report and summary database. The contractor shall prepare a final report that summarizes the results of the three interim reports and also compares the final year's data with the first three years. The final report shall also encompass an analysis of the data from the experimental design study. The contractor shall include a detailed technical appendix of the methodology used in the data analysis. 17.1 Prepare final report. The contractor shall prepare a draft outline and submit it to the COTR in June of 2001. The COTR will comment within 2 weeks. The contractor shall submit the final outline to the COTR within June of 2001. The contractor shall submit five copies of the draft final report to the COTR on October 15, 2001. The COTR shall review the draft and provide comments within 3 weeks. The contractor shall revise the report and submit 10 copies of a second draft report to the COTR by November 20 of 2001 that the COTR will send to appropriate offices within ED for review. The COTR will provide comments on the second review within 4 weeks. The contractor shall incorporate the second round of comments and submit 10 copies of a final draft report to the COTR by January 8 of 2002. The COTR will provide comments from a third review within 4 weeks. The contractor shall incorporate the third round of comments and submit 100 copies of the final report to the COTR by February 19 of 2002. 17.2 Prepare summary database. The contractor shall prepare and submit the final database containing summary data submitted by all Even Start grantees. The database shall contain measures of participant characteristics; core and support services; recruitment, screening, and assessment; test scores; staffing; and cooperative arrangements. The contractor shall deliver the database to the COTR on diskette in a format that is accessible to the COTR at the same time as the final report. Task 18. Develop and produce individual project profiles. The Contractor shall produce project-specific reports of key results from the national evaluation. The Contractor shall provide each participating project with a report presenting analysis results of the project's evaluation data mapped against comparison group results (national or state, for example). The Contractor shall also provide State Coordinators with copies of the individual profiles for each project in their state. 18.1 Conduct survey. The Contractor shall select nine local projects and nine state coordinators to survey on the usefulness of previous project profiles in consultation with the COTR. The Contractor shall create a questionnaire, listing the national evaluation data and comparison groups for each measure included in previous project profiles. The Contractor shall call the selected projects and state coordinators to obtain their agreement to be involved and mail the questionnaires within 46 weeks of the award of this contract. The Contractor shall then call the participants again to obtain their responses regarding the choice of measures and comparison groups within 2 weeks of mailing the survey. The Contractor shall provide the COTR with a summary of the survey results no later than 50 weeks after the award of this contract. 18.2 Produce draft prototype report. The Contractor shall develop a draft report format and draft narrative text and graphs. The Contractor shall deliver a draft prototype report to the COTR no later than 56 weeks after the award of this contract. The COTR will provide comments on the draft within 2 weeks. 18.3 Produce final prototype report. The Contractor shall deliver a final draft of the prototype report to the COTR no later than 60 weeks after the award of this contract. 18.4 Analyze data and produce yearly individual project profiles. The Contractor shall produce individual reports for each project based on the data they provide each year, prepare each report for mailing, and mail to projects with COTR's approval by March of 1999, 2000, 2001, and 2002. The contractor shall also provide a sample of the profiles to the COTR upon request. Task 19. Produce Special Analysis of Migrant Projects 19.1 Produce outline. The Contractor shall develop a draft outline for a report that shall analyze data pulled from the ESPIRS on all the Migrant projects. The outline shall contain areas that the report will cover, variables to be included, and plans to relate findings to the nationwide findings. The Contractor shall deliver this outline to the COTR no later than 54 weeks after the award of this contract. The COTR will provide comments within 2 weeks of receipt. 19.2 Produce draft of Special Analysis of Migrant Projects. The Contractor shall perform data analysis and deliver a draft special report to the COTR on the Migrant Even Start projects based on the outline no later than November of 1998, 1999, 2000, and 2001. The COTR will provide comments no later than 2 weeks after receipt. 19.3 Produce final draft report. The Contractor shall incorporate COTR comments and deliver a final draft version of the Special Analysis of Migrant Projects to the COTR no later than January of 1999, 2000, 2001, 2002. The COTR will provide further comments within 3 weeks of receipt. 19.4 Produce final report . The Contractor shall incorporate COTR comments and deliver a final version of the Special Analysis of Migrant Projects to the COTR no later than March of 1999, 2000, 2001, 2002. Task 20. Produce Special Analysis of Tribal Projects 20.1 Produce outline. The Contractor shall develop a draft outline for a report that shall analyze data pulled from the ESPIRS on all the Tribal projects. The outline shall contain areas that the report will cover, variables to be included, and plans to relate findings to the nationwide findings. The Contractor shall deliver this outline to the COTR no later than 54 weeks after the award of this contract. The COTR will provide comments within 2 weeks of receipt. 20.2 Produce draft of Special Analysis of Tribal Projects. The Contractor shall perform data analysis and deliver a draft special report to the COTR on the Tribal Even Start projects based on the outline no later than November of 1998, 1999, 2000, and 2001. The COTR will provide comments no later than 2 weeks after receipt. 20.3 Produce final draft report. The Contractor shall incorporate COTR comments and deliver a final draft version of the Special Analysis of Tribal Projects to the COTR no later than January of 1999, 2000, 2001, 2002. The COTR will provide further comments within 3 weeks of receipt. 20.4 Produce final report . The Contractor shall incorporate COTR comments and deliver a final version of the Special Analysis of Tribal Projects to the COTR no later than March of 1999, 2000, 2001, 2002. PART D: UPDATE NATIONAL EVALUATION DESIGN Section 7: Review and Recommend Changes to National Evaluation Design Task 21. Review and update Even Start program performance indicators The contractor shall review the Even Start program performance indicators (see Government-provided materials list in appendix) and suggest revisions based on the information that the national this project has provided. The Contractor shall deliver a draft set of revised performance indicators to the COTR within 222 weeks of the award of this contract. The COTR will review within 3 weeks. The contractor shall deliver a final set of performance indicators to the COTR within 229 weeks of the award of this contract. Task 22. Recommend changes to the Even Start national evaluation design. The contractor may propose changes to the system (ESPIRS, instruments, experimental design component, data analysis plan, for example) after having developed experience with the system, especially after data submissions by local projects, and from communications with State coordinators and local users through the annual conferences and telephone conversations. In Year One, the contractor shall submit the memo to the COTR proposing changes to the evaluation system no later than August of 1998. The COTR shall review and comment within 1 week. In Year Three, the contractor shall conduct [see task 5.8 and 5.9] two meetings of the expert work group to advise the contractor on the issues of changes to the system and recommendations for the outcomes assessment, indicators and standards. The contractor shall prepare recommended changes and points for discussion for the work group within 111 weeks of the award of this contract. The contractor shall gather input from the Fall 1999 work group meeting and submit to the COTR a draft redesign of the national evaluation within 136 weeks of the award of this contract. The work group shall discuss this draft redesign at the Spring 2000 meeting. The contractor shall incorporate work group input and COTR comments and deliver a revised redesign for the national evaluation to the COTR no later than 2 weeks (by May 26, 2000) after the meeting. The COTR will review and comment within 2 weeks of receipt. PART E: ADAPT NATIONAL EVALUATION DESIGN (OPTIONS) Section 8: Prepare Revised Evaluation Design (options) Task 23. Edit and test forms (option). 23.1 Revise forms. Within 2 weeks of receipt of the recommended changes memo from under task 22, the COTR will send comments and a markup of the current paper forms for revision. The contractor shall revise the forms and deliver a hard copy of the revised forms to the COTR within 146 weeks after the award of this contract. The COTR will provide comments within 1 week of receipt. 23.2 Test forms. The COTR will provide names of potential project directors to test the forms within 147 weeks of the start of this contract. The contractor shall contact these project directors and arrange for them to test the revised forms within 150 weeks of the award of this contract. The contractor shall incorporate both the comments of the COTR and the comments of those who test the forms into the revised forms. The contractor shall deliver to the COTR the revised forms both in hard copy and on disk or electronically in a format that is accessible to the COTR within 152 weeks of the award of this contract. Task 24. Prepare and support IMCD/OMB forms clearance request (option). In the event that ED decides to change the design of the national evaluation, the contractor shall prepare an OMB review package for the approved Even Start national evaluation design to include revised forms and a full supporting statement to the request for OMB review of collection of information (pursuant to the Paperwork Reduction Act and 5 CFR 1320). The COTR will be responsible for obtaining approval from the appropriate offices. The contractor shall make available an appropriate staff member to assist in responding to questions raised in the Federal review, and to clarify and amend the supporting statement if needed. The contractor shall include in the forms clearance package brief, concise statements of (a) the Even Start mandate for evaluation; (b) types of information to be requested; (c) steps taken to minimize respondent burden; (d) the sampling plan; (e) plans for tabulating data and statistical tests for analyzing it; (f) the data collection schedule and estimated time lapse before results are to be made available; (g) the steps taken to have plans reviewed by outside persons; (h) estimates of burden for each type of respondent and how such figures were estimated; (I) discussion of "sensitive questions," if any; and (j) estimates of the cost of the activity. The package brief shall include a copy of the instruments, marked to show the data collection's mandate and the voluntary nature of the respondent's participation, and shall include table shells indicating how data shall be tabulated and analyzed. The Contractor shall review each section of the original request and provide information and rationale for the changes requested, identified by section. The contractor shall include a table showing, by form, the approved data burden and costs, the proposed changes, and the effects (increase or decrease in burden and cost). The COTR will approve the revised forms within 153 weeks of this contract. The contractor shall submit a draft of the OMB forms clearance package to the COTR within 155 weeks of the award of this contract. The COTR will review and comment within 1 week. The contractor shall submit the final version of the OMB forms clearance package to the COTR within 157 weeks of the award of this contract, in hard copy and on disk or electronically in a format that is accessible to the COTR. Task 25. Update the data collection forms, computerized system, and user's manual (option). When the forms receives OMB approval, the contractor shall make any necessary modifications to the paper forms. The contractor shall also update the computer system, and the user's manual. The Contractor shall provide a demonstration of the data entry system to the COTR as well as a draft copy of the updated user's manual within 8 weeks after OMB approval. The COTR will provide comments within 2 weeks. The contractor shall prepare and test the final data entry system and submit it to the COTR on diskette, taking into account the COTR's comments, within 12 weeks after OMB approval. The Contractor shall also deliver a hard copy of the forms and user's manual as well as on diskette to the COTR within 12 weeks of OMB approval. B. Reporting Requirements 1. Monthly Management Report. In addition to any reports required as a deliverable for particular tasks, the contractor shall prepare monthly management reports due within ten work days after the end of each month. In this report, the contractor shall: þ Summarize the major activities and accomplishments for the reporting period. þ Provide information for each project task regarding significant findings and events, problems encountered, and staff use. þ Specify the extent to which the project is on schedule. þ Briefly describe the activities planned for the next month. þ Identify and discuss significant deviations from the substantive and time factors in the baseline management plan. If there are no exceptions, state that there are no exceptions. If there are exceptions to the baseline management plan, describe the plan for solving the problems. þ Identify and discuss any decisions which may be needed from the COTR. 2. Monthly Manpower/Expenditure Report. This report is due within ten work days after the end of each month. The report, prepared and signed by the project director, shall: þ Summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. It shall project similar assignment information for the upcoming month. þ Exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip. C. Timeline The contractor shall follow the following timeline for all tasks in this contract. The contractor shall only change due dates after receiving approval from the COTR. Due dates for submission or completion of tasks are expressed in time after the award of this contract and the corresponding calendar dates. Due Date, After the Task Award of this Contract PART A: PROJECT MANAGEMENT Section 1: Project Management Task Description Due Date, After the Contract's Beginning Date Estimated Due Date Task 1: Update baseline management plan. 1.1 Meet with Dept of Education and summarize meeting 5 working days September 5, 1997 1.2 Revise Baseline Management Plan 2 weeks September 12, 1997 1.3 Approval of Baseline Management Plan ED's comments Final Plan 3 weeks 4 weeks September 19, 1997 September 26, 1997 1.4 Provide quarterly management updates Once every quarter Task 2: Provide technical assistance on the evaluation through telephone calls and written communications. Include the number of calls received or made and the topic of the inquiries in the monthly report Monthly Management Report by the tenth day of each month Task 3: Establish Performance Measurement System Monthly Management Report by the tenth day of each month Section 2: Project Design and Improvement Task Description Due Date, After the Contract's Beginning Date Estimated Due Date Task 4: Develop overview booklet on this project Draft overview booklet COTR comments Final overview booklet 5 weeks 7 weeks 9 weeks October 3, 1997 October 17, 1997 October 31, 1997 Task 5: Establish an expert work group 5.1 Proposed list of work group members Selection of work group members 2 weeks 6 weeks September 12, 1997 October 10, 1997 5.2 Fall 1997 meeting Draft presentation materials COTR comments Send materials Hold meeting Summary of meeting 7 weeks 9 weeks 11 weeks 12 weeks 15 weeks October 17, 1997 October 31, 1997 November 14, 1997 November 21, 1997 December 12, 1997 5.3 Winter 1998 meeting Draft presentation materials COTR comments Send materials Hold meeting Summary of meeting 19 weeks 21 weeks 23 weeks 24 weeks 27 weeks January 9, 1998 January 23, 1998 February 6, 1998 February 13, 1998 March 6, 1998 5.4 Coordinate independent review of draft 1998 Interim Report Send copies to reviewers Deliver review comments to COTR 60 weeks 65 weeks October 23, 1998 November 27, 1998 5.5 Fall 1998 meeting Draft presentation materials COTR comments Send materials Hold meeting Summary of meeting 59 weeks 61 weeks 63 weeks 64 weeks 66 weeks October 16, 1998 October 30, 1998 November 13, 1998 November 20, 1998 December 4, 1998 5.6 Spring 1999 meeting Draft presentation materials COTR comments Send materials Hold meeting Summary of meeting 84 weeks 86 weeks 88 weeks 89 weeks 91 weeks April 9, 1999 April 23, 1999 May 7, 1999 May 14, 1999 May 28, 1999 5.7 Coordinate independent review of draft 1999 Interim Report Send copies to reviewers Deliver review comments to COTR 114 weeks 119 weeks November 5, 1999 December 10, 1999 5.8 Fall 1999 meeting Draft presentation materials COTR comments Send materials Hold meeting Summary of meeting 111 weeks 113 weeks 115 weeks 116 weeks 118 weeks October 15, 1999 October 29, 1999 November 12, 1999 November 19, 1999 December 3, 1999 5.9 Spring 2000 meeting Draft presentation materials COTR comments Send materials Hold meeting Summary of meeting 136 weeks 138 weeks 140 weeks 141 weeks 143 weeks April 7, 2000 April 21, 2000 May 5, 2000 May 12, 2000 May 26, 2000 5.10 Coordinate independent review of draft 2000 Interim Report Send copies to reviewers Deliver review comments to COTR 166 weeks 171 weeks November 3, 2000 December 8, 2000 5.11 Spring 2001 meeting Draft presentation materials COTR comments Send materials Hold meeting Summary of meeting 189 weeks 191 weeks 193 weeks 194 weeks 196 weeks April 13, 2001 April 27, 2001 May 11, 2001 May 18, 2001 June 1, 2001 5.12 Coordinate review of final report Send copies to reviewers Deliver review comments to COTR 218 weeks 223 weeks November 2, 2001 December 7, 2001 PART B: EVEN START PERFORMANCE INFORMATION REPORTING SYSTEM (ESPIRS) Section 3: Revise and Implement Universe Data Entry System (ESPIRS) for States and Projects Task Description Due Date, After the Contract's Beginning Date Estimated Due Date Task 6: Review data collection forms and evaluation design for clarification COTR provides forms Question and comments COTR response 1 week 2 weeks 4 weeks September 5, 1997 September 12, 1997 September 26, 1997 Task 7: Adapt computer system for State and local project universe data entry 7.1 Update computerized forms system 10 weeks November 7, 1997 7.2 Test the system 7.3 Demonstrate and deliver data entry system Demonstration to COTR COTR comments Submit diskette 14 weeks 15 weeks 18 weeks December 5, 1997 December 12, 1997 January 2, 1998 Task 8: Update the data entry system user guide and prepare a State version. Draft guides COTR comments Final guides 18 weeks 19 weeks 21 weeks January 2, 1998 January 9, 1998 January 23, 1998 Section 4: Implement Universe Data Entry System (ESPIRS) Task Description Due Date, After the Contract's Beginning Date Estimated Due Date Task 9: Conduct annual evaluation conference and training program for local Even Start project directors and evaluators and State Even Start coordinators 9.1 Plan first evaluation conferences 9.2 Submit agenda, invitation letter, and proposed site Propose hotel Draft agenda/invitation COTR comments Mail letter of invitation 10 weeks 14 weeks 15 weeks 21 weeks November 7, 1997 December 5, 1997 December 12, 1997 January 23, 1998 9.3 Hold first evaluation conferences March-April 1998 9.4 Memo reporting on conference 37 weeks May 15, 1998 9.5 Future evaluation conferences Propose hotel/dates COTR comments Draft agenda/invitation COTR comments Mail letter of invitation Hold conference Memo reporting on conference at least 2 mns before conf. Apr/Aug 98, 99, 00 2 wks after receipt May/Spt 98, 99, 00 2 wks after receipt Aug/Dec 98, 99, 00 Oct/Feb 98, 99, 00 within 3 weeks of each year's conference Task 10: Develop and maintain tracking system of projects 10.1 Contact State Coordinators Mail letter/forms Follow-Up 2 weeks 4 weeks September 12, 1997 September 26, 1997 10.2 Develop tracking system Copy of tracking system to COTR 6 weeks October 10, 1997 10.3 Update tracking system Reminder letter/forms Follow-Up June/Oct 98, 99, 00 within 4 weeks of letter Task 11: Provide projects with new grantees package 11.1 Prepare evaluation instructional mailing packages Draft package COTR comments Send package 4 weeks 5 weeks 6 weeks September 26, 1997 October 3, 1997 October 10, 1997 11.2 Ensure projects have new grantees package Mail packages to projects who have not received one ASAP and/or (Oct '98, '99, '00) Task 12: Receive, edit, and enter July data submitted by local projects Update the COTR bi-weekly during the data collection period PART C: EVALUATION OF EVEN START PROGRAM Section 5: Develop and Implement Experimental Design Study Component Task Description Due Date, After the Contract's Beginning Date Estimated Due Date Task 13. Develop materials to implement an experimental design study on effectiveness 13.1 Develop design Draft design plan 7 weeks October 17, 1997 13.2 Prepare draft design Draft design COTR comments Revised design 3 wks after rcpt. 19 weeks December 12, 1997 January 2, 1998 January 9, 1998 13.3 Prepare final design March 6, 1998 13.4 Prepare and support OMB clearance package Draft package COTR comments Final package to OMB 19 weeks 21 weeks 28 weeks January 9, 1998 January 23, 1998 March 13, 1998 Task 14: Select Even Start projects for randomized study and develop data collection protocol 14.1 Select projects Negotiation with projects List of projects/draft letter COTR approval Final list/letter/data collection schedules 28 weeks 43 weeks 45 weeks 47 weeks March 13, 1998 June 26, 1998 July 10, 1998 July 24, 1998 14.2 Implement data collection Confirmation letter Datasets for analysis 50 weeks August 14, 1998 by July 15th of each year the study takes place 14.3 Prepare a study design paper for the experimental design study 156 weeks August 25, 2000 14.4 Maintain database of participants in the experimental design study Updated database throughout contract upon completion of contract deliver upon completion of contract Section 6: Implement National Evaluation Framework and Prepare Evaluation Reports Task Description Due Date, After the Contract's Beginning Date Estimated Due Date Task 15: Update analysis and evaluation plan Draft plan COTR comments Final plan 10 weeks 12 weeks 13 weeks November 7, 1997 November 21, 1997 November 28, 1997 Task 16: Prepare interim reports and summary database 16.1 Analyze data 16.2 Prepare interim reports Draft outline COTR comments Outline First draft Second draft Final draft Final report June '98, '99, '00 June '98, '99, '00 June '98, '99, '00 Oct 15 '98, '99, '00 Nov 20 '98, '99, '00 Jan 8 '99, '00, '01 Feb 19 '99, '00, '01 16.3 Prepare summary database Submit summary database with final interim reports Task 17: Prepare final report and summary database 17.1 Prepare final report Draft outline COTR comments Outline First draft Second draft Final draft Final report June 2001 June 2001 June 2001 Oct 15, 2001 Nov 20, 2001 Jan 8, 2002 Feb 19, 2002 17.2 Prepare summary database Submit summary database with final report Task 18: Develop and produce individual project profiles 18.1 Conduct survey Mail questionnaires Call participants Summary of results 46 weeks 48 weeks 50 weeks July 17, 1998 July 31, 1998 August 14, 1998 18.2 Produce draft prototype report Draft report COTR comments 56 weeks 58 weeks September 25, 1998 October 9, 1998 18.3 Produce final prototype report 60 weeks October 23, 1998 18.4 Analyze data and produce yearly individual project profiles Mail reports to projects and State Coordinators (with COTR's approval) by March '99, '00, '01, '02 Task 19: Produce Special Analysis of Migrant Projects 19.1 Produce outline Outline COTR comments 54 weeks 56 weeks September 11, 1998 September 25, 1998 19.2 Produce draft of Special Analysis of Migrant Projects Draft report COTR comments Nov '98, '99, '00, '01 within 2 weeks 19.3 Produce final draft report Final draft COTR comments Jan '99, '00, '01, '02 within 3 weeks 19.4 Produce final report Mar '99, '00, '01, '02 Task 20: Produce Special Analysis of Projects for Indian tribes and tribal organizations 20.1 Produce outline Outline COTR comments 54 weeks 56 weeks September 11, 1998 September 25, 1998 20.2 Produce draft of Special Analysis of Tribal Projects Draft report COTR comments Nov '98, '99, '00, '01 within 2 weeks 20.3 Produce final draft report Final draft COTR comments Jan '99, '00, '01, '02 within 3 weeks 20.4 Produce final report Mar '99, '00, '01, '02 PART D: UPDATE NATIONAL EVALUATION DESIGN Section 7: Update National Evaluation Design Task Description Due Date, After the Contract's Beginning Date Estimated Due Date Task 21: Review and update Even Start program performance indicators Draft performance indicators 222 weeks November 30, 2001 COTR comments 225 weeks December 21, 2001 Final performance indicators 229 weeks January 18, 2002 Task 22: Recommend changes to the Even Start national evaluation design System change memo COTR comments Recommended changes Draft redesign Revised redesign COTR comments within 1 week 111 weeks 136 weeks 2 wks after mtg. 2 wks after rcpt August 1998 October 15, 1999 April 7, 2000 May 26, 2000 June 9, 2000 PART E: ADAPT NATIONAL EVALUATION DESIGN (OPTIONS) Section 8: Prepare Revised Evaluation Design (options) Task Description Due Date, After the Contract's Beginning Date Estimated Due Date Task 23: Edit and test forms (option) 23.1 Revise forms Hard copy COTR comments 146 weeks 147 weeks June 16, 2000 June 23, 2000 23.2 Test forms Test Deliver revised forms 150 weeks 152 weeks July 14, 2000 July 28, 2000 Task 24: Prepare and support IMCD/OMB forms clearance request (option) COTR approval Draft forms clearance request COTR comments Forms clearance request 153 weeks 155 weeks 156 weeks 157 weeks August 4, 2000 August 18, 2000 August 25, 2000 September 1, 2000 Task 25: Update the data collection forms, computerized system, and user's manual (option) Demonstration of data entry system Draft user's manual COTR comments Final data entry system Final forms and user's manual (Time after OMB approval) within 8 weeks within 8 weeks within 2 weeks of receipt within 12 weeks within 12 weeks D. Deliverables The contractor shall submit the following deliverables to the COTR in the quantities specified below. Dates for submission are expressed in time after the award of this contract and the corresponding calendar dates. All deliverables and products of this contract shall become government property. Due Date, After the Deliverable Award of this Contract Task Deliverable Due Date, After the Contract's Beginning Date Estimated Due Date # of Copies Task 1: Update baseline management plan. 1.1 Meeting summary 5 working days September 5, 1997 3 1.2 Revise Baseline Management Plan 2 weeks Sept 12,1997 3 1.3 Approval of Baseline Management Plan Final Plan 4 weeks Sept 26,1997 3 1.4 Provide quarterly management updates Once every quarter 3 Task 2: Provide technical assistance on the evaluation through telephone calls and written communications. Include the number of calls received or made and the topic of the inquiries in the monthly report Monthly Management Report by the tenth day of each month 3 Task 3: Establish Performance Measurement System Monthly Management Report by the tenth day of each month 3 Task 4: Develop overview booklet on this project Draft overview booklet Final overview booklet 5 weeks 9 weeks October 3, 1997 October 31, 1997 3 5 Task 5: Establish an expert work group 5.1 Proposed list of work group mem. Selection of work group members 2 weeks 6 weeks Sept 12, 1997 October 10, 1997 3 3 5.2 Fall 1997 meeting Draft presentation materials Final materials Summary of meeting 7 weeks 11 weeks 15 weeks October 17, 1997 Nov. 14, 1997 Dec. 12, 1997 3 3 3 5.3 Winter 1998 meeting Draft presentation materials Final materials Summary of meeting 19 weeks 23 weeks 27 weeks January 9, 1998 February 6, 1998 March 6, 1998 3 3 3 5.4 Coordinate independent review of draft 1998 Interim Report Review comments to COTR 65 weeks Nov. 27, 1998 3 5.5 Fall 1998 meeting Draft presentation materials Final materials Summary of meeting 59 weeks 63 weeks 66 weeks October 16, 1998 Nov. 13, 1998 December 4, 1998 3 3 3 5.6 Spring 1999 meeting Draft presentation materials Final materials Summary of meeting 84 weeks 88 weeks 91 weeks April 9, 1999 May 7, 1999 May 28, 1999 3 3 3 5.7 Coordinate independent review of draft 1999 Interim Report Review comments to COTR 119 weeks Dec. 10, 1999 3 5.8 Fall 1999 meeting Draft presentation materials Final materials Summary of meeting 111 weeks 115 weeks 118 weeks October 15, 1999 Nov. 12, 1999 December 3, 1999 3 3 3 5.9 Spring 2000 meeting Draft presentation materials Final materials Summary of meeting 136 weeks 140 weeks 143 weeks April 7, 2000 May 5, 2000 May 26, 2000 3 3 3 5.10 Coordinate independent review of draft 2000 Interim Report Review comments to COTR 171 weeks December 8, 2000 3 5.11 Spring 2001 meeting Draft presentation materials Final materials Summary of meeting 189 weeks 193 weeks 196 weeks April 13, 2001 May 11, 2001 June 1, 2001 3 3 3 5.12 Coordinate review of final report Review comments to COTR 223 weeks December 7, 2001 3 Task 6: Review data collection forms and evaluation design for clarification Question and comments 2 weeks September 12, 1997 3 Task 7: Adapt computer system for State and local project universe data entry 7.3 Demonstrate and deliver data entry system Demonstration to COTR Submit diskette 14 weeks 18 weeks December 5, 1997 January 2, 1998 3 Task 8: Update the data entry system user guide and prepare a State version. Draft guides Final guides 18 weeks 21 weeks January 2, 1998 January 23, 1998 2 2 Task 9: Conduct annual evaluation conference and training program for local Even Start project directors and evaluators and State Even Start coordinators 9.2 Submit agenda, invitation letter, and proposed site Propose hotel Draft agenda/invitation Letter of invitation 10 weeks 14 weeks 21 weeks November 7, 1997 December 5, 1997 January 23, 1998 1 3 1 9.4 Memo reporting on conference 37 weeks May 15, 1998 3 9.5 Future evaluation conferences Propose hotel/dates Draft agenda/invitation Mail letter of invitation Memo reporting on conference AprAug 98, 99, 00 MaySpt 98, 99, 00 AugDec 98, 99, 00 within 3 weeks of each year's conference 1 3 1 3 Task 10: Develop and maintain tracking system of projects 10.2 Develop tracking system Copy of tracking system to COTR 6 weeks October 10, 1997 2 Task 11: Provide projects with new grantees package 11.1 Prepare evaluation instructional mailing packages Draft package Final package 4 weeks 6 weeks Sept 26, 1997 October 10, 1997 3 1 Task 12: Receive, edit, and enter July data submitted by local projects Update the COTR bi-weekly during the data collection period 3 Task 13. Develop materials to implement an experimental design study on effectiveness 13.1 Develop design Draft design plan 7 weeks October 17, 1997 3 13.2 Prepare draft design Draft design Revised design 19 weeks Dec. 12, 1997 January 9, 1998 3 3 13.3 Prepare final design March 6, 1998 5 13.4 Prepare and support OMB clearance package Draft package Final package 19 weeks 28 weeks January 9, 1998 March 13, 1998 3 5 Task 14: Select Even Start projects for randomized study and develop data collection protocol 14.1 Select projects List of projects/draft letter Final list/letter/data collection schedules 43 weeks 47 weeks June 26, 1998 July 24, 1998 3 3 14.2 Implement data collection Confirmation letter 50 weeks August 14, 1998 3 14.3 Prepare a study design paper for the experimental design study 156 weeks August 25, 2000 5 14.4 Maintain database of participants in the experimental design study Updated database upon completion of contract deliver upon completion of contract 5 Task 15: Update analysis and evaluation plan Draft plan Final plan 10 weeks 13 weeks November 7, 1997 November 28, 1997 3 3 Task 16: Prepare interim reports and summary database 16.2 Prepare interim reports Draft outline Outline First draft Second draft Final draft Final report June '98, '99, '00 June '98, '99, '00 Oct 15 '98, '99, '00 Nov20 '98, '99, '00 Jan 8 '99, '00, '01 Feb 19 '99, '00, '01 3 3 5 10 10 100 16.3 Prepare summary database Submit summary database with final interim reports 5 Task 17: Prepare final report and summary database 17.1 Prepare final report Draft outline Outline First draft Second draft Final draft Final report June 2001 June 2001 Oct 15, 2001 Nov 20, 2001 Jan 8, 2002 Feb 19, 2002 3 3 5 10 10 100 17.2 Prepare summary database Submit summary database with final report 5 Task 18: Develop and produce individual project profiles 18.1 Conduct survey Summary of results 50 weeks August 14, 1998 3 18.2 Produce draft prototype report Draft report 56 weeks Sept 25, 1998 3 18.3 Produce final prototype report 60 weeks October 23, 1998 3 Task 19: Produce Special Analysis of Migrant Projects 19.1 Produce outline Outline 54 weeks Sept 11, 1998 3 19.2 Produce draft of Special Analysis of Migrant Projects Draft report Nov'98,'99,'00,'01 5 19.3 Produce final draft report Final draft Jan'99,'00,'01,'02 10 19.4 Produce final report Mar '99, '00, '01, '02 20 Task 20: Produce Special Analysis of Projects for Indian tribes and tribal organizations 20.1 Produce outline Outline 54 weeks Sept 11, 1998 3 20.2 Produce draft of Special Analysis of Tribal Projects Draft report Nov'98,'99,'00,'01 5 20.3 Produce final draft report Final draft Jan'99,'00,'01,'02 10 20.4 Produce final report Mar '99, '00, '01, '02 20 Task 21: Review and update Even Start program performance indicators Draft performance indicators 222 weeks November 30, 2001 Final performance indicators 229 weeks January 18, 2002 5 10 Task 22: Recommend changes to the Even Start national evaluation design System change memo Recommended changes Draft redesign Revised redesign 111 weeks 136 weeks 2 wks after mtg August 1998 October 15, 1999 April 7, 2000 May 26, 2000 5 5 5 5 Task 23: Edit and test forms (option) 23.1 Revise forms Hard copy 146 weeks June 16, 2000 3 23.2 Test forms Deliver revised forms 152 weeks July 28, 2000 5 Task 24: Prepare and support IMCD/OMB forms clearance request (option) Draft forms clearance request Forms clearance request 155 weeks 157 weeks August 18, 2000 September 1, 2000 3 5 Task 25: Update the data collection forms, computerized system, and user's manual (option) Demon. of data entry system Draft user's manual Final data entry system Final forms and user's manual (Time after OMB approval) within 8 weeks within 8 weeks within 12 wks within 12 wks 2 2 2