IMPORTANT NOTE: THIS TEXT VERSION OF THE STATEMENT OF WORK HAS BEEN CREATED SOLELY TO MAKE MORE INFORMATION ABOUT THIS PROCUREMENT VIEWABLE ONLINE. INFORMATION/FORMATTING MAY HAVE BEEN LOST IN THE CONVERSION TO ASCII TEXT. ATTACHMENT A STATEMENT OF WORK National Household Education Survey TABLE OF CONTENTS I. INTRODUCTION AND BACKGROUND . . . . . . . . . . . . . . . .1 A. PURPOSE AND NATURE OF THE PROCUREMENT1 B. BACKGROUND1 B.1 Enabling Legislation. . . . . . . . . . . . . . . . .1 B.2 National Household Education Survey (NHES) Program History1 B.3 Completed NHES Collections. . . . . . . . . . . . . .2 B.3a 1989 Field Test2 B.3b NHES:913 B.3c NHES:933 B.3d NHES:954 B.3e NHES:965 C. GENERAL FEATURES OF THE NHES . . . . . . . . . . . . . . . . .5 C.1 Multiple Components in Each Collection. . . . . . . . . . . . .6 C.2 Sampling through Random Digit Dialing . . . . . . . . . . . . .6 C.3 Sampling to Produce Greater Minority Representation . . . . . .6 C.4 Sampling with Intent to Assess Change Over Time . . . . . . . .7 C.5 Use of Computer-Assisted Telephone Interviewing (CATI) Techniques . . . . . . 7 C.6 CATI Interviews in Spanish. . . . . . . . . . . . . . . . . . .8 C.7 Fast Turnaround . . . . . . . . . . . . . . . . . . . . . . . .8 C.8 Commitment to Data Quality. . . . . . . . . . . . . . . . . . .8 C.8a Cognitive Laboratory Work. . . . . . . . . . . .8 C.8b Field Tests. . . . . . . . . . . . . . . . . . .8 C.8c Maximizing Response Rates. . . . . . . . . . . .9 C.8d Reinterviews . . . . . . . . . . . . . . . . . .9 C.9 Technical Review Panels . . . . . . . . . . . . . . . . . . . 10 C.10 Adherence to NCES Standards . . . . . . . . . . . . . . . 10 D. GENERAL PLANS FOR THE NHES:99. . . . . . . . . . . . . . . . 11 EXHIBIT 1: Preliminary List of NHES Indicators . . . . . . . . . 13 II. SCOPE OF WORK. . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Task 1. Project Planning and Management . . . . . . . . . . . . . 15 1.1 Contractor Review of the NHES Background Materials. . . . . 15 1.2 Initial Meeting with NCES . . . . . . . . . . . . . . . . . 15 1.3 Project Management Plan . . . . . . . . . . . . . . . . . . 15 1.4 General Project Management and Monthly Progress Reports . . . . .15 1.5 Other Meetings with NCES. . . . . . . . . . . . . . . . . . 16 1.6 Briefings Materials and Briefings . . . . . . . . . . . . . 16 1.7 Project Brochure. . . . . . . . . . . . . . . . . . . . . . 16 1.8 Project Bibliography. . . . . . . . . . . . . . . . . . . . 17 1.9 Data Security Plan. . . . . . . . . . . . . . . . . . . . . 17 Task 2. Technical Review Panel. . . . . . . . . . . . . . . . . . 18 2.1 Develop TRP Plan. . . . . . . . . . . . . . . . . . . . . . 18 2.2 Solicit Participation in TRP. . . . . . . . . . . . . . . . 18 2.3 Plan and Arrange TRP Meetings . . . . . . . . . . . . . . . 18 2.4 Provide Meeting Summaries . . . . . . . . . . . . . . . . . 19 2.5 Administer Other TRP Activities . . . . . . . . . . . . . . 19 Task 3. Develop Survey Instruments and Procedures . . . . . . . . 19 3.1 Identify Published Indicators . . . . . . . . . . . . . . . 19 3.2 Consult with Customers of Key Indicators. . . . . . . . . . 19 3.3 Review Designs and Instruments of Extant Surveys. . . . . . 20 3.4 Prepare List of Research Questions to be Addressed and Content Outline. . . . . 20 3.5 Develop Survey Instruments. . . . . . . . . . . . . . . . . 21 3.6 Develop Data Collection Procedures. . . . . . . . . . . . . 21 3.7 Conduct Cognitive Laboratory Research . . . . . . . . . . . 22 Task 4. Sample Design . . . . . . . . . . . . . . . . . . . . . . 22 Task 5. Survey Design Report. . . . . . . . . . . . . . . . . . . 23 Task 6. IMT/OMB Clearance of the NHES Instruments . . . . . . . . 24 Task 7. Develop CATI System . . . . . . . . . . . . . . . . . . . 25 7.1 Specification of Range and Logic Checks . . . . . . . . . . 25 7.2 Programming of Survey Instruments . . . . . . . . . . . . . 26 7.3 Testing of the CATI System. . . . . . . . . . . . . . . . . 26 7.4 Delivery of Testable CATI Instrument. . . . . . . . . . . . 26 Task 8. Field Test of Survey Instruments and Procedures . . . . . 26 8.1 Submit Field Test Plan. . . . . . . . . . . . . . . . . . . 26 8.2 Conduct Field Test. . . . . . . . . . . . . . . . . . . . . 27 8.3 Submit Field Test Report with Revised CATI Instruments. . . . . .27 8.4 Submit Design/Instrument Changes in IMT/OMB Memo. . . . . . 27 8.5 Submit Final CATI Instruments in English and Spanish. . . . . . .28 Task 9. Hiring and Training CATI Interviewers . . . . . . . . . . 28 9.1 Develop Interviewer Training Materials. . . . . . . . . . . 28 9.2 Recruitment of Interviewers . . . . . . . . . . . . . . . . 29 9.3 Interviewer Training Sessions . . . . . . . . . . . . . . . 29 Task 10. Data Collection . . . . . . . . . . . . . . . . . . . . . 30 10.1 Data Collection Schedule. . . . . . . . . . . . . . . . . . 30 10.2 Quality Control Procedures. . . . . . . . . . . . . . . . . 30 10.3 Progress Reports. . . . . . . . . . . . . . . . . . . . . . 31 Task 11. Data File Preparation and Documentation . . . . . . . . . 31 11.1 Data Editing and Coding . . . . . . . . . . . . . . . . . . 31 11.2 Data Conversion . . . . . . . . . . . . . . . . . . . . . . 33 11.3 Creation of Composite and Classification Variables. . . . . 33 11.4 Item Imputation . . . . . . . . . . . . . . . . . . . . . . 33 11.5 Sample Weights. . . . . . . . . . . . . . . . . . . . . . . 34 11.6 Standard Errors . . . . . . . . . . . . . . . . . . . . . . 34 11.7 Public Release Files and User's Manuals . . . . . . . . . . 35 11.8 Adjudication of Datasets. . . . . . . . . . . . . . . . . . 36 11.9 Disclosure Review Board . . . . . . . . . . . . . . . . . . 36 11.10 Restricted-use Data Files and User's Manuals. . . . . . . 37 11.11 Maintenance of Necessary Data Files . . . . . . . . . . . 37 Task 12. Data Analysis and Reporting . . . . . . . . . . . . . . . 37 Task 13. Methodology Report. . . . . . . . . . . . . . . . . . . . 39 III. STUDY REQUIREMENTS. . . . . . . . . . . . . . . . . . . . . . . . 41 A. REPORTING . . . . . . .41 B. SCHEDULE OF DELIVERABLES . . . . . . . . . . . . . . . . . . 41 EXHIBIT 2: Schedule of Deliverables for NHES:99. . . . . . . . . 42 C. EXPECTATIONS CONCERNING QUALITY OF, CORPORATE SUPPORT FOR, AND TIMING OF DELIVERABLES . . . . . . . . . . . . . 45 D. THE ADJUDICATION PROCESS . . . . . . . . . . . . . . . . . . 45 E. FORMS CLEARANCE. . . . . . . . . . . . . . . . . . . . . . . 45 F. ELECTRONIC COMMUNICATIONS. . . . . . . . . . . . . . . . . . 46 APPENDIX A. NHES PUBLICATION LIST . . . . . . . . . . . . . . . . . . 47 APPENDIX B. NHES:95 OPEN-ENDED ITEMS. . . . . . . . . . . . . . . . . 52 I. INTRODUCTION AND BACKGROUND A. PURPOSE AND NATURE OF THE PROCUREMENT The National Center for Education Statistics (NCES) of the Office of Educational Research and Improvement (OERI), United States Department of Education (ED), requires a contract to continue the National Household Education Survey (NHES) program. This procurement includes the design and conduct of the 1999 National Household Education Survey (NHES:99). The NHES:99 will be an eclectic collection in which as many as possible of the key indicators reported from the NHES program are repeated as measures of the decade's progress. The NHES:99 requires random digit dialing (RDD) sampling methods and computer assisted telephone interviewing (CATI) techniques. Large amounts of data need to be collected, processed, analyzed, and reported under tight time schedules. The period of performance for the contract shall be 30 months. It will be incrementally funded. B. BACKGROUND B.1 Enabling Legislation The National Household Education Survey (NHES) program was established by the NCES in response to the immediate and continuing need for quality and timely data pertaining to the condition of education in the United States. Specifically, the NHES is undertaken in compliance with the mandate stated in section 404 of the National Center for Education Statistics Act, P.L. 103-382 (20 USC, 9003): "The duties of the Center are to collect, analyze, and disseminate statistics and other information related to education in the United States and in other nations..." B.2 National Household Education Survey (NHES) Program History The National Center for Education Statistics (NCES) more often collects data through school-based surveys of teachers, students, and schools, and through its administrative records surveys of school districts, and state education agencies. Data collected in this manner have been the foundation of the NCES' program to fulfill its legislative mandate to collect and report information on the condition of education in the United States. The collection of data from noninstitutional samples of individuals, particularly household-based data collections, was limited prior to the addition of the NHES program. The NHES is a vehicle for collecting detailed information on educational issues from a relatively large and targeted sample of households in a timely fashion. It fills a need that existing household surveys, such as the Current Population Survey (CPS), sponsored by the Bureau of Labor and the Bureau of the Census, and the Survey of Income and Program Participation (SIPP), sponsored by the Bureau of the Census, cannot satisfy because such surveys were designed to focus on issues other than education. If data are collected on education issues, it is usually done as a supplement to the main surveys. The level of detail in these supplements is often extremely limited because the sponsoring agency has practical constraints in obtaining the amount of data needed to address the basic issues of the survey as well as any supplementary data. As a result, data collected in this manner have often failed to provide NCES with the level of detail needed for desired analyses. In addition, because the existing sample designs of the main survey are established to meet specific objectives, it is not possible to target them to meet the design objectives associated with the educational issues. Consequently, sample sizes are sometimes inadequate for the applications required by NCES. NCES added the NHES to its repertoire of programs in 1988, with the award of a contract to conduct a field test collection in 1989. An option was exercised under that contract to conduct the first full-scale NHES in 1991 (NHES:91). A second competitively awarded contract covered the conduct of the NHES:93, the NHES:95, and the NHES:96. That contract ended after a 60 month period of performance. B.3 Completed NHES Collections One large-scale field test and four full-scale NHES collections (NHES:91, 93, 95, and 96) have been completed to date. The topics addressed by these collections are discussed in the following sections. Detailed information about the design of these collections can be found in the papers and reports listed in Appendix A. B.3a. 1989 Field Test A field test of the NHES as a methodology for collecting education data was designed and conducted in the fall of 1989. Data on two topics of high priority to the NCES were collected during the field test--school dropouts and early childhood education. Other than the size of the sample, the field test included all of the design features of a full-scale NHES. This included the use of random digit dialing (RDD) and computer assisted telephone interviewing (CATI) methods. The findings from the field test were encouraging for the future of the NHES. The field test demonstrated that many of the concerns surrounding the use of a telephone survey to study education issues could be handled adequately in practice. It also revealed that each survey topic has its own unique set of circumstances that must be explored individually. B.3b. NHES:91 NCES implemented the first full-scale NHES in the spring of 1991. The topics selected for the NHES:91 were early childhood education and adult education. The NHES:91 early childhood education component was an expansion of the 1989 early childhood field test topic. The age range of the children covered by the survey was extended from 3- to 5-year-olds to 3- to 8-year-olds. Information on the early childhood education experiences similar to that collected in the field test was collected for children not yet enrolled in first grade. The focus was on non-parental care and education, characteristics of programs and care arrangements and activities children engaged in with parents and other family members. For those children enrolled in primary school, the survey focused more heavily on: educational experiences to date; delayed entry into kindergarten and first grade; retention; and parental involvement in schooling. The second topical component of the 1991 survey concerned the educational activities of members of the U.S. adult population. Specifically, the survey collected up-to-date information on the participation of persons 16 years and older in a wide array of adult education activities. This survey component was based in large part on the Current Population Survey (CPS) supplement on adult education, last conducted in 1984. Data were collected on the numbers and types of courses in which adults had participated over the previous 12 months. For the four most recent courses, information was collected on course content, provider, sources of payment, and reason for taking the course. The NHES adult education component, unlike the adult education supplement to the CPS, also asked questions of nonparticipants. These questions focused on the need for adult education, its availability, and barriers to participation. In the NHES:91, 120,000 telephone numbers were dialed, and screening interviews were completed in about 60,500 households. One or more topical interviews were completed in about 21,500 of the households. Altogether, about 14,000 early childhood interviews and 12,500 adult education interviews were conducted. The early childhood interviews were conducted with the parent or guardian identified by the screener respondent as being the most knowledgeable about the sample child. The adult education interviews were conducted with the sampled adults. B.3c. NHES:93 Topics covered in the NHES:93 were School Readiness and School Safety and Discipline. The population of interest in the School Readiness component was children age 3 through age 7 and others enrolled up through 2nd grade. Issues addressed were: developmental characteristics of preschoolers, school adjustment and teacher feedback to parents for kindergartners and primary students, center-based program participation, early school experiences, home activities and health status. Extensive family and child background characteristics, including parent language and education, income, receipt of public assistance and household composition, were collected to permit the identification of at-risk children. The population of interest in the School Safety and Discipline component was children enrolled in grades 3 through 12. Parents of these children were interviewed about school learning environment, discipline policy, safety at school, victimization, the availability and use of alcohol/drugs, and alcohol/drug education. Peer norms for behavior in school and substance use were also included. A subsample of the children in grades 6 through 12 were also interviewed in the School Safety and Discipline component. The youths were asked essentially the same items as their parents. However, parents provided the extensive family and household background information as well as characteristics of the school attended by the child. In the NHES:93, about 130,000 telephone numbers were dialed, and screening interviews were completed in about 64,000 households. One or more topical interviews were completed in about 19,500 of the households. Altogether, about 11,000 school readiness interviews and 12,500 school safety and discipline interviews were conducted with knowledgeable parents or guardians. Also, about 6,500 school safety and discipline interviews were conducted with 6th through 12th grade youth. B.3d. NHES:95 For the most part, topics included in the NHES:95 were a repeat of those included in the NHES:91, early childhood education and adult education. The early childhood component focused a bit more on characteristics of non-parental care and education arrangements and thus was renamed early childhood program participation. For the first time, the population of interest included infants and toddlers, and interviews were conducted with parents or guardians of children from birth through 3rd grade, up to and including age 10. The NHES:95 adult education questionnaire was different from the questionnaire used in 1991. The earlier questionnaire was closely patterned after that used in the 1984 CPS supplement on adult education. Intervening cognitive lab work suggested that a different questionnaire structure might better capture the totality of adult education activities. To preserve the trend measure and to allow for crosswalk between the 1991 and 1995 estimates of adult education participation, a sample of adults in 1995 received the 1991 version of the adult education participation items. The NHES:95 also included a small methodological study to test the feasibility of expanding the screening interview to allow for collection of data from the complete sample of households. In the NHES:95, about 120,000 telephone numbers were dialed, and screening interviews were completed in about 45,500 households. One or more topical interviews were completed in about 31,500 of the households. Altogether, about 14,000 early childhood interviews and 19,500 adult education interviews were conducted. The early childhood interviews were conducted with the parent or guardian identified by the screener respondent as being the most knowledgeable about the sample child. The adult education interviews were conducted with the sampled adults. B.3e. NHES:96 The NHES:96 covered the topics of family involvement in education; civic and community involvement among adults and youth; and household library use. Children from age 3 through 12th grade were sampled for the parent involvement component. The most knowledgeable parent was interviewed about the child's school experiences, family involvement in school and schoolwork, school practices to involve families, and family involvement outside of school. Questions were also included about the involvement of the non-custodial parent. For adults, the civic and community involvement component included questions about exposure to national news, participation in community and political activity and political attitudes and knowledge. These questions were blended in with the parent involvement items for the parents of 6th through 12 graders and some emphasis was given to how the family models these involvement behaviors to the youth. For youth in grades 6 through 12, the civic and community involvement component included many of the same items asked of adults, but also items that captured involvement in activities that promote or indicate personal responsibility, including service learning activities. All households contacted in the NHES:96 were asked to respond to a brief set of items about household use of public libraries. In households in which no persons were sampled for either of the two larger topical components, the library items followed the screening items. In households in which someone was sampled for the more extensive topical interviews, the library items appeared as part of the topical interview. In the NHES:96, about 160,000 telephone numbers were dialed, and screening interviews were completed in about 55,000 households. Altogether, about 21,000 Parent Involvement in Education/Parent Civic Involvement interviews, 8,000 Youth Civic Involvement interviews, and 2,250 Adult Civic Involvement interviews were completed. C. GENERAL FEATURES OF THE NHES The NHES is a telephone survey of the noninstitutionalized civilian population of the United States. Households are selected for the survey using random digit dialing (RDD) methods. Data are collected using computer assisted telephone interviewing (CATI) techniques. The methodology for any single fielding of the NHES is linked to the research issues under study, the level of data required to address these issues, and how precise the estimates generated from the survey data need to be in order to meet the objectives of the study. However, while the specifications for each survey will vary, there are general features of the NHES methodology that will stay relatively constant from one survey to the next. The general characteristics of the NHES sample design and the data collection approach are described in the next sections. C.1. Multiple Components in Each Collection Because of the high costs associated with screening large numbers of households in order to meet the sample size requirements of NHES components concerning young children, more than one population and set of issues has been addressed concurrently. Each of NHES:91, NHES:93 and NHES:95 included two topical components; the NHES:96 included two larger components and one smaller component imbedded within the screening interview. The components selected and the final sampling designs have sought to maximize the probability of a household qualifying for inclusion in the survey, but limit the burden placed on each household. (More details about these design features can be found in Overview of the National Household Education Survey 91/93/95/96, NHES Technical Report, and in User's Manuals and working papers for specific collections. These documents are fully referenced in Appendix A.) C.2. Sampling through Random Digit Dialing Random digit dialing (RDD) has been used for each NHES collection. However, the particular method of RDD utilized has varied. For the NHES:91 and the NHES:93, a modified Mitofsky-Waksberg method described by Brick and Waksberg (1991) was used. A list-assisted method was used in the NHES:95 and again in the NHES:96. The list-assisted method eliminates the sequential difficulties associated with the modified Mitofsky-Waksberg in that it is a single-stage sample. It also produces an unclustered and self-weighting sample. C.3. Sampling to Produce Greater Minority Representation One of the goals of the NHES program is to produce reliable estimates of the characteristics of children's and adults' educational experiences for totals and for subdomains defined by race and ethnicity. In each NHES, Census information about telephone numbers has been used to increase the likelihood of calling into minority households. In the NHES:95, for example, 100-banks were classified as "high minority" if at least 20 percent of its population was black or at least 20 percent of its population was Hispanic. In NHES:93, "high minority" was defined as having a population at least 20 percent black, Hispanic or Asian. In both collections, "high minority" numbers were sampled at twice the rate as "low minority" numbers. "Asian/Pacific Islander" was included in the 1993 definition because design work suggested that a definition based on only blacks and Hispanics tended to depress the number of Asians and Pacific Islanders encountered. Adding "Asian/Pacific Islander" to the definition was intended to increase the numbers of blacks and Hispanics without hurting the number of Asians. It was not possible to use this same approach in the NHES:95 or NHES:96 because the sample was purchased from a provider that did not include information about Asians/Pacific Islanders on the sample frame. However, the approach did not turn out to be as promising as the design work had indicated. C.4. Sampling with Intent to Assess Change Over Time The NHES program was conceived of as a program that would provide trend information on important educational indicators. Thus, each component has been designed with the intent of assessing change over time. Estimates by race and ethnicity are of great interest especially for monitoring educational trends over time. C.5. Use of Computer-Assisted Telephone Interviewing (CATI) Techniques The NHES has several features that require the use of a computer-assisted telephone interviewing (CATI) methodology. First, the topics covered by the NHES are complex and difficult skip patterns often arise. Consequently, the types of questionnaires that are developed to collect information on such detailed topics as child care or preschool arrangements for young children would be difficult to administer by telephone without the aid of a CATI system. Second, as noted earlier, the NHES will usually cover more than one topic and require sampling from more than one population. Here again, it seems unrealistic to implement such a design without the assistance of an effective and efficient CATI system. Third, one of the features of the NHES that is important to NCES is the timely dissemination of the data gathered by this system. The use of a CATI system shortens the amount of time devoted to interview followup, data processing and editing, and also hastens release of public-use data files and statistical reports. All interviews conducted as a part of the NHES shall use a CATI methodology. The CATI system developed for the NHES shall perform the necessary functions associated with: 1. sampling the appropriate households and individuals for interviews; 2. scheduling interviews -- the CATI shall provide the telephone numbers online to interviewers to attempt initial contacts, maintain a list and schedule of callback appointments, and establish a schedule for attempting to contact numbers when initial attempts at contact fail; 3. guiding the interviewer through the questionnaire based on the data requirements for given classes of respondents -- the programming of skip patterns will reduce the number of interviewer errors and improve the overall quality of the survey data; 4. monitoring and reporting the progress of the data collection on a daily and weekly basis; 5. coding and editing of the interview data -- most of the coding of the interview data shall be accomplished by the CATI system during the conduct of the interview. However, there may be exceptions to this. Editing of the answers provided by respondents shall be performed during the interview and at a minimum shall include both range and consistency checks. C.6. CATI Interviews in Spanish Because NHES oversamples Hispanics, a number of households that are contacted through regular CATI methods require a Spanish-language version in order to complete the interview. In order to follow the complex skip patterns, a Spanish-language version of the CATI instrument must be available for interviewers who can conduct interviews in Spanish. C.7. Fast Turnaround Historically, NCES has been able to release an adjudicated "Statistics in Brief" report on each dataset prior to the end of September of the data collection year. Thus, data cleaning, imputation, weighting, analysis, report writing, and report adjudication have been accomplished within 5 months of the completion of each data collection. C.8. Commitment to Data Quality Since its inception, the NHES program has dedicated resources to both assessing and improving the quality of the data. In the sections that follow are descriptions of data quality work that has taken place. C.8a. Cognitive Laboratory Work Cognitive laboratory work has been used extensively in the NHES program. Focus groups have been used at even the earliest stages of development to explore general perceptions concerning the topics. As questionnaires have begun to take shape, focus groups and one-on-one think-aloud or debriefing interviews have been used to help refine items as well as the general flow of the questionnaires. Multiple rounds of cognitive lab work have allowed for iterative testing of the instruments. (See Use of Cognitive Laboratories and Recorded Interviews in the National Household Education Survey, NHES Technical Report, as listed in Appendix A.) C.8b. Field Tests Field tests have been used to improve the quality of the data in all NHES collections. In the NHES:91 and NHES:93, due to limited planning time, field tests were done in the fall preceding the collection. Three field tests primarily served to ensure that the CATI program was operating according to specifications. In both NHES:95 and NHES:96, extensive field testing of the topical components was done in the spring prior to the collection year. Both times, three rounds of field tests were performed, though the sample size devoted to each round varied. In the NHES:96, the first phase was small, and the purpose was to test the overall flow of the questions, and to determine if there were any major problems with general wording. The second phase was larger and more attention was paid to individual items, including the appropriateness of response categories. The third phase was smaller and was done to ensure that changes made as a result of the earlier phases were properly programmed. Methodological tests, such as efforts to identify means for increasing response rates, have been incorporated into field testing as deemed necessary. C.8c. Maximizing Response Rates High response rates are critical to the NHES and all surveys because the potential for nonresponse bias increases as the response rate decreases. The most appropriate way to avoid serious nonresponse bias is to achieve high response rates. Response rates in the NHES are the product of the completion rate for the final stage of the survey and the completion rates for all earlier stages. The Screener completion rates in 1991 and 1993 were about 10 percentage points higher than the Screener completion rates in 1995 and 1996. In each survey year, most nonresponse to the Screener was accounted for by refusals, which comprised 68 to 84 percent of nonresponse. A screening experiment conducted in the field test for the NHES:96 revealed that a major reason for lower Screener response rates was the screening approach. The screen-out approach (an early eligibility item that allows many noneligible households to "opt out") results in Screener completion rates about 10 percentage points higher than either no screen-out or full household enumeration approaches. (See An Experiment in Random-Digit-Dial Screening, NHES Technical Report, as listed in Appendix A.) Other approaches to maximize response rates that have been fielded by NHES include: continuing to recycle telephone numbers known to be residential until a refusal is confirmed, or a completed interview or other outcome is confirmed; leaving messages on answering machines; sending a letter and brochure explaining the survey to potential nonrespondents; sending a letter to initial refusal cases eliciting their cooperation in a possible interview; and sending out a letter in advance of the data collection (done only in NHES:96). It should also be noted that the percentage of telephone numbers yielding residences is lower for the list-assisted method than the modified Waksberg-Mitofsky method (50-52 percent of numbers dialed are considered to be residential in the list-assisted sampling method versus 60-62 percent for Waksberg-Mitofsky). Thus, for NHES:95 and NHES:96, more telephone numbers had to be dialed in order to establish a sufficient number of potential respondents in the sample than for the Waksberg-Mitofsky method, which clusters the banks of telephone numbers. Achieving high levels of cooperation from the telephone numbers sampled can maximize the efficiency of the sample and reduce the amount of interviewer time. Strategies to maximize response rates are an essential part of future NHES collections. C.8d. Reinterviews Each NHES has included a reinterview survey in which a random sample of respondents has been called and re-asked a subsample of the items on the original interview to assess item reliability. The purposes of the reinterview have been to identify survey items that are not reliable and to quantify the magnitude of the response variance for items collected from the same respondent at two different times. The NHES:93 included reinterviews on both components; the NHES:91 and NHES:95 have included reinterviews with the early childhood and adult education components, respectively. (See The 1995 National Household Survey: Reinterview Results for the Adult Education Component, NHES Working Paper, as listed in Appendix A.) The NHES:96 reinterview included reinterviews on both parent and family involvement in the education of their children ages 3 through 5th grade, and civic involvement of parents and youth. Adults with no children in the specified age/grade range were not reinterviewed, under the assumption that reliability of items on civic involvement for adults would not be significantly different than for parents. C.9. Technical Review Panels Studies of this scope, complexity, and importance require input from a large number of individuals and organizations in order to address, in a technically competent and meaningful way, the data needs of policymakers and of those performing policy studies and educational research. Thus, the contractor for the program has worked with a number of Technical Review Panels (TRPs). Historically, a TRP has been formed for each topical component being covered by the NHES. TRPs have played an active role through reviewing and commenting on overall research priorities, identifying policy and research questions, providing input about questionnaire content, proposing analytical models and methods, reviewing work plans and their implementation, and reviewing and suggesting modifications to draft reports. C.10. Adherence to NCES Standards NCES has developed and implemented a set of standards that set guidelines to ensure the quality of NCES' work. These standards are to be followed by NCES staff and their contractors in performing the day-to-day work of the NCES. A copy of the NCES publication NCES Statistical Standards (U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics, June 1992 (NCES 92-021)) can be obtained by calling the National Education Data Resource Center at (703) 845-3151. A list of the relevant standards covered in this publication is provided below. Standard for Planning of Statistical Surveys Standard for the Design of a Survey Standard for Testing Survey Systems Standard for Managing Survey Operations Standard for Achieving Acceptable Response Rates Standard for Imputation of Item Nonresponse Standard for Computation of Response Rates Standard for Codes, Abbreviations and Acronyms Standard for Variance Estimation Standard for Analysis and Statistical Comparisons Standard for Maintaining Confidentiality Standard for Tabular and Graphical Presentations Standard for Dissemination of Survey Data and Reports Standard for Timely Processing and Release of Data and Data Tapes Standard for Maintaining Data Series Standard for Machine Readable Products Standard for Evaluation of Survey Standard for Documenting a Survey System Standard for Survey Documentation in Center Reports The contractor should be aware that NCES is reviewing its standards and policies. Should any standard change prior to contract award, the contractor shall be informed. D. GENERAL PLANS FOR THE NHES:99 NHES data from previous collections have been reported as important indicators of children's and adults' educational status. For the most part, the NHES:99 is envisioned to be a composite collection of previously-administered educational indicators. For these indicators, it will provide the second, third, fourth, or fifth point on a trend line. Shown as Exhibit 1 is a preliminary list of indicators that have been reported by NCES and/or NCES' customers. In the NHES:99, breadth of coverage in topics and in populations to be sampled will necessitate less depth of coverage. One of the greatest challenges of the NHES:99 will be to develop instruments that flow and make sense to the respondents. Previous NHES collections have dealt with topics in greater depth and groups of items were designed around common themes. While the indicators will have highest priority in the NHES:99, it will be possible to address a limited number of additional research questions. Between the items needed for the indicators and the items needed to address the additional research questions, it is felt that a cohesive, cost effective whole can be found. The government anticipates that the NHES:99 will include a basic screener, as opposed to the expanded screener that was implemented for the first time in NHES:96. At the same time, the government does not anticipate any reinterview efforts in the NHES:99. These features are not being considered primarily because of cost considerations. EXHIBIT 1. Preliminary List of NHES Indicators Population: Children 3 to 5 years, not yet in Kindergarten (Parent respondents) Indicators: Reading Storytelling Center-based participation, by income group Center-based participation of disabled Medical care within the previous 12 months, by income group Dental care within the previous 12 months, by income group Reading, by parents' highest education Storytelling, by parents' highest education Songs or music, by parents' highest education Arts and crafts, by parents' highest education Play or concert attended, by parents' highest education Errands or chores, by parents' highest education Talked about family history, by parents' highest education Attended event, by parents' highest education Center-based participation of disabled children, by parents' highest education Center-based participation of disabled children, by household income Center-based participation of children, by household demographic characteristics Population: Children 3 and 4 years (Parent respondent) Indicator: Enrollment rate in center-based child care, by household characteristics Population: Children in Kindergarten through grade 12 (Parent respondents) Indicators: Parent involvement in school, by child's grade Parent participation in specific school activities Parent involvement in child's academic activities Perceptions of quality of child's school performance Reports of specific school communications Grade retention, by demographic characteristics School type (choice), by demographic characteristics Choice of where to live, by demographic characteristics Population: Children in grades 6 through 12 (Youth respondents) Indicators: Ease of obtaining drugs/alcohol at school, by school level Ease of obtaining drugs/alcohol at school, by school size Ease of obtaining drugs/alcohol at school, by urbanicity Student membership in gangs, by student race/ethnicity Student membership in gangs, by school size Student membership in gangs, by urbanicity Community service, by demographic characteristics Peers do not approve of drugs/alcohol, by demographic characteristics Population: Civilians, 16 years +, not enrolled at elem./secondary level (Adult respondent) Indicators: Adult ed. participation, by educational attainment Adult ed. participation, by labor force status Adult ed. participation by age Adult ed. participants by occupation Participation in Adult Basic Education/GED, by demographic characteristics Participation in credential program, by demographic characteristics Participation in work-related program, by demographic characteristics Participation in personal education program, by demographic characteristics Adult reading of newspapers Adult reading of magazines Adult reading of books II. SCOPE OF WORK A number of activities are required to successfully design and implement the NHES:99 data collections. These activities are described below as a series of tasks and subtasks. Except as noted, the contractor shall allow two weeks for NCES review of a draft deliverable. One week shall be allowed for the contractor to respond to any requests for revisions. Task 1. Project Planning and Management 1.1 Contractor Review of NHES Background Materials. Immediately following award, the Contracting Officer's Technical Representative (COTR) will provide project materials from the earlier studies (i.e., 1989 field test, NHES:91, NHES:93, NHES:95, and NHES:96), including the NHES technical reports and data files. To take advantage of the experience gained during the prior NHES surveys and to ensure that the new studies will be comparable in important ways with previous NHES surveys, the contractor shall review these reports, and data files. 1.2 Initial Meeting with NCES. On or before October 10, 1997, the contractor shall meet with NCES staff to review the study's tasks and to discuss issues related to the conduct of the work. This conference shall be considered part of the background review and as input for the project management plan. 1.3 Project Management Plan. The contractor shall prepare and submit a project management plan for the NHES:99. The plan shall describe and justify any revisions to the original plan specified in the contract. These might include changes in data collection approach, data processing, plans for analysis and reporting, as well as changes in the proposed project organization, staffing plan, and schedule. The plan shall fully describe the rationale for any proposed changes. The plan shall also detail the management techniques that will be employed to ensure timely, efficient, and cost-effective data collection, processing, analysis and reporting. A draft management plan for NHES:99 shall be submitted on or before November 3, 1997. NCES will provide comments on the plan within two weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by NCES and submit a revised plan on or before November 24, 1997. Final approval of the revised plan will be provided within two weeks of receipt of the final plan. 1.4 General Project Management and Monthly Progress Reports. Under this task, the contractor shall perform its general project management functions. A routine aspect of this management shall be the development and submission of regular monthly project reports. The monthly progress reports are described in Part III, Study Requirements, Section A. 1.5 Other Meetings with NCES. Key contractor staff shall be prepared to travel to NCES' offices in Washington, DC, for meetings throughout the contract period. It is anticipated that about six one-day meetings (other than the initial meeting described in Task 1.2 and any TRP meetings described Task 2) shall be needed over the course of the contract. The designation of attendees to each meeting shall be made only after the meeting goals and agenda have been established. An average of three of the contractor's staff members is typically in attendance at each meeting. The contractor shall participate in the development of the goals and agenda for each meeting and shall be responsible for distributing meeting materials, if any, to about 12 meeting attendees. Materials to be distributed shall be approved, in advance, by the COTR. 1.6 Briefing Materials and Briefings. From time to time throughout the course of the project, NCES will provide interested individuals and agencies with information about the nature, findings, and progress of the NHES. The contractor shall support these activities by developing up to four sets of briefing materials to be used by NCES for the survey. Each set of briefing materials shall be no more than 10 typewritten pages of text (double-spaced) with accompanying graphs and/or tables suitable for display (such as transparencies or slides). Within two weeks of a request, a set of briefing materials shall be submitted to NCES for review and comment. The contractor shall make necessary revisions to the materials based on this review and resubmit the materials within one week of receipt of comments. The contractor shall also be prepared to give up to six briefings and/or demonstrations about the NHES program. The COTR will give as much notice at possible for these briefings and/or demonstrations. Examples of the kinds of meetings at which the contractor might be asked to demonstrate NHES data are the Annual Meeting of the American Educational Research Association and NCES' Summer Data Conference. All materials prepared for these briefings shall also be approved, prior to use, by NCES. The contractor shall submit briefing materials for approval no less than 10 working days before the materials are to be presented. The contractor shall make revisions as appropriate and provide final copies to the COTR within one week of receipt of comments. 1.7 Project Brochure. The contractor, in consultation with NCES, shall prepare a brochure describing the NHES. A draft of the brochure shall be submitted no later than January 5, 1998. NCES shall review the brochure and provide comments and suggestions within two weeks. The contractor shall revise the materials based on NCES suggestions and submit camera-ready, color-separated copy of the materials no later than February 2, 1998. An electronic version of the brochure's text and graphics should be delivered with the camera-ready copy; the electronic version should be in a Web-compatible format in accordance with existing NCES guidelines. 1.8 Project Bibliography. Throughout the course of the contract, the contractor shall maintain a bibliography of reports and articles that cite any of the NHES surveys. The first bibliography shall be submitted for review no later than February 2, 1998. NCES will review the bibliography and provide comments within two weeks of receipt. The contractor shall revise the bibliography and submit one camera-ready, ten stapled copies, and an electronic file no later than March 2, 1998. This bibliography shall be updated on an annual basis thereafter to include citations of reports or articles that cite previous NHES surveys or the current NHES survey (once available). 1.9 Data Security Plan. In order to ensure the anonymity of individual respondents, the contractor must comply with Section 408 of the National Education Statistics Act of 1994, P.L. 103-382 (20 U.S.C. 9007). A copy of the Act is available upon request from ED's Grants and Contracts Service. The Act authorizes fines or imprisonment for disclosure of individually identifiable information for any purpose other than statistical purposes. Under no circumstances may the contractor release personally identifiable information. Information that identifies persons must be maintained in files which are physically separate from other research data and which are accessible only to sworn agency and contractor personnel. Individual identifiers used during the course of the project shall be associated with data only for purposes of data gathering, matching new data with old, establishing sample composition, authenticating data collections, editing data based on callbacks, or obtaining missing information. The contractor shall enforce strict procedures for ensuring confidentiality. These procedures shall apply to all phases of the project and should include but not be limited to: (1) data collection in the field; (2) coding and editing phases of data prior to machine processing; and, (3) safeguarding response documents, including CATI records. Any employee needing access to confidential information shall first swear to an Affidavit of Nondisclosure. A copy of the Affidavit is available upon request from ED's Grants and Contracts Service. The contractor shall execute these Affidavits of Nondisclosure and the originals shall be transmitted to the NCES COTR within 2 weeks of the award of the contract. As part of the transmission, the contractor shall indicate the position in the organization of the person signing the Affidavit of Nondisclosure, and the person's functional relationship to this project. As new persons are assigned to this project, an Affidavit of Nondisclosure shall be executed for them on the first working day of assignment to the project. Throughout the life of the contract, the Affidavits of Nondisclosure for new project staff, as well as interviewers and other short-term personnel shall be submitted on a schedule designated by the COTR or at a minimum of three times a year. All individuals asked to respond to the survey shall be informed of the following: NCES' enabling legislation; the purposes for which the information is needed; uses that may be made of the data; and the methods of reporting the data so that an individual's responses are not revealed. The contractor shall maintain security on the complete set (and deliverable backups) of all master survey files and documentation. The contractor shall present a detailed security plan that expands upon what was presented in the proposal to the NCES COTR for approval by no later than November 3, 1997. Task 2. Technical Review Panel The challenge for the NHES:99 is to develop a questionnaire, based on previous data items, that serves as an end-of-decade measure of educational indicators. This questionnaire will have to be developed very quickly, and the various topics should have a logical flow. The NHES:99 Technical Review Panel (TRP) should be set up to address these issues. The TRP for the NHES:99 indicators shall consist of no fewer than four persons who the contractor believes will have expertise to bring to the project, as the topics cross the spectrum of education areas (such as early childhood education to adult education). Candidates for the NHES:99 TRP should be drawn from data users of the indicators, to the extent possible. The role of each TRP shall be to review the technical and substantive issues associated with the NHES. 2.1 Develop TRP Plan. By no later than November 3, 1997, the contractor shall submit a plan to the COTR for the NHES:99 TRP. This plan shall contain a list of nominees of individuals with the needed expertise for possible membership on the TRP, shall detail a schedule of meetings, and outline the role of the TRP. NCES must approve the nominations or suggest alternate nominees. The contractor shall recommend a tentative schedule of panel meetings in keeping with the overall plans presented in its proposal. 2.2 Solicit Participation in TRP. Once NCES has approved an individual for membership, which shall take no more than four weeks, the contractor shall take no more than two weeks to contact the individuals to solicit participation. 2.3 Plan and Arrange TRP Meetings. All materials and correspondence required for the TRP's consideration and review must be developed and/or prepared for mailing. After NCES review and approval, those materials should be sent out to TRP members for review and comment. The contractor shall arrange for TRP meetings and pay all associated expenses. It is anticipated that the NHES:99 TRP shall convene for as many as three one-day meetings. Panelists will need one day, prior to each meeting, to review materials and to otherwise prepare for the meeting. All meetings shall be held in Washington, DC at the offices of the NCES. Non-government panelists should be paid an honorarium plus per diem and related expenses. The same rates would apply for other work panel members might perform. 2.4 Provide Meeting Summaries. The contractor shall tape record meetings and submit a written summary to NCES within two weeks of each meeting. The meeting summary shall not be a verbatim transcript of the meeting. Instead, it shall summarize the discussions and activities that took place during the meeting, highlighting major issues that were raised and decisions made. 21.Administer Other TRP Activities. The contractor shall arrange and pay for specific work products not tied to a meeting. For example, the NHES:99 TRP might be asked to comment on a technical report. Again, non-government panelists should be paid an honorarium. The contractor shall supply NCES with copies of all correspondence and other materials exchanged with panelists. Task 3. Develop Survey Instruments and Procedures Even though the NHES:99 "key indicators" are drawn from items that have been fielded previously, additional conceptual work (both substantive and methodological) is required to design the sample, data collection procedures, instruments, and analyses of the NHES:99. The contractor shall implement an approach to the design of the NHES:99 that follows the steps outlined in the following subtasks. 3.1 Identify Published Indicators. The contractor shall identify all published reports that contain indicators which utilize NHES data. Examples of such reports would include NCES' annual Condition of Education, the National Education Goals Panel's annual Goals Reports, the Trend Reports published by the Department of Health and Human Services' Assistant Secretary for Planning and Evaluation, the Summary Report published by the Interagency Forum on Child and Family Statistics, and Statistical Abstracts published by the Department of Commence. The contractor shall prepare a summary document that contains a listing of all previously published indicators, the variables needed to create those indicators, and full citations for where they appear. This summary document shall be submitted to NCES no later than November 3, 1997. 3.2 Consult with Customers of Key Indicators. The contractor shall consult with customers of previously fielded indicators to confirm the continued utility of these indicators, to determine interest in other possible indicators, and to establish priorities. Following completion of work on this task, the contractor shall prepare and submit a report no later than November 17, 1997. At a minimum, this report shall contain a written summary of each meeting. It shall also contain a summary of the meetings as a whole. This report should strive to present a balanced view of the indicators/issues presented by all of the various groups. In particular, the summary should highlight recommendations that emerge from more than one source or from consensus. 3.3 Review Designs and Instruments of Extant Surveys. The contractor shall review the designs and instrumentation of key studies and data collection programs that are related to the key indicators of NHES:99. The primary objective of this review is to collect information that will help to place the key indicators into perspective with respect to how NHES data has been used and other extant studies. The contractor shall prepare a summary of these studies emphasizing the areas of comparability and differences between these studies and the NHES. At a minimum, the summary shall contain detailed information on the purpose of each survey, its sample design, the inference population, the methods used to collect data pertaining to the education issues being addressed, the characteristics for which information is being sought, and the types of estimation/analyses the data collection supports. The summary shall also show how the NHES:99 complements other studies and extends the information based upon the key indicator topics being addressed. Finally, the summary of extant studies/data will provide the necessary background information needed to develop a plan for conducting a comparative analysis of estimates derived from the NHES. As a part of the analysis and reporting requirements associated with each NHES, the contractor shall compare estimates from the NHES with similar estimates derived from other studies (see Task 13). A draft of the summary of extant surveys shall be due no later than November 17, 1997. Following NCES review, the contractor shall make any revisions necessitated by NCES' comments and submit a revised summary of the extant surveys for NCES review and approval no later than December 8, 1997. 3.4 Prepare List of Research Questions to be Addressed and Content Outline. The contractor shall prepare and submit a list of research questions to be addressed and a content outline for the NHES:99 instrument(s) no later than December 1, 1997. The research questions shall be restatements of the indicators and other areas of inquiry deemed important for the collection. The content outline shall specify the areas of inquiry to be included along with a brief rationale for each. Following NCES review of the list of research questions and the outline, revisions that incorporate suggestions made by NCES shall be submitted no later than December 21, 1997. The approved list of research question and content outline shall be provided to the Technical Review Panel for review and comment. All comments from the TRP members shall be summarized by the contractor and presented to the NCES COTR no later than January 12, 1998. The list of research questions will undergo constant revision. As the instruments are developed, the list shall also indicate the questionnaire items that will be used to address each research question. Each submission of a questionnaire shall be accompanied by a revised research question list. In that way, as items are revised or deleted or as design changes are made, their impact on the research and policy questions to be addressed will be known. The contractor should plan on about four revisions to the research questions list. 3.5 Develop Survey Instruments. Draft copies of the survey instruments shall be submitted to NCES for review and comment no later than January 12, 1998. NCES will provide comments on the draft instruments within two weeks. Within one week of receipt of NCES' review comments, the contractor shall revise the instruments incorporating the comments and suggestions of NCES, and submit a second draft of the instruments no later than February 2, 1998. Following the cognitive laboratory research, discussed below, and no later than April 13, 1998, the contractor shall prepare a third draft of the instruments. Following NCES review and approval of this draft (within two weeks of its submittal), the contractor shall make revisions necessitated by NCES' review. 3.6 Develop Data Collection Procedures. Data collection procedures are the decision rules about how to handle various types of situations encountered in the field, such as how many Ring No Answers to allow before finalizing the case as a no contact, whether to use Advance Letters as an aid to reducing possible nonresponse, what type of message to leave on answering machines, the use of an "800" number for callbacks, and so forth. These procedures shall be documented by the contractor. For RDD surveys such as the NHES, NCES expects to achieve a minimum acceptable response rate of 70 percent. That overall response rate must reflect all stages of the interview: the household screener completion rate and the extended interview completion rate. The NHES method of calculating response rates is the "weighted business office method," in which some portion of the unknown residential status telephone numbers is included in the denominator. (See National Household Education Survey of 1995: Adult Education Data File User's Manual, and Unit and Item Response Rates, Weighting, and Imputation in the 1996 National Household Education Survey, listed in Appendix A, for more detail.) The proportion of unknown residential status telephone numbers to include in the denominator is based upon a check with telephone business offices. Response rates are developed for each type of interview respondent (e.g., parents of 3- to 8-year-old children or adults with no children in the household). Therefore, in a typical NHES the contractor shall develop and establish survey procedures that shall result in at least a 70 percent response rate for each type of interview respondent. A good screener response rate is critical, as the households that never have a chance to participate in the main interview by refusal at the household screening level may differ from more cooperative households, resulting in a biased estimate. Note that as response rates are cumulative, even target response rates of 85 percent at the screener level, times 85 percent at the content level, yield an overall response rate of 72.25 percent. An outline of data collection procedures together with a proposal that contains a written description of the activities and decision rules for nonresponse and noncontact cases shall be submitted to the COTR by no later than December 12, 1997. The COTR will comment on the data collection procedures within three weeks of receipt. Upon completion of the Cognitive Laboratory testing and subsequent revisions to the survey instrument, a revised and more detailed version of the data collection procedures shall be submitted to the COTR by no later than May 11, 1998. This revised and more detailed version shall specify the criteria under which any alternative procedures might be used. The COTR will provide comments on the revised version within two weeks of receipt. A final version, incorporating comments from the COTR, shall be due no later than November 2, 1998. 3.7 Conduct Cognitive Laboratory Research. Because the NHES:99 collection will cover more topics than have been covered in previous NHES collections, it is important to ensure that the items have a logical flow that respondents can follow. In support of this objective, the contractor shall design and implement cognitive laboratory research on the NHES instruments. The cognitive laboratory research shall be directed toward an understanding of the extent to which the questionnaire items are understood by respondents and how the questionnaire flow might be improved. This research shall seek to identify any areas of ambiguity and confusion. Prior to conducting any cognitive laboratory research, the contractor shall develop and submit a plan for the conduct of such research no later than January 12, 1998. The plan shall describe the contractor's general approach to this work and identify the specific methods and procedures that shall be used. Justification shall be provided for the selection of certain methods over others relative to the objectives of the research. The plan shall also describe how the outcomes of the cognitive laboratory research will be used to improve the overall quality of the NHES instruments and procedures. The contractor shall allow two weeks for NCES review of the plan. Implementation of the plan shall be contingent upon NCES approval. A draft of the Cognitive Laboratory Research Report shall be due no later than February 23, 1998. The contractor shall allow two weeks for NCES review of the draft. The report shall be revised in response to NCES comments and the final report submitted no later than March 16, 1998. Task 4. Sample Design The contractor shall design an appropriate sampling strategy that will result in a sample sufficiently large to support the types of analyses to which the data will be subjected. In designing the sample, the contractor shall weigh the advantages and disadvantages of different sampling approaches. The contractor shall carefully consider the total number of households that will need to be screened in order to obtain the target sample sizes. The impact of the screening rates on both the precision of the sample estimates and on the costs of the survey shall be evaluated. The contractor shall take the following features of the NHES into account when designing the sample: À"À In developing the sample design, the contractor shall give special attention to increasing the minority (black and Hispanic) representation in the sample and to increasing the precision of estimates for different race/ethnic groups. Increasing the number of minority households in the sample may be accomplished in several ways. The simplest method, but not necessarily the most efficient method, is to increase the overall sample size (i.e., the number of households screened). Other methods involve the use of information on neighborhood characteristics associated with different telephone exchanges to identify and oversample high minority exchanges and the use of more screening items at the household level. À"À The NHES:99 shall be designed in such a way as to increase the power of detecting changes across time in the key estimates. For estimates of percentages in the 30-60 range, the ability to detect at least a 10-15 percent relative change in key estimates (i.e., relative change as a percent of the estimate p) is desirable. À"À The contractor shall consider the merits of including in sample all household members who meet the eligibility criteria for an area of interest (e.g., 3- to 8-year-old children) versus sampling within the household among those members who meet the criteria. The contractor shall prepare a sample design plan for the NHES:99 survey and submit it to NCES for review and approval prior to its implementation. A draft of this plan shall be submitted to NCES no later than February 9, 1998. Following NCES review, the contractor shall make any necessary revisions to the plan and submit the revised plan to NCES for review. The revised plan shall be submitted within one week of receipt of NCES' comments on the draft plan--no later than March 2, 1998. The revised plan is expected to be the final sample design. Task 5. Survey Design Report The contractor shall prepare a Survey Design Report. This report shall be in large part a compilation of the materials prepared under Tasks 3 and 4. At a minimum, it shall contain: (1) an introduction to the survey with a comprehensive statement of its intended purpose; (2) a description of the sampling design of the survey; (3) a description of the survey procedures; (4) copies of the NHES instruments along with appropriate justifications and definitions for key variables; and (5) a plan for analyzing the data from the survey. The Survey Design Report shall also contain, in a separately bound appendix, detailed projected costs for Tasks 7 through 13 based on the same assumptions used in the design. The contractor shall submit an outline of the Survey Design Report to NCES for approval no later than February 2, 1998. Once the outline of the report is approved by NCES and no later than February 23, 1998, the contractor shall submit a draft of the Survey Design Report to NCES. Within two weeks of receipt of the draft report, NCES will provide the contractor with comments on the draft document. The contractor shall revise the Survey Design Report incorporating NCES' comments and suggestions. The revised Draft Survey Design Report shall be submitted to NCES within one week of receipt of NCES' review comments on the draft report--no later than March 16, 1998. The contractor shall provide the NHES COTR and members of the NHES TRP with copies of the report for their review and comment. The contractor shall request that the members of the TRP provide written comments and be prepared to discuss the NHES design at a TRP meeting. With NCES' approval, the contractor shall modify the Survey Design Report (and the NHES survey design) based on the suggestions of the TRP members. NCES assumes that the most of the contents of the Survey Design Report shall be used to prepare the IMT/OMB Forms Clearance Package. The Survey Design Report will continue to be updated throughout the data collection process, to incorporate the field test results and implementation. The final version of the Survey Design Report is due by no later than April 5, 1999. Task 6. IMT/OMB Clearance of the NHES:99 Instruments The contractor shall prepare a forms clearance package for obtaining IMT (Information Management Team of the Department of Education) and OMB (Office of Management and Budget) approval of the NHES. The draft package shall be due no later than March 24, 1998. The contractor shall follow specifications as provided in Standard Form 83-I. The forms clearance package shall include: a supporting statement that describes the reason for the study, detailed justification of all items to be included in the survey, sample design specifications, data collection procedures, analysis plan, estimated response burden, description of the NHES program, other information required by IMT and OMB, and survey materials to be used in the study-survey questionnaires, and other materials to be used with the respondents. Following NCES review of the draft package, the contractor shall revise the package incorporating NCES' comments and suggestions and submit a revised package no later April 13, 1998. This version of the package will be distributed more widely within the Department of Education for further review. The contractor shall expect additional comments and suggestions within four weeks. Following this review, the contractor shall have one week to submit a revised package incorporating NCES' comments and suggestions. This revision shall be due no later than May 18, 1998. Following approval on or about May 22, the contractor shall provide NCES with 10 copies of the IMT/OMB Clearance Package. Submitting the package this far in advance of the scheduled start of data collection will help ensure that OMB approval is obtained prior to the conduct of the field test (see Task 8). NCES will submit the package to IMT/OMB. Following submission of the package, IMT/OMB has up to four months to respond with questions or revisions before granting clearance; the contractor is expected to provide any technical assistance to NCES as necessary to answer such inquiries. Task 7. Develop CATI System To the greatest extent possible, the contractor shall design the CATI system in such a way as to minimize disruptions to the system when requests for changes are made. The contractor shall program the approved survey instrument into a CATI system in both English language and Spanish language versions. 7.1 Specification of Range and Logic Checks. The contractor shall prepare specifications for every questionnaire item response that detail the allowable range of response and of internal consistency checks between sets of related items. For example, if the age of the respondent is collected in the demographic items, then other items relating to the person's age should be consistent with the reported age. The contractor shall develop response categories that distinguish between "Not Applicable" responses (i.e., valid missing data due to skip patterns) and item nonresponse. The contractor shall develop procedures for ensuring the overall quality of the data collected in the NHES. Most of the data editing associated with the conduct of the NHES shall take place during the interview session via edits contained in the CATI system. However, even with elaborate edit specifications for the survey instruments, it is assumed that errors in the data still may occur. As a consequence, the contractor shall implement manual editing procedures. Procedures shall be developed and in place for resolving unanticipated problems. At a minimum, editing of the survey data during the conduct of the interview shall include range checks (both hard and soft) and consistency checks. Inappropriate characters (such as words in a numeric field) should be prevented from being entered by the interviewer. Procedures shall be established that permit interviewers to respond appropriately to survey participants who provide out-of-range or inconsistent information. In developing these procedures, the contractor shall weigh the needs for accurate information and limited interviewer interference against the need for completing the interview as efficiently as possible. For each NHES instrument, the contractor shall prepare CATI edit specifications. (Post-CATI edit specifications are covered in Task 11.1, below.) These specifications shall be submitted to NCES for review and comment no later than May 18, 1998. NCES will provide comments within two weeks, and CATI edit specifications revised in response to NCES comments shall be submitted no later than June 8, 1998. Open-ended items (such as an occupation or education provider name) will have to be coded and edited either online (with screens that provide precoded response categories) or handled in post-interview manual edits (See Appendix B). The online capability is most desirable but would need to work seamlessly with the rest of the CATI software. The contractor should propose how to handle such open-ended items. 7.2 Programming of Survey Instruments. Programming the survey instruments into the CATI system is one of the key activities of the project. For the survey instrument to capture the data reported by respondents accurately, the CATI system shall be able to appropriately handle all contingencies associated with the conduct of the interview. Each response shall lead to the next appropriate point in the survey and each response provided by the participants shall be accounted for. Because of the complexity of the CATI programming of a NHES instrument, the contractor shall develop the procedures and materials necessary to ensure that all program specifications are correct and complete. The contractor shall propose appropriate procedures for verification of specifications, such as having multiple reviewers of varying levels of expertise test out the screens. 7.3 Testing of the CATI System. Prior to the conduct of the field test (see Task 8), the contractor shall test all features of the CATI system, including those features of the system associated with sampling, scheduling, interview management (e.g., skip patterns), data entry and editing, and case control. Testing of the system shall involve the review of project staff at all levels of the project (e.g., project management, staff with substantive expertise pertaining to the collection, telephone operations staff, and programmers). It shall also involve the use of interview scenarios so that the system can be tested under simulated interview conditions. Every branch of the interview skip patterns should be tested, and corrected if necessary, prior to delivery of the testable CATI instrument, to be sure that all contingencies have been foreseen. The COTR shall be informed of the nature of changes to the CATI instrument due to error resolution, and any differences between the OMB-approved questionnaire and the CATI instrument that is programmed shall be documented and justified with the COTR's knowledge and consent. 7.4 Delivery of Testable CATI Instrument. The COTR shall have a minimum of 10 days to test out the CATI system. The contractor shall propose how to make the CATI system testable to the COTR, whether at the contractor's facilities or at NCES. A testable CATI is one in which the sampling and dialing functions do not have to be demonstrated, but all screens should be ready to test out in their entirety. It should be possible for the COTR or other assigned project staff to go through a variety of simulated interviews (everything except dialing and sampling), testing out the responses to various scenarios and types of household respondents. The COTR shall notify the contractor promptly of all questions or problems with the CATI programming. The contractor shall remedy all problems reported during the testing phase. A testable CATI instrument shall be made available to the COTR no later than 3 weeks prior to the Field Test (see Task 8), to allow time for reprogramming as necessary. Task 8. Field Test of Survey Instruments and Procedures 8.1 Submit Field Test Plan. The contractor shall design a Field Test for NHES:99 that can be done quickly and is mostly to "shake out" the CATI system. Items on NHES:99 have been fielded before. The contractor shall submit a plan for a small-scale Field Test by no later than July 6, 1998. NCES will provide comments within two weeks, and a revised plan that incorporates these comments shall be due no later than July 27, 1998. The Field Test plan shall detail the schedule for not only the Field Test, but also the subsequent analysis and revisions to the survey instruments. Once the cognitive laboratory work is complete, and the data collection instruments have been developed and approved (by NCES and OMB), the contractor shall conduct a field test of the instruments and the CATI system. The contractor shall allow enough time between the completion of this field test and the commencement of interviewer training (Task 10) to permit an analysis of the field test and to allow any proposed changes to the data collection instruments and procedures to be evaluated and made. 8.2 Conduct Field Test. The contractor shall conduct a field test of each NHES topical component prior to its full-scale implementation. All sampling and data collection procedures scheduled for the full-scale survey (see Survey Design Report under Task 5) shall be used to the extent feasible during the conduct of the field test; however, if necessary, purposive sampling may be substituted in order to guarantee a sufficient number of rare population households. The field test shall be used to test the entire CATI system, including its dialing, scheduling, and interview progress reporting functions. It shall also be used to test the wording and flow of the questionnaire items including appropriate skip patterns. Responses of field test sample members shall be reviewed, analyzed and results shall be documented. Item response rate tables shall be generated. Special attention shall be given to skip pattern errors, consistency and other edit checks, and the sensitivity of items. 8.3 Submit Field Test Report with Revised CATI Instruments. The contractor shall prepare a short summary of the field test findings for review by NCES and, at NCES' request, members of the NHES TRP. Any problems encountered during the conduct of the field test and the contractor's recommendations for overcoming these problems shall be documented and reported. Recommendations for changes to the survey instruments shall be given in detail with justification of any additional costing. The field test report shall be due no later than October 5, 1998. 8.4 Submit Design/Instrument Changes in IMT/OMB Memo. Based on field test results, modifications may be called for in the data collection procedures and instruments. Following NCES approval of any revisions to the survey instruments and procedures, the contractor shall submit these changes to IMT/OMB through NCES. Notification of these changes shall be made in the form of a memorandum from NCES to IMT/OMB. A draft of the memorandum shall be prepared by the contractor and submitted to the NCES COTR no later than October 12, 1998. The contractor shall modify the memorandum based on the comments of NCES staff and submit a final memorandum to NCES no later than November 2, 1998. The contractor shall provide 10 copies of this memorandum that shall contain hard copies of the final NHES:99 instruments. 8.5 Submit Final CATI Instruments in English and Spanish. To allow for the review of the Spanish-language version of the CATI instrument, the contractor shall provide hard copies of both the English and the Spanish-language CATI screens. Drafts of the screens, reflecting any IMT/OMB revisions, shall be submitted to the NCES COTR no later than November 9, 1998. NCES will provide comments within four weeks, and the revised Spanish-language version CATI screens shall be submitted no later than December 21, 1998. No later than November 16, 1998, the contractor shall also provide the COTR with access to a testable CATI instrument in English. Task 9. Hiring and Training CATI Interviewers The success of the NHES is due in large part to the skills and dedication of the CATI interviewers. It is critical that all persons assigned to these positions have the necessary verbal, interpersonal and typing skills required for successful CATI interviewing. These skills are developed through a combination of prior experiences working on other CATI projects and training. 9.1 Develop Interviewer Training Materials. In order to ensure that the CATI interviewers assigned to the NHES have the skill levels necessary to perform successfully the demanding tasks associated with the conduct of the survey, the contractor shall design and implement a CATI interviewer training program. Prior to developing any training materials and no later than November 2, 1998, the contractor shall submit an outline of its interviewer training program to NCES for review. This outline shall include: (1) a training program agenda that identifies the format of the session (lecture, interactive, role-playing, etc.), the topics to be covered (e.g., study background, overview of instruments, survey topical component), the length of time the session is scheduled to run; (2) an outline of the study materials that interviewers will be provided; and (3) a preliminary training program schedule that identifies when and where each group of interviewers will be trained. Within two weeks, NCES shall review and provide feedback on the outline. Incorporating that feedback, the contractor shall develop and submit drafts of all training materials to NCES by no later than November 30, 1998. NCES will provide a review of those materials within two weeks. The contractor shall make revisions as necessary based on NCES' review of the materials and submit final training materials to the NCES COTR by no later than December 28, 1998. The contractor shall monitor all data collection activities to ensure consistent high quality data throughout the collection period. Therefore, as a part of the training plan, the contractor shall include awareness of interviewer responsibilities under the Privacy Act of 1974 (5 U.S.C. 552a), the Privacy Act Regulations (34 CFR Part 5b), the National Center for Education Statistics Act of 1994 (P.L. 103-382, Section 9007), detailing the Center's responsibility for keeping all individual information confidential. The contractor shall also follow NCES Standards and Policies, and design methods of review and checking which will take place during and after training to ensure that only high quality CATI personnel are retained and utilized. In addition to supervisory monitoring, the contractor shall include plans that address issues of quality control such as tracking interviewer performance, maintenance of standards, identification of poor performance, and procedures for correcting such performance. 9.2 Recruitment of Interviewers. The contractor shall recruit and hire all CATI interviewers needed to complete the data collection within the time constraints imposed by the project schedule. Because the NHES requires the conduct of a large number of interviews within a relatively short time period, the contractor will need a large number of trained staff committed to the project. Whenever possible, the contractor shall recruit interviewers with prior CATI interviewing experience. Because the NHES oversamples Hispanics, the contractor shall have staff who are qualified to conduct the interviews in Spanish. 9.3 Interviewer Training Sessions. The contractor shall have responsibility for the conduct of the training sessions. All sessions shall take place at the contractors' facilities or facilities arranged for by the contractor. The contractor shall be responsible for ensuring that all interviewers assigned to the project successfully complete the training program as specified in the training plan. The contractor shall have responsibility for the production of all interviewer training materials and shall have sufficient materials available for all staff trained during the conduct of the project. Because of the complex nature of the NHES, the contractor shall develop and implement a multi-day training program which shall take place in the week prior to the start of data collection. (During the conduct of the spring 1991 NHES, approximately 20 hours of project-specific training was required of each interviewer in addition to basic training in general interviewing techniques and the use of the CATI system.) At a minimum, the training sessions shall consist of lecture, interactive, and role-playing sessions. The last session of each interviewer training program shall involve on-line interviewing with actual respondents under the close supervision of the contractor's staff. Training should take place in the week preceding the start of data collection. Confidentiality requirements in NCES contracts mandate that interviewers shall complete a sworn Affidavit of Nondisclosure. These affidavits shall be signed in the presence of a public notary and must be dated the first day the interviewer is on the payroll. (This requirement applies to all project staff having access to the data.) Task 10. Data Collection The NHES is scheduled to be conducted in the late winter/spring of each year (January through March) with data collection spanning a period of no more than three months (a shorter time period is certainly more desirable). All interviewing associated with the NHES, unless otherwise approved by NCES, shall be done by telephone using a CATI methodology. Once a telephone number is sampled for the NHES, the contractor shall screen the household (telephone number) for eligibility and, if determined to be eligible under the conditions of the survey, conduct all screener and extended interviews for which the household qualifies. All interviewing shall be conducted at the contractor's facilities and shall be under the supervision of the contractor's staff. 10.1 Data Collection Schedule. As stated above, the data collection period shall be a maximum of three months, beginning no earlier than the first week of January and ending no later than the first of April. All interviews shall be completed within this period. Interviews shall be scheduled so as to maximize the number of completed interviews and minimize the number of non-productive interviewer hours charged to the project. When a telephone number is sampled for the NHES, at least seven attempts (telephone calls) to complete the household screening interview shall be made over a two-week period. These attempts shall be scheduled so that they cover different days of the week and hours of the day. For reporting purposes, Unable to Contact (Ring No Answers or FAX/Data line) should be distinguishable from Contact, Noninterview (such as only Answering Machine/Voicemail responses). At least 14 attempts to complete all interviews associated with a household shall be made once a household is determined to qualify for the survey. Once again, these attempts shall be staggered over different days of the week and time of day (e.g., morning versus afternoon or evening hours). 10.2 Quality Control Procedures. The contractor shall develop and implement a set of quality control procedures that will ensure the collection of high quality data throughout the data collection period. Project supervisory staff shall closely monitor interviewer activities in order to ensure that all data collection procedures are followed and that all standards are adhered to. Problems that are identified shall be addressed immediately and consistently. Statistics on interviewer response rates should be maintained and checked regularly to pinpoint response rate problems. Because of the number of staff who will be assigned to the NHES data collection activities (both supervisory and interviewer staff), it is important that any decisions that are made about the conduct of the survey be disseminated immediately and clearly to all staff involved in the project. Procedures shall be in place from the beginning of training until the completion of data collection that will ensure that all parties involved with the collection of data use the same solutions to and interpretations of problems that arise during the course of the survey. 10.3 Progress Reports. Throughout the data collection period, the contractor shall provide the NCES COTR with weekly progress reports. The contractor's CATI system shall be designed in such a way as to be able to produce computer-generated reports that show the progress that is being made during the interview phase of the project. These reports shall contain detailed information on the interim and final status of all telephone numbers sampled for the survey. At a minimum, they shall include information on the number of potential households (telephone numbers) screened, the number of eligible telephone numbers sampled, the number of cases for which contact has been attempted and established, the number of cases for which interviews have been completed or that have refused to participate in the survey, the number of cases by reason for all types of noninterview or out-of-scope (such as: language barrier, phone in residence used for business, entire HH not eligible, phone disconnected after initial contact, ring no answer after maximum number of calls, etc.), the number of missed scheduled calls, the number of interviewer hours (aggregated), the number of cases referred to refusal conversion, and the number of initial refusals that were completed during refusal conversion. Where appropriate this information shall be reported separately for each type of eligible household or respondent. Response rates as well as completion rates for all cases should be calculated and reported to the COTR on a weekly basis, for all phases of the operation, from screening through final interview, with a summary that certifies that progress is at or above target levels, or with explanation and actions taken if sufficient progress is not being made. In addition to the computer-generated reports described above, these weekly reports shall identify any problems encountered and either describe how these problems were resolved or recommend alternative ways of resolving these problems. These weekly progress reports are in addition to the monthly reporting requirements described in Section III (Study Requirements, A. Reporting). Any problems encountered that either have consequences for the project's budget or time schedule shall be brought to the attention of the NCES COTR in these weekly progress summaries. Task 11. Data File Preparation and Documentation The data collected during the conduct of the NHES shall be entered directly into a computer file through the CATI system. Nevertheless, additional steps will be necessary to prepare the data for public release and analysis. The contractor shall take the necessary steps to convert the raw data entered by the CATI interviewers to a more useable form. 11.1 Data Editing and Coding. Although most data edits shall occur on-line as the CATI interviewers enter responses into the computer (see Task 7), additional data editing and coding shall be performed at the conclusion of the data collection period. Respondent data records shall be reviewed for completeness and for any other problems (e.g., inappropriate skips, out-of-range values, input errors) that may have occurred during the conduct of the survey. Problem records shall be identified and appropriate corrective actions taken. These editing and coding activities shall involve both computer-assisted and manual activities. Examples of completeness and accuracy in data editing and file preparation include: 1. There are no unreadable fields on any record nor unallowable characters. 2. Valid skips (missing data due to skip patterns) are not imputed and can be distinguished from valid zeros and nonzero survey data. 3. The files carry all of the sampling variables agreed to between the contractor and the NCES COTR, and those variables are documented. 4. The data on the data file to be delivered for COTR review have been checked and basic tables run. 5. The data are checked against known data for reasonableness (i.e., there are not twice as many parents of children in the age group being studied as reported on CPS for a roughly comparable year) and discrepancies are resolved. 6. Indications that specified edits are not working as expected (i.e., very high failure rates) are brought quickly to the attention of the COTR and modifications are implemented with COTR's knowledge and summary review if necessary. A plan for post-CATI edit checks shall be submitted to the NCES COTR no later than February 16, 1999. NCES will review the plan and provide comments and suggestions within two weeks. Incorporating the comments and suggestions made by NCES, the contractor shall revise its post-CATI editing plan and submit a revised plan no later than March 8, 1999. The contractor shall submit specifications for the coding of open-ended items no later than February 16, 1999. When appropriate, the coding specifications, which shall be approved in advance by NCES, shall be consistent with those used in previous surveys (i.e., NHES spring 1995). The contractor shall receive comments from NCES within two weeks and have one week to revise the specifications based on NCES' comments and to submit the final coding specifications to NCES no later than March 8, 1999.. The contractor shall code responses to all open-ended questions according to the approved coding specifications. Examples of the types and numbers of open-ended questions that the contractor can expect to find in the NHES include industry and occupation items and major field of study. Appendix B lists the open-ended items from NHES:95. The contractor shall use common formats and procedures for editing, coding, error resolution, and documentation for each NHES. In addition, for those collections that involve administration of items from prior NHES surveys, the contractor shall use the same format and procedures that were used during the earlier collections, unless alternative formats and procedures are approved by the NCES COTR. The use of these common formats and procedures will increase the comparability of the data from the different survey administrations and will enable users to compare data from the different collections with a minimum of effort. A status report on data editing and data coding shall be submitted to the NCES COTR by no later than May 10, 1999. This report shall include cumulative summary statistics on the numbers of problems detected for the various types of items and the status and method of corrective actions taken. 11.2 Data Conversion. The contractor shall design, establish, and carry out the procedures necessary to convert the data in the CATI system to CD-ROM with electronic codebook format. The datasets on the CD-ROM shall be made available in four formats: (1) ASCII format; (2) SAS/PC; (3) SPSS/PC, and (4) SPSS for Windows. ASCII-readable input files shall be used by the contractor to create Electronic Codebooks (ECBs) for the separate files. The contractor shall evaluate the previous NHES CD-ROM and propose enhanced features, such as Windows capability. The contractor shall submit a plan to the NCES COTR that describes the proposed structure and specifications for the NHES:99 data files and the ECBs. This plan shall be submitted no later than February 16, 1999. NCES will review the plan and provide comments within two weeks. A revised plan shall be submitted no later than March 8, 1999. The plan shall not be implemented without the approval of NCES. 11.3 Creation of Composite and Classification Variables. The contractor shall propose and create composite (e.g., SES using income measures, parent's education and occupation) and classification variables (e.g., race/ethnicity, highest level of parent education, family/household composition, count of adults in household, etc.) for use by analysts. A plan for the creation of composite and classification variables shall be submitted by no later than February 16, 1999. NCES shall review the plan and provide comments and suggestions within two weeks. A revised plan incorporating the changes requested by NCES shall be submitted within one week of receipt of NCES' comments no later than March 8, 1999. Once approved, these variables shall be included on the public use data files and their definitions and code included in the NHES data file User's Manual. 11.4 Item Imputation. Since the data from the NHES will be used to monitor the participation of children and adults in a variety of education-related activities over time, it is important that the impact of item nonresponse on survey estimates be considered carefully. The contractor shall analyze the level of item nonresponse for the individual survey items and propose a plan for full imputation of missing item data. The imputed data shall be placed on the public-use and restricted-use data files, along with appropriate documentation and imputation flag variables. The contractor shall propose a plan for full imputation of missing data and present it to NCES no later than February 16, 1999. Following approval or modification of the imputation scheme, the contractor shall prepare imputation specifications; these should be delivered to NCES upon finalization, but no later than April 26, 1999. 11.5 Sample Weights. The contractor shall develop sample weights to apply to the data. The weights are necessary to produce estimates required for various summaries and analyses. The weights shall incorporate a unit (complete interview) nonresponse rate adjustment for individuals in different "weighting classes," as well as an adjustment for non-telephone households. The contractor should propose a strategy also for dealing with households not on the telephone lists but which do possess telephones; at a minimum, the contractor shall evaluate the potential bias for excluding this type of household. Weighting classes and adjustment procedures shall be consistent with NCES Standards. The following are a few examples of data quality checks that should be performed after the sampling weights are applied to all cases on the file: 1. The sum total of weighted interviews is equal to the population totals used in the poststratification process, and all subgroup totals sum to the grand total. 2. All imputed data items have a valid in-range value and are flagged as imputed. 3. Imputed data are checked against the distribution of unimputed data and match that distribution within a reasonable margin of error. The contractor shall document the procedures planned for use in developing the weights and submit these plans to NCES by no later than February 16, 1999. Two weeks shall be allowed for NCES review. The contractor shall have one week to make any revisions resulting from NCES' review and submit a revised plan no later than March 8, 1999. 11.6 Standard Errors. The contractor shall develop and implement procedures (see NCES Statistical Standards) to enable users of the NHES to estimate the sampling errors of survey estimates. The contractor shall calculate estimates of design effect values for a minimum of thirty variables for the total population and for various subgroups of selected respondent characteristics (e.g., sex, race/ethnicity, SES, and at least two other variables recommended by the contractor). Means and standard deviations of the design effect distributions shall be calculated. The same set of procedures shall be followed for each public release file developed under this contract. The data files created by the contractor for each NHES topical component shall contain the elements necessary to support the calculation of standard errors for complex designs using two methods--jackknife repeated replication (JRR) and Taylor series procedures (TSP). If the contractor uses a software package to calculate these standard errors other than those available to NCES, it shall make this package available to NCES. For use with JRR, the contractor shall provide replicate codes that indicate the computing strata and the half-sample to which each sample unit belongs and the contractor shall provide, for each sample unit, the replicate weights for all replicates that were formed in order to calculate variances. For use with TSP, the contractor shall provide the stratum code and PSU code that identifies each sample unit. Regardless of the method used, the contractor shall describe in detail the method used to calculate the standard errors of survey estimates in the User's Manual. This description shall instruct users how to use the data elements and the associated software to calculate sample variances for the NHES. A short example of the method and the use of the available software shall be provided in the manual. The contractor shall prepare a plan for measuring sampling errors and submit it to NCES for review no later than March 8, 1999. At the same time, a plan for describing non-response patterns for each data collection and for estimating non-response bias for key variables shall be submitted. The plan will be reviewed within two weeks by NCES, and the contractor shall submit a revised plan within one week of receipt of comments no later than March 29, 1999. Results of the analyses shall be included in the Methodology Report (see Task 13). 11.7 Public Release Files and User's Manuals. By no later than July 6, 1999, the contractor shall prepare (following NCES standards on data files and documentation) and deliver to NCES preliminary or draft copies of the following computer-related products: 1. NHES CD-ROM containing a separate data file for each topical component. Files on CD-ROM shall be in ASCII and SPSS for Windows formats. Data files shall contain all administrative variables needed to analyze the data (e.g., case id), questionnaire variables, composite variables, weights, and imputation flag variables. 2. User's manual (patterned after NHES:95) for each of the files. The User's Manuals shall also include a codebook print file containing well-documented, weighted and unweighted frequency tables for all variables contained in the datasets. 3. A User's Guide that provides suggestions for using the NHES:99 datasets. This guide shall be patterned after the Guide produced for the NHES:95. 4. An ECB User's Guide which instructs CD-ROM users in the use of the ECB. This guide shall also be patterned after the ECB User's Guide produced for the NHES:95 5. READ.ME files that will be included on the CD-ROM that will provide guidelines for accessing the data files and describing the data files contained on the CD-ROM. The User's Manuals shall include documentation that is of sufficient detail to enable any user to fully understand the files. The manuals shall include, but not be limited to the following information: À"À a description of the NHES sample design À"À a description of the data collection procedures À"À a description of the response rates for each stage of the sample and for each topical component À"À a discussion and evaluation of the design effects of the NHES sample À"À a codebook that contains the unweighted and weighted frequencies and percentages for each survey item The User's Manuals using data from the NHES:95 study shall be used as a model for the User's Manuals developed under this procurement. (See full reference in Appendix A.) NCES review of data files, User's Manuals, and Guides will be completed four weeks after receipt of the draft/preliminary deliverables. The contractor shall have two weeks to submit revised copies of the data files, User's Manuals, and Guides following NCES review; this would be no later than August 16, 1999. NCES' comments and suggestions shall be taken into consideration when developing these final products. The contractor shall produce and deliver and Electronic Codebook (ECB) similar to those produced for previous NHES collections. The draft ECB shall be due no later than August 30, 1999. NHES shall review the ECB and provide comments within four weeks, and a revised ECB shall be submitted no later than November 1, 1999. 11.8 Adjudication of Datasets. To allow important customers, such as the National Education Goals Panel and the Office of Educational Research and Improvement (OERI), to have access to the NHES:99 datasets before NCES has released its first report, a dataset adjudication is needed. The contractor shall prepare all materials needed for the dataset adjudication. For the most part, the materials needed for a dataset adjudication are materials that are required in the Methodology Report (Task 13.) Thus, early draft versions of select sections of the Methodology Report can serve the purpose. The specific sections include those on unit and item nonresponse, imputation procedures, weighting and standard error calculation, and, most importantly, comparisons of key estimates with those obtained in other data sources. Drafts of these sections shall be provided to the NHES COTR no later than July 26, 1999. NCES shall review the materials and provide comments within one week, and the contractor shall submit materials revised to respond to NCES' comments no later than August 2, 1999. These revised materials shall be distributed, along with the draft User's Manuals submitted under Task 11.8, to members of the adjudication panel. The adjudication meeting is generally held on or about August 10. 11.9 Disclosure Review Board. NCES standards for release of public-use data require that the data ready for release be submitted to the Disclosure Review Board (DRB). This process usually entails having the dataset tested out and ready to certify as "clean," meaning that all of the procedures in NCES' Statistical Standard for Machine-Readable Products (IV-06-92), and that data suppression or conversion has been performed on individually-identifiable data (NCES Statistical Standard for Maintaining Confidentiality, IV-01-92). There should be a data release product, usually a Statistics in Brief, and a User's Manual, to submit to the DRB at the same time. No later than August 16, 1999, the contractor shall submit a draft memorandum detailing those items that might be considered disclosure risks (generally, a combination of any two variables which results in fewer than three cases in any cell in a cross-tabulation). This memorandum shall also contain an analysis of the data suppression technique used, such as a distribution of cases "before" and "after" the suppression. NCES shall provide comments on the memorandum within two weeks, and the contractor shall revise the memorandum to incorporate those comments no later than September 7, 1999. If the Disclosure Review Board should have questions about the memorandum and/or other aspects of the data files, the contractor shall respond within a week of receiving a specific question. 11.10 Restricted-use Data Files and User's Manuals. Following the same steps presented in Task 11.7 above for public release files, the contractor shall produce data files that contain restricted-use data. These data files will contain all the variables not included on the public-use files, including potential identifiers of respondents (e.g., State codes, Census data associated with ZIP codes, etc.). These files will be distributed by the government through NCES site-license agreements. The restricted-use files shall be provided on CD-ROM in ASCII, SAS/PC, SPSS/PC, and SPSS for Windows formats. The contractor shall provide draft data files and User's Manuals no later than November 29, 1999. NCES shall review those products and provide feedback within three weeks. Revised restricted-use data files and User's Manuals shall be submitted no later than January 10, 2000. 11.11 Maintenance of Necessary Data Files. The contractor shall maintain complete data files for all administrations of the NHES and all supplemental files created and obtained in support of this study. Maintenance of data files means keeping data file documentation available for reproduction on request from data users on a cost-reimbursable basis and keeping a machine-readable copy of each data file, so that copies can be supplied to NCES on request. Certain of the data files that shall be maintained may not be made available to the public because of privacy considerations. In these cases, file maintenance means that the data files shall be kept intact, in a form that can be updated or copied. Data files shall not be allowed to expire, to be released, to be overwritten, or otherwise destroyed. These files shall be maintained with a software system capable of easily interlinking them for analytic (or other) purposes while, at the same time, preserving the necessary confidentiality requirements of the study. Task 12. Data Analysis and Reporting The contractor shall perform analyses of the survey data and prepare a Statistics in Brief publication for each data file. The contractor shall assume that three data files will be produced for the NHES:99: a parent file, a youth file, and an adult file. The first Statistics in Brief reports serve two major purposes: (1) to allow for release of the data files, and (2) to stimulate interest in the NHES data. A Statistics in Brief report is no more than 16 pages in length and may contain the following information: À"À Short introduction with brief review of relevant literature; À"À Description of NHES survey; À"À Results or highlights section; À"À Conclusions; À"À A number of major tables, with standard errors; À"À Figures from table data (optional); À"À Description of NHES survey design and methods; À"À Characteristics of sample used in report; À"À Generalizabilty of sample -- potential for bias; À"À Significance testing and sampling errors; À"À Acknowledgments section. The contractor will use as a model previous Statistics in Briefs written, using NHES data. The exact format will be determined by the COTR and all reports prepared by the contractor shall follow NCES publications' standards. Outlines for the Statistics in Brief reports shall be submitted to NCES by no later than April 12, 1999. NCES will take two weeks for review, and the contractor shall submit the revised outlines, incorporating NCES review comments, in one week no later than May 3, 1999. Once the outline is approved by NCES, the contractor shall begin preparation of the reports. NCES expects that several iterations will be required for each report. The first draft of the first-release reports shall be due no later than Aug. 2, 1999. The contractor and NCES will jointly author the reports. No information in a report shall be released prior to NCES' release of the report. Examples of completeness and accuracy in reports are as follows: 1. Numbers reported are consistent within a report (i.e., between tables) and with any prior reports based upon the same data. 2. Tables/graphs/figures are clearly labeled and understood, and include all necessary notation. 3. Labels are consistent among tables, graphs, and figures. 4. Suppression rules are followed. 5. Results of tests of significance are reported in the text of drafts. 6. All text follows a logical and coherent progression from introduction to purpose to results to discussion. 7. Statements in the text are supported by test results. 8. Statements in the text are consistent with the meaning/intent of questionnaire items. 9. Ambiguous terms are defined. 10. The contractor shall bring to the attention of the NCES COTR all data that appear unexpected or unusual. 11. Materials meet standards set forth in "NCES Statistical Standards" and the "OERI Publications Guide." All differences cited in the text of the reports shall be supported by an appropriate statistical test (e.g., Bonferroni-adjusted t-test, chi square, etc.). Standard errors of the estimates should be calculated according to one of the methods specified in Task 11.6 and must be included in the reports. Statistics in Brief and most other published products go through a peer review process termed "adjudication." This process follows internal review through the Group level, and requires that the publication be sent out to several academic or government reviewers who are chosen for their expertise in the subject area, as well as other parts of the Department of Education. Five weeks should be allowed for circulation of the report to be adjudicated. An adjudication meeting is scheduled at the end of the review period, and all members convene or send in their comments and all comments are discussed. Comments are accepted, modified, or rejected, by consensus, and the contractor must incorporate the comments and write up a memorandum to the Adjudicator detailing the changes within two weeks following the meeting. At that point, once all parties have given their approval, the publication camera-ready copy and electronic file are prepared. All Statistics in Brief reports must be adjudicated and released on or before September 29, 1999. Task 13. Methodology Report By no later than May 17, 1999, the contractor shall submit an outline for a methodology report that documents the entire project including: (1) a description of the sample design, weighting, and imputation scheme; (2) item nonresponse rates, unit nonresponse rates, and bias analysis; (3) instrument development and CATI specifications; (4) data collection procedures; (5) data quality efforts; and, (6) comparisons of estimates. NCES shall provide comments on the outline within two weeks, and the contractor shall submit a revised outline within one week of receiving the comments no later than June 7, 1999. It is important that the methodology report describe as fully as possible the study design and all procedures used to conduct the survey. Special features of the study shall be described in detail and any problems encountered either during the design or implementation of the survey identified. Changes that were made to the design and/or procedures in response to unanticipated problems shall be documented. The report shall include major findings pertaining to the methods used and/or experimented with during the conduct of the study and discuss their implications for future surveys. A draft of the complete methodology report shall be delivered no later than November 22, 1999. After a four week NCES review, the contractor shall revise the report based on the comments and suggestions of NCES staff. The revised methodology report shall be submitted to NCES within two weeks of receipt of NCES review comments no later than January 3, 2000. The revised version will be more extensively reviewed within the Department of Education and the contractor shall receive comments and suggestions on or about January 31, 2000, from a variety of reviewers. The final methodology report that incorporates the comments and suggestions made by NCES shall be due within two weeks of receipt of comments on or before February 14, 2000. In addition to hardcopies, the final methodology report shall be delivered on floppy diskette in a format specified by NCES. III. STUDY REQUIREMENTS A. REPORTING The contractor shall submit three copies of monthly letters of progress to NCES for the duration of the contract: one copy to the Contracting Officer (CO), and two copies to the COTR. Progress reports shall describe the work in progress, indicate any problems encountered, and in the event of delays, shall propose ways of bringing the effort back on schedule. Progress letters shall be submitted by the 15th day of each month and shall cover the same period covered by payment vouchers. A summary of project expenditures shall also be included in each letter of progress. Project expenditures shall include: (1) a table summarizing, by task, the budgeted cost for the month, the actual cost for the month, the cumulative budgeted cost, the cumulative actual cost, the percent of contracted amount spent to date, the estimated cost to complete, the total estimated cost, the contracted cost, and the difference between the total estimated cost and the contracted cost; (2) appropriate records and information to permit the COTR to certify that the services that are listed on monthly bills are actually used and are for official purposes; and (3) a manpower report prepared and signed by the project director that summarizes actual personnel assignments for the month just completed, showing for each named individual the hours charged by task. The format chosen shall be consistent from month to month. Also, the same format shall be used to report on any subcontractor activity and costs. The contractor shall notify the Contracting Officer if the projected costs of the project are expected to exceed the project budget. The contractor shall communicate regularly (several times each week) and work closely with the NCES COTR on all aspects of the study. To facilitate this communication, NCES expects the contractor to set up an electronic system for transferring information via electronic mail and microcomputer. . B. SCHEDULE OF DELIVERABLES Completion dates for required products are indicated in the task descriptions and Exhibit 1, below. Many of these dates are approximations. A project schedule shall be submitted for NCES approval as part of the management/planning report required under Task 1. For each deliverable, a draft copy shall be submitted to the NCES COTR for review prior to final submission. EXHIBIT 2. Schedule of Deliverables for NHES:99 Deliverable (Task) Completion/Due Date Initial Meeting (1.2) Oct. 10, 1997 Draft Project Management Plan (1.3) [6 copies**] Nov. 3, 1997 Data Security Plan (1.9) Nov. 3, 1997 TRP Plan (2.1) Nov. 3, 1997 Summary of Published Indicators (3.1) Nov. 3, 1997 Summary Report of Meetings with Customers (3.2) Nov. 17, 1997 Draft Summary Report of Extant Surveys (3.3) Nov. 17, 1997 Revised Project Management Plan (1.3) Nov. 24, 1997 Draft Research Questions and Content Outline (3.4) Dec. 1, 1997 Final Summary Report of Extant Surveys (3.3) Dec. 8, 1997 Data Collection Procedures Plan (3.6) Dec. 12, 1997 Revised Research Questions and Content Outline (3.4) Dec. 21, 1997 Draft Project Brochure (1.7) Jan. 5, 1998 TRP comments added to Research Qs. and Content Outline (3.4) Jan. 12, 1998 First Draft of Instruments (3.5) Jan. 12, 1998 Draft Plans for Cognitive Laboratory Research (3.7) Jan. 12, 1998 Final Project Brochure (1.7) Feb. 2, 1998 Draft Project Bibliography (1.8) Feb. 2, 1998 Revised Instruments (3.5) Feb. 2, 1998 Outline of Survey Design Report (5) Feb. 2., 1998 Draft Sample Design Plan (4) Feb. 9, 1998 Draft Cognitive Laboratory Research Report (3.7) Feb. 23, 1998 Draft of Survey Design Report (5) Feb. 23, 1998 Revised Project Bibliography (1.8) Mar. 2, 1998 Revised Sample Design Plan (4) Mar. 2, 1998 Final Cognitive Laboratory Research Report (3.7) Mar. 16, 1998 Revised Survey Design Report (5) Mar. 16, 1998 Draft IMT/OMB Clearance Package (6) Mar. 24, 1998 Revised Instruments after Cognitive Lab (3.5) Apr. 13, 1998 Revised IMT/OMB Clearance Package (6) Apr. 13, 1998 Revised Data Collection Procedures Plan (3.6) May 11, 1998 Final IMT/OMB Clearance Package (6) [10 copies] May 18, 1998 Draft CATI Edit Specifications (7.1) May 18, 1998 Revised CATI Edit Specifications (7.1) June 8, 1998 Draft Field Test Plan (8.1) July 6, 1998 Revised Field Test Plan (8.1) July 27, 1998 Field Test Report (8.3) Oct. 5, 1998 Draft IMT/OMB Memo of Changes (8.4) Oct. 12, 1998 Final Data Collection Procedures Plan (3.6) Nov. 2, 1998 Revised IMT/OMB Memo of Changes (8.4) [10 copies] Nov. 2, 1998 Deliverable (Task) Completion/Due Date Outline of Interviewer Training Program (9.1) Nov. 2, 1998 Draft Spanish and English language CATI Screens (8.5) Nov. 9, 1998 Revised English CATI Instrument (8.5) Nov. 16, 1998 Draft Training Materials (9.1) Nov. 30, 1998 Revised Spanish CATI Screens (8.5) Dec. 21, 1998 Revised Training Materials (9.1) Dec. 28, 1998 DATA COLLECTION BEGINS Early Jan., 1999 Draft Plan for Post-CATI Editing (11.1) Feb. 16, 1999 Draft Plan for Coding of Open-Ended Items (11.1) Feb. 16, 1999 Draft Plan for Data File Specifications (11.2) Feb. 16, 1999 Draft Plan for Creation of Composite Variables (11.3) Feb. 16, 1999 Plan for Item Imputation(11.4) Feb. 16, 1999 Draft Plan for Sample Weighting (11.5) Feb. 16, 1999 Revised Plan for Post-CATI Editing (11.1) Mar. 8, 1999 Revised Plan for Coding of Open-Ended Items (11.1) Mar. 8, 1999 Revised Plan for Data File Specifications (11.2) Mar. 8, 1999 Revised Plan for Creation of Composite Variables (11.3) Mar. 8, 1999 Revised Plan for Sample Weighting (11.4) Mar. 8, 1999 Draft Plan for Standard Errors (11.6) Mar. 8, 1999 Revised Plan for Standard Error Calculations (11.6) Mar. 29, 1999 DATA COLLECTION ENDS Apr. 1, 1999 Final Survey Design Report (5) Apr. 5, 1999 Draft Outlines for Statistics in Brief Reports (12) Apr. 12, 1999 Final Specifications for Item Imputation (11.4) Apr. 26, 1999 Revised Outlines for Statistics in Brief Reports (12) May 3, 1999 Status Report on Data Editing (11.1) May 10, 1999 Draft Outline of Methodology Report (13) May 17, 1999 Revised Outline of Methodology Report (13) June 7, 1999 Draft/Preliminary Data Files (11.7) July 15, 1999 Draft User's Manuals and Guides (11.7) July 15, 1999 Draft Dataset Adjudication Materials (11.8) July 26, 1999 Revised Dataset Adjudication Materials (11.8) Aug. 2, 1999 Draft Data Release Reports (12) Aug. 2, 1999 Dataset Adjudication Meeting (11.8) Aug. 2, 1999 Draft Disclosure Review Memo (11.9) Aug. 16, 1999 Revised Data Files (11.7) Aug. 16, 1999 Revised User's Manuals and Guides (11.7) Aug. 16, 1999 Draft ECB (11.7) Aug. 30, 1999 Revised Disclosure Review Memo (11.9) Sept. 7, 1999 Final Data Release Reports (12.1) Sept. 29, 1999 Revised ECB (11.7) Nov. 1, 1999 Draft of Methodology Report (13) Nov. 22, 1999 Deliverable (Task) Completion/Due Date Draft Restricted-use Data Files and User's Manuals (11.10) Nov. 29, 1999 Revised Methodology Report (13) Jan. 3, 2000 Revised Restricted-use Data Files and User's Manuals (11.10) Jan. 10, 2000 Final Methodology Report (13) [6 copies**] Feb. 14, 2000 NOTES: À"À Unless otherwise indicated, NCES will review all deliverables and provide feedback within two weeks. À"À Unless otherwise indicated, five copies of each deliverable are required. Where indicated by an asterisk **, six copies are required. The sixth copy should be sent to the Contract Specialist. À"À Not covered in this exhibit are the following: 1) Monthly letters of progress (Task 1.4); 2) Briefing materials (Task 1.6); 3) Summary Reports and Recordings of all Technical Review Panel Meetings (Task 2); and, 4) Weekly progress reports throughout data collection (Task 10.3); C. EXPECTATIONS CONCERNING QUALITY OF, CORPORATE SUPPORT FOR, AND TIMING OF DELIVERABLES Although the term "draft" is used frequently in the Scope of Work, it is important that the contractor understand what NCES means and does not mean by "draft". A draft is a complete product of high quality that the contractor (and the government) would be proud to distribute. A "draft" is not simply what is available on the due date regardless of completeness or quality. NCES anticipates that only minor changes to drafts will be needed for them to become final deliverables. Every deliverable shall be sent with a cover memorandum signed or initialed by the project director or someone of higher corporate stature than the project director. The cover memorandum shall certify that the contractor's organization stands behind the quality of the product. This holds for all products even those prepared primarily by consultants or subcontractors. At no time will the government take direct delivery of a product from a consultant or subcontractor. All draft deliverables shall be due at the office of the COTR on or before their due date. Draft deliverables of 10 pages or less may be sent by electronic mail to the COTR and other appropriate project staff. Draft deliverables of more than 10 pages must be sent to the COTR such that they arrive on or before their due date. And, unless otherwise specified, five copies of each deliverable shall be required. Contractors must justify use of and obtain prior approval for use of any delivery service more expensive than USPS priority mail. D. THE ADJUDICATION PROCESS Adjudication of NCES reports is a process of internal review, followed by a peer review meeting. At the adjudication meeting, comments from reviewers both within NCES and from subject-matter experts or technical experts are reconciled. All NCES reports except for Working Papers go through adjudication. Following adjudication, the report must be revised per the agreements reached in the meeting. A post-adjudication memorandum detailing the changes that were agreed-upon is a standard operating procedure. Some comments can be answered without necessarily making revisions to the report, provided that such comments are deemed to be beyond the scope of the report. However, all comments received must be addressed. E. FORMS CLEARANCE The contractor shall prepare all written materials needed in the formal clearance and approval process according to IMT/OMB requirements. This approval process typically involves the following series of steps: 1) Submission of draft copies of questionnaires to the COTR for review (part of Task 3); 2) Revision of draft copies and presentation to NHES TRP members for comments (General Task 1.5); 3) Revision of drafts and presentation to NCES Interdivisional Review panel (General Task 1.5); 4) Submission of draft of clearance package including instruments and justification for NCES, Office of the Commissioner review (part of Task 6); 5) Revision of package and submission to IMT for review (part of Task 6); 6) Revision of and submission to OMB for review (part of Task 6); and 7) Revision and submission of package for final approval based on OMB review and comments (part of Task 6). For the IMT/OMB clearance package, the contractor shall provide item-by-item justification as well as justification for each major section of the questionnaire (e.g., background, education history, work history). Not only shall each item be justified by itself, it shall be justified as being a part of a major content area. Reasons for including selected items and not including other items shall be provided. The contractor staff shall be prepared to join the COTR in person for a review of the materials with IMT/OMB staff (if necessary). Eighteen (18) weeks (four calendar months) shall be allowed for the IMT/OMB review process following submission of the revised clearance package at 32 weeks prior to the start of data collection. The contractor shall ensure that authorization by IMT/OMB has been obtained before field test data collection begins. All hard copies of the data collection forms shall bear the approval number assigned by OMB. CATI interviewers shall have this number available to respond to survey participant inquiries. F. ELECTRONIC COMMUNICATIONS The contractor shall transmit all correspondence directly to the NCES COTR via cc:Mail or another mutually agreed-upon communications package. NCES currently uses Word, PowerPoint, and Excel as its primary software packages for communicating documents, charts, and tables. NCES uses the Department of Education's cc:Mail software as its communications package. All deliverables are to be in Word, PowerPoint, Excel, and/or other software packages that are compatible with NCES' software packages. APPENDIX A National Household Education Survey Publication List CIVIC INVOLVEMENT Youth Involvement in Community Service, forthcoming. (xx pages, NCES 97-331) Student Interest in National News and Its Relationship to School Courses Related to National Issues, forthcoming. (13 pages, NCES 97-xxx) * National Household Education Survey: Adult Civic Involvement in the United States, February 1997. (19 pages, NCES-97-906) HOUSEHOLD USE OF PUBLIC LIBRARIES Use of Public Library Services by Households in the United States: 1996, February 1997. (12 pages, NCES 97-446) PARENT/FAMILY INVOLVEMENT IN EDUCATION * Parents' Reports of School Practices to Involve Families, October 1996. (14 pages, NCES 97-327) Parents' Involvement in Their Children's Schools, forthcoming. (xx pages, NCES 97-xxx) Father's Involvement in Their Children's School, forthcoming. (xx pages, NCES 97-445) EARLY CHILDHOOD/SCHOOL READINESS The Elementary School Performance and Adjustment of Children Who Enter Kindergarten Late or Repeat Kindergarten, forthcoming. (xx pages, NCES 97-324) Approaching Kindergarten: A Look at Preschoolers in the United States, October 1995. (72 pages, NCES 95-280) Child Care and Early Education Program Participation of Infants, Toddlers, and Preschoolers, October 1995. (11 pages, NCES 95-824) * Family-Child Engagement in Literacy Activities: Changes in Participation Between 1991 and 1993, December 1994. (9 pages, NCES 95-689) Access to Early Childhood Programs for Children At Risk, May 1994. (112 pages, NCES 93-372) Readiness for Kindergarten: Parent and Teacher Beliefs, September 1993. (10 pages, NCES 93-257) Profile of Preschool Children's Child Care and Early Education Program Participation, February 1993. (38 pages, NCES 93-133) Home Activities of 3- to 8-year-olds, January 1992. (8 pages, NCES 92-004) Experiences in Child Care and Early Childhood Programs of First and Second Graders, January 1992. (6 pages, NCES 92-005) SCHOOL SAFETY AND DISCIPLINE Student Reports of Availability and Peer Approval of the Use of Tobacco, Alcohol and Other Drugs at School, March 1997. (19 pages, NCES 97-279) * Student Victimization at School, October 1995. (8 pages, NCES 95-204) Student Strategies to Avoid Harm at School, October, 1995. (7 pages, NCES 95-203) Gangs and Victimization at School, July 1995. (2 pages, NCES 95-740) Use of School Choice, June 1995. (2 pages, NCES 95-742R) Parent and Student Perceptions of the Learning Environment at School, September 1993. (17 pages, NCES 93-281) ADULT EDUCATION Participation of Adults in English as a Second Language Courses, forthcoming. (xx pages, NCES 97-319) * Participation of Adults in Basic Skills Courses, March 1997. (20 pages, NCES 97-325) Forty Percent of Adults Participate in Adult Education Activities: 1994-95, November 1995. (12 pages, NCES 95-823) Adult Education: Employment-Related Training, May 1994. (29 pages, NCES 94-471) Adult Education: Main Reasons for Participating, June 1993. (8 pages, NCES 93-451) TECHNICAL REPORTS * An Overview of Response Rates in the National Household Education Survey: 1991, 1993, 1995, and 1996, forthcoming. (61 pages, NCES 97-948) * An Overview of the National Household Education Survey 1991, 1993, 1995, and 1996, forthcoming. (29 pages, NCES 97-448) An Experiment in Random-digit-dial Screening, forthcoming. (27 pages, NCES 97-343) Measuring Participation in Adult Education, March 1997. (44 pages, NCES 97-341) Feasibility of Conducting Followup Surveys to the National Household Education Survey, January 1997 (22 pages, NCES 97-335) Adjusting for Coverage Bias Using Telephone Service Interruption Data from the National Household Education Survey 1993, December 1996. (34 pages, NCES 97-336) Reinterview Results for the School Readiness and School Safety and Discipline Components of the National Household Education Survey 1993, December 1996. (73 pages, NCES 97-339) Use of Cognitive Laboratories and Recorded Interviews in the National Household Education Survey, September 1996. (36 pages, NCES 96-332) Overview of the NHES Field Test, July 1992. (44 pages, NCES 92-099) Telephone Undercoverage Bias of 14- to 21-year-olds and 3- to 5-year-olds, July 1992. (42 pages, NCES 92-101) Multiplicity Sampling for Dropouts in the NHES Field Test, July 1992. (23 pages, NCES 92-102) Proxy Reporting of Dropout Status in the NHES Field Test, July 1992. (29 pages, NCES 92-103) Effectiveness of Oversampling Blacks and Hispanics in the NHES Field Test, July 1992. (20 pages, NCES 92-104) WORKING PAPERS Comparison of Estimates from the 1996 National Household Education Survey, forthcoming (xx pages, NCES Working Paper 97-xx) Undercoverage Bias in Estimates of Characteristics of Households and Adults in the 1996 National Household Education Survey, forthcoming. ( xx pages, NCES Working Paper 97-xx) Unit and Item Response Rates, Weighting, and Imputation in the 1996 National Household Education Survey, forthcoming. (xx pages, NCES Working Paper 97-xx) Design, Data Collection, Interview Timing and Data Editing in the 1996 National Household Education Survey, forthcoming. (xx pages, NCES Working Paper 97-xx) * Comparison of Estimates from the 1995 National Household Education Survey, February 1997. (75 pages, NCES Working Paper 96-30) * Unit and Item Response Rates, Weighting, and Imputation in the 1995 National Household Education Survey, February 1997 (107 pages, NCES Working Paper 97-06) * Design, Data Collection, Interview Timing and Data Editing in the 1995 National Household Education Survey, forthcoming. (107 pages, NCES Working Paper 97-08) Comparison of Estimates from the 1993 National Household Education Survey, forthcoming. (xx pages, NCES Working Paper 97-xx) Telephone Coverage Bias and Recorded Interviews in the 1993 National Household Education Survey, February 1997. (89 pages, NCES Working Paper 97-02) Unit and Item Response Rates, Weighting and Imputation in the 1993 National Household Education Survey, February 1997. (53 pages, NCES Working Paper 97-05) Design, Data Collection, Interview Timing and Data Editing in the 1993 National Household Education Survey, February 1997. (95 pages, NCES Working Paper 97-04) * Undercoverage Bias in Estimates of Characteristics of Adults and 0- to 2-Year-Olds in the 1995 National Household Education Survey, December 1996. (22 pages, NCES Working Paper 96-29) Estimation of Response Bias in the NHES:95 Adult Education Survey, June 1996. (41 pages, NCES Working Paper 96-13) The 1995 National Household Education Survey: Reinterview Results for the Adult Education Component, June 1996. (48 pages, NCES Working Paper 96-14) QUESTIONNAIRE WORKING PAPERS 1996 National Household Education Survey (NHES:96) Questionnaires: Screener, Household and Public Library, Parent and Family Involvement in Education, Youth Civic Involvement and Adult Civic Involvement, forthcoming. (xx pages, NCES Working Paper 97-xx) 1991 and 1995 National Household Education Survey Questionnaires: NHES:91 Screener, NHES:91 Adult Education, NHES:95 Basic Screener, and NHES:95 Adult Education, February 1997. (80 pages, NCES Working Paper 97-03) 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early Childhood Program Participation, and Adult Education, October 1996. (112 pages, NCES Working Paper 96-22) 1993 National Household Education Survey (NHES:93) Questionnaires: Screener, School Readiness, and School Safety and Discipline, October 1996. (66 pages, NCES Working Paper 96-21) 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early Childhood Education, and Adult Education, October 1996. (58 pages, NCES Working Papers 96-20) DATA SETS * NHES:91/93/95 CD-ROM. (NCES 97-827) The NHES:91/93/95 CD-ROM contains the following data files: 1991 Adult Education, Adult file; 1991 Adult Education, Course file; 1991 Preprimary file; 1991 Primary file; 1993 School Readiness file; 1993 School Safety and Discipline file; 1995 Adult Education file; and 1995 Early Childhood Program Participation file. The CD-ROM contains an Electric Codebook (ECB) program that allows researchers to examine the variables in each of the NHES data sets as well as create SAS, SPSS for DOS, and SPSS for Windows programs that generate an extract data file. In addition, the Data Files User's Manuals from each of the NHES components, A Guide to Using Data from the National Household Education Survey, and the NHES:91/93/95 Electronic Codebook User's Guide are included on the CD-ROM. Must read documents in files on the CD-ROM include: National Household Education Survey of 1995: Early Childhood Program Participation Data File User's Manual, October 1996. (417 page, 96-825) National Household Education Survey of 1995: Adult Education Data File User's Manual, October 1996. (398 pages, NCES 96-826) A Guide to Using Data from the National Household Education Survey, September 1996, (55 pages, NCES 96-891) NHES:91/93/95 Electronic Codebook (ECB) User's Guide, September 1996, (78 pages, NCES 96-890) APPENDIX B Examples of open ended questions drawn from the National Household Education Survey of 1995 D4. What was the major subject or field of study of your (CREDENTIAL)? CRMAJOR1- CRMAJOR3/R SPECIFY______________________________________ D10. Let's talk about courses you took as a part-time student in the past 12 months. What (was/were) the name(s) of the course(s) and what was the general subject matter for each course in (CREDENTIAL) in (SUBJECT)? CR1CLS1-CR1CLS14/RNAME SUBJECT__________________ CR2CLS1-CR2CLS14/RNAME SUBJECT__________________ CR3CLS1-CR3CLS14/RNAME SUBJECT__________________ CR1SUB1-CR1SUB14/RNAME SUBJECT__________________ CR2SUB1-CR2SUB14/R CR3SUB1-CR3SUB14/R Questionnaire items that are the same as D10-F3, G3 E2. In what trade or craft (are you an/did you) apprentice? APTRADE/R SPECIFY_______________________________________ I32. What (is/was) your job title and what (are/were) your most important duties? [JOB PROBE: For example, electrical engineer, stock clerk, typist, or farmer] [IMPORTANT DUTY PROBE: For example, typing, keeping account book, filing, selling cars, operating printing press, and finishing concrete.] PROFESS1-PROFESS5/R IMPORTANT DUTY______________________ DUTIES-DUTIES5/R JOB TITLE______________________________