STATEMENT OF WORK FOR A STUDY TO ASSESS THE QUALITY OF VOCATIONAL EDUCATION IN THE UNITED STATES

1. INTRODUCTION

This work statement describes a 26-month study to be conducted as part of a National Assessment of Vocation Education (NAVE). The purpose of this study is to assess the quality of public vocational education in the United States, primarily at the secondary level.

The NAVE is a congressionally mandated evaluation of the Carl Perkins Vocational and Technical Education Act (Perkins III) reauthorized in 1998. The NAVE’s mission is to "conduct an independent assessment" of current practice in vocational education, and the implementation of federal legislation. Findings and recommendations that result from NAVE activities are due to Congress by July 1, 2002. Congress and senior Executive Branch officials are the main audience for the NAVE results.

The NAVE will undertake studies in several areas. The study described in this work statement is particularly important, for it will examine whether: 1) nearly a decade of federal efforts to reform and improve the quality of programs have found their way into classroom practice, and 2) where there are differences in student outcomes, whether these are associated with the quality of vocational instruction.

The statement of work lays out the basic policy and research issues to be addressed, and a strategy for addressing these questions. Offerors are encouraged to suggest alternative approaches that they believe are more effective or efficient ways to achieve the objectives of this evaluation.

2. OVERVIEW OF STUDY

For over a decade federal policy has attempted to strengthen the connection between vocational education and mainstream educational objectives at the high school level. Federal vocational legislation has increasingly emphasized several major themes—integration of academic and vocational education, linkages between secondary vocational programs and postsecondary education, and broadening instruction around careers and industries— not just entry level occupations.

In part, these vocational improvements are intended to keep pace with and complement other reform efforts in high schools. States and local districts have been raising the academic coursework and skills required for graduation, making high academic achievement the paramount marker of schools’ success. While other measures of school performance are also important (e.g., technical competency, placement into higher education or career related employment, and high school graduation rates), efforts to increase academic attainment are likely to continue as a focus of school improvement. Thus, a major policy issue facing vocational education, then, is whether and how it can support this, now central mission for high schools.

This statement of work describes a study to examine the quality of vocational instruction, mainly in high schools. The study focuses upon the classroom as the primary unit of analysis, in an attempt to understand how district, state and federal policies enhance or impede the capacity to offer sound programs of instruction in vocational education. Using a variety of data collection methods, including case studies and a national survey of teachers, the contractor shall examine the extent to which actual practice is consistent with legislative and other views of what constitutes "effective" vocational programs. Case studies of schools and classrooms will be nested in an examination of district, state and federal policy that might be expected to shape practice. The case studies shall identify and describe the quality of vocational education, the teacher survey will be used to determine the prevalence of "promising" practices across the country.

While the focus of this study is primarily upon high schools, a special component will examine several key issues related to the implementation of Perkins III at the postsecondary level, where nearly 40 percent of Perkins funds are spent. This special study will focus primarily on public community and technical colleges. Both Perkins III and the recently enacted Workforce Investment Act (WIA) envision community and technical colleges as central to workforce development policies and practices. Congress will want to know if recent incentives to integrate WIA and Perkins have had any substantive impact on postsecondary occupational programs or the role of Perkins funding.

The general approach is summarized below:

  1. Survey career preparation teachers. A nationally representative sample of classified vocational teachers and others who teach career preparation courses will be drawn and surveyed once, in spring 2001. The questionnaire will be designed to collect information about curriculum and instructional methods, focusing on indicators of quality. New questioning techniques will be used to ask about classroom practice in ways that elicit detailed, reliable responses.
  2. Conduct case studies of selected states, districts, schools, and teachers. A sample of approximately 50 high performing and lower performing high schools will be drawn using state performance data (or other sources). In-depth case studies will provide greater detail about state and local policy and school practice. Through intensive site visits, the case studies will obtain descriptive information about the extent and quality of vocational programs, state and district efforts to support and improve the programs, and the linkages between these efforts and schools reform and workforce development initiatives. A set of states will be selected, and then districts, schools, and teachers will be sampled within those states for more focused study.
  3. Focus group discussions with teachers, employers, students, counselors and parents. With school reform efforts beginning to change the landscape of school experience, it will be important to assess the attitudes, perceptions, and opinions of key stakeholders about the purpose and current implementation of vocational education. Focus group discussions will be conducted in each district area selected for the case studies.
  4. Substudy of community college practices. Site visits to postsecondary institutions (public community colleges and technical institutes - approximately 20) shall be included in a select set of communities (coincident with secondary site locations) to collect Perkins funding information. Implications of WIA on postsecondary vocational education shall also be investigated.

This study is different from, but builds on, earlier assessments of vocational education in several ways. The current study approach reflects several design considerations:

  • Broad interpretation of vocational education. The Perkins Act defines vocational education as a sequence of courses that provide individuals with the academic and technical knowledge and skills the individuals need to prepare for further education and for careers that require less than a baccalaureate degree. While traditional interpretations focused on courses and programs that help student acquire job-specific skills, the scope of this study should also include broader career preparation initiatives, such as Tech-Prep and career academies.
  • Emphasis on teachers and classroom as unit of analysis. To adequately examine the quality of vocational education, the focus of data collection and analysis must be on classroom practice and those who choose, adopt, interpret, and implement curriculum frameworks.
  • Development of quality indicators. While there is no definition of exactly what "good quality" vocational education is, some consensus on key concepts of practices are emerging. Practitioners and researchers—and the Perkins Act itself—cite integration of academic and vocational education, teaching all aspects of the industry, and linkages to postsecondary education as examples of features of high quality, high school vocational programs. There is much less consensus, however, on how to identify or evaluate the extent of these broad features in classroom practice. An important component of this study will be to develop detailed indicators of quality that can be applied as part of a survey questionnaire and classroom observation.
  • Linkages to other initiatives. Any changes to secondary vocational education in response to Perkins III will likely be taking shape in the context of both state school reform efforts and regional workforce development initiatives. Examination of state, district, and school implementation of vocational policy and practice should include the influence these other efforts are having on vocational education, and vice-versa.
  • Early response to the new Perkins Act. This study of vocational education quality will be concluded only a few years after passage of the new legislation. Most states and localities will not have fully adjusted their policies and practices to reflect the federal changes. It will therefore be important to capture not only current implementation efforts, and how these have changed recently, but also the extent and direction of future changes anticipated.

 

3. BACKGROUND

The recently amended Carl Perkins Vocational and Applied Technology Education Act maintains the same basic structure for funding vocational education as did Perkins II, and continues to emphasize similar improvement strategies; i.e., integration of academic and vocational education, improving linkages between secondary and postsecondary vocational programs, and broadening instruction to include "all aspects of the industry."

However, the Act introduced several key changes in federal funding and oversight of vocational education. First, the Act places a greater emphasis on academics, requiring vocational programs to be linked to state academic standards and states to report on vocational student attainment of those standards, as well as of vocational and technical competencies. Second, the Act provides states and local communities the opportunity to integrate vocational education into the new workforce investments systems being developed under the Workforce Investment Act of 1998. Third, Perkins III requires states to allocate a greater share of their Perkins grant to local programs. Finally, the new Act also significantly increases flexibility at the local level, by reducing certain funding and administrative mandates contained in the earlier Perkins II.

These changes are expected to alter the way vocational education is planned and implemented at both the secondary and postsecondary levels. Funds distributed under the Act may lead to new collaboration, more rigorous vocational education, and a focus on technology—depending on how state and local agencies respond to the new federal provisions. The extent and ways in which changes are taking place, and the consequences of them, are of great interest to the U.S. Congress and to other policymakers.

4. KEY POLICY AND RESEARCH ISSUES

This study provides an opportunity to address several major issues (outlined below) in federal vocational education policy. In performing the tasks described in the statement of work, the contractor shall include and address, but not be limited to, the policy issues and research questions listed below. The contractor shall refine these questions and identify other relevant topics.

How can schools improve the performance of "vocational students" and what, if anything, is the relationship between vocational education and those efforts?

Once focused primarily upon developing entry- level job skills for a manufacturing economy, vocational education has embraced broader and varied objectives in response to changes in the economy, the demand for higher- level skills, and shifting education emphases. Federal vocational education policy (Perkins III) now places priority on ensuring that secondary vocational education students are academically well prepared for careers and success in postsecondary education. This study will therefore examine, primarily through case studies, how schools are changing to address this priority. Research issues shall include:

  • Do high schools that have succeeded in improving student achievement organize vocational education differently than do schools that have been less successful? If so, in what ways? Do courses cover more technically challenging material? Are they taught differently? Do they consciously support academic objectives?
  • What vocational education practices and approaches appear most promising in promoting academic achievement and technical competence?
  • What is the relationship between vocational education improvement and school reforms underway in many states and local communities?
  • What do different stakeholders view as the primary purpose of vocational education? Are stakeholder perceptions about vocational education and its objectives changing? Why? How, and to what extent, are different purposes and objectives of vocational education reflected in state or local policy?
  • To what extent is vocational education included in state or local education standards, policy guidelines, or assessment systems?
  • Have there been any recent changes to curriculum frameworks for vocational education? If so, in what ways are they different from what came before? Why were they changed?
  • Where does secondary vocational education fit into state workforce development? To what extent is high school vocational education included in workforce development planning or implementation (policy, regulations, and guidelines)? For example, are secondary vocational offerings tied into labor market needs identified by workforce development boards?
  • What role does federal vocational education policy, as implemented by states and districts, play in facilitating or impeding implementation of effective programs of study at the secondary level?

 

What is the quality of vocational education and to what extent are federal strategies for improving vocational education quality reflected in actual classroom practice?

Perkins III continues to emphasize several basic reform strategies such as integrating academic and vocational education, linking secondary and postsecondary vocational programs, and broadening vocational curriculum beyond its traditional emphasis on entry- level job preparation. The genesis of these reforms was in Perkins II, nearly 10 years ago. This study will use both the teacher survey and the case studies to examine such questions as:

  • Are states and communities making progress in implementing key reform strategies emphasized in Perkins III? What other strategies reflect "quality" in vocational education e.g. vocational student organizations, internships, etc.?
  • How prevalent are these strategies? To what extent have the reforms emphasized in Perkins penetrated down to the classroom level?
  • To what extent has professional development been used to support these strategies? What are the priorities for and quality of teacher training and have these changed over the last several years? What preparation do vocational teachers typically have? What type and amount of training is available to them?
  • What is the level of demand (enrollment) in vocational programs/courses? Are there too many, or too few vocational teachers/courses? What strategies have been undertaken to deal with excess demand for or supply of vocational teachers?
  • How does the quality of vocational instruction and teaching vary among different types of institutions (comprehensive high schools, vocational high schools, and area vocational schools), different types of communities, or the demographic characteristics of students? What factors affect the depth or quality of vocational education (e.g., employer involvement, teacher preparation?

What is the role of Perkins funding at the postsecondary level? What is the [projected] impact on postsecondary occupational education of aligning Perkins with the Workforce Investment Act?

Although the funding formula specifically affecting vocational education at the postsecondary level was not changed under Perkins III, the most recent legislation provides institutions with greater flexibility in the use of Perkins funds with heightened emphasis on accountability. The 1998 law presents opportunities to integrate vocational education with workforce investment systems (through the optional use of state unified plans between WIA and Perkins and the provision of workforce development services at postsecondary providers - - One- Stop Career Centers). A special component of this study will therefore address such issues as:

  • What is the level and role of Perkins funding at postsecondary institutions?
  • What is the impact on postsecondary occupational education (mission, offerings, service delivery, etc.) of aligning Perkins with the Workforce Investment Act?
  • What is the relationship between local workforce boards and postsecondary vocational programs? To what extent do local workforce boards impact postsecondary vocational programs?
  • What is the interaction/relationship between postsecondary institutions and secondary vocational programs (e.g., do tech- prep funds play a role on this front)?

Other Key Concerns

Role of Employers

  • To what extent and in what ways do employers participate in vocational education planning and implementation? Does this participation reflect long- term partnerships with schools or more recent, specific activities?
  • What types of employers participate and in what occupational fields?
  • Why do employers participate? What are their incentives and what benefits do they derive?
  • How satisfied are employers with the vocational programs in their communities and with their participation in planning and/or implementation?

Special Populations

  • To what extent, if at all, has participation of special populations in vocational education changed since the introduction of Perkins III? How has access to high quality vocational programs changed?
  • Are there barriers to participation for special populations? If so, what are the obstacles? In what types of states or communities are these barriers most severe?
  • What specific strategies at the state and local levels are being undertaken to promote access to high quality vocational education for special populations?
  • Has the elimination of set- asides in Perkins III altered state and local strategies for serving special populations? Are there programs or key staff that have been added or eliminated?

 

Funding

  • How are Perkins resources used?
  • What are the mechanisms for allocating funds (state administrative and leadership as well as local funds)- - within the state to districts, within districts to schools, within schools to programs?
  • Are there specific priorities for Perkins funding?
  • What proportion of funds are spent on professional development, curriculum development, equipment and materials, teacher salaries, initiatives for special populations, other needs?
  • What other federal, state, or local resources are leveraged to support secondary vocational education? How are these used?

5. SCOPE OF WORK

This section specifies the tasks and subtasks the contractor shall perform. All deliverables shall be submitted to ED’s Contracting Officer’s Technical Representative (COTR) in draft. Unless otherwise specified, the COTR will notify the contractor within two weeks of receipt of changes required in the deliverable, and the contractor shall have two weeks to make the necessary changes and submit a final deliverable to ED.

 

Task 1: Revise Study Design

The purpose of this task is for the contractor to: (1) become familiar with existing information on state and local vocational education programs, and (2) make necessary adjustments in the study design based on this additional information.

The contractor shall:

  • Meet with the COTR, the Contract Specialist (CS), and program staff from the Office of Vocational and Adult Education (OVAE) within two weeks after contract award. The contractor shall allocate one day in Washington, DC for this initial meeting. The purpose of this meeting shall be to discuss procedures for conducting the study and answer questions that the contractor may have. Two weeks later, the contractor shall prepare a memo for submission to ED summarizing key issues discussed at the meeting.
  • Present the study design to the NAVE Advisory Panel, at a meeting scheduled for November 15- 16, 1999. The contractor shall be prepared to spend one- day informing the panel members about the key elements of the study and any outstanding design issues to be resolved.
  • Review available literature (other studies, recently submitted state plans, state evaluations, etc.) to help guide the development of data collection plans. As an initial step, the contractor shall contact the COTR to arrange a time and place to review state Perkins plans to gain background information on the vocational education activities and implementation approaches expected to be undertaken by the states. The contractor shall arrange to review these documents within 2 weeks after the effective date of the contract.

 

Based upon the data obtained from these sources, the contractor shall make necessary adjustments to the study design originally submitted in response to the Statement of Work. The revised study design (draft) shall specify the issues, data sources, and methodology to be employed in completing the required tasks. In particular, it should include any refinements to the strategy for sampling case study sites or for developing the survey instrument. The contractor shall submit the revised study design to ED within 2 months after the effective date of the contract.

The contractor shall also prepare a brief (2- page), non- technical summary of the study design suitable for distribution to a broad audience of policymakers and practitioners. The contractor shall submit this document one week after the revised study is accepted by ED.

Deliverables: Memo Summarizing Initial Meeting

Revised Study Design (draft and final)

Non- technical Design Summary

 

Task 2: Design Quality Indicators and Data Collection Instruments

The current study seeks information about both the broad scale of vocational program implementation and the quality of instructional practice in schools and classrooms. Therefore, the contractor will need to devote substantial effort to designing a variety of data collection instruments that reliably capture detailed information and that triangulate with each other. Several types of instruments will be necessary: a mail survey of a nationally- representative sample of high school vocational teachers and other career preparation teachers, a classroom observation guide consistent with the teacher survey, and protocols for site visits to selected states and communities. The contractor shall develop each of these to address the policy issues and research questions described under Section 4, avoiding unnecessary or duplicative data collection by building on existing data sources, as described below.

Subtask 2.1: Develop Quality Indicators for Use in Data Collection Instruments

As a first step, the contractor shall identify the features of vocational education that define "high quality" practice and develop indicators that allow objective measurement of these features in schools and classrooms. The indicators, in turn, will be translated into detailed survey items that can be used for both the national teacher survey and focused classroom observation in select sites.

The contractor shall develop a strategy for consulting with experts to develop a set of valid quality indicators and survey questions that collect the relevant information. The survey items must go beyond broad measures of implementation, as "Do you integrate academic and vocational education?" to probe for specific practices that consensus suggests identify high quality instruction. For example, detailed objective questions might be asked about the content of course subject matter or the nature of assignments given students. From responses inferences may be drawn about the quality of instruction. The quality features the contractor should consider focusing on include, but are not limited to: (1) academic rigor, (2) occupational/technical skills, (3) integration of academic and vocational instruction, (4) use of technology, (5) all aspects of the industry, and (6) linkages with postsecondary education.

Based on input from a panel of experts, the contractor shall prepare a memorandum specifying the quality indicators that will be used. The contractor shall submit this memo to ED 6 months after the effective date of the contract.

Deliverable: Memo on Quality Indicators

 

Subtask 2.2: Develop Career Preparation Teacher Survey Questionnaire

The contractor shall develop a questionnaire to be administered to a nationally representative sample of classified vocational teachers and others who teach career preparation courses in public high schools (comprehensive and vocational) and area vocational centers. In crafting this questionnaire, the contractor should focus on the quality indicators but also build on other related survey instruments, to allow comparisons across time and with other groups of teachers. For example, the contractor should consider including some items from the teacher survey administered in the previous National Assessment or from such national efforts as the School and Staffing Survey or the teacher survey used in data collection for the National Education Longitudinal Study (NELS). The contractor shall consider a variety of survey techniques to maximize response rates, including the possibility of administering the survey over the Internet. It is expected that the survey, however administered, will take no longer than 30 minutes to complete.

The contractor shall submit a draft of the questionnaire to ED 7 months after the effective date of the contract.

Deliverable: Vocational Teacher Survey Questionnaire (draft and final)

 

Subtask 2.3: Develop Classroom Observation Guide

The contractor shall develop an instrument for the purposes of documenting and rating the quality of vocational education practices in the classroom. This observation guide should include many of the same indicators as used in the teacher survey, so that the quality features in selected classrooms can be measured against the national prevalence of these features. The guide should also allow the observer room to record any special strategies or practices employed by the selected teachers. In addition, the guide should include instructions for the site visitor to review and record information about lesson plans, competency tests, planned projects, and student work as concrete evidence of instructional practice.

The contractor shall submit a draft of the classroom observation guide to ED 7 months after the effective date of the contract.

Deliverable: Classroom Observation Guide (draft and final)

Subtask 2.4: Develop Secondary School Case Study Protocols

The case studies are the major focus of this study, and shall be designed to gather detailed information on vocational education policy and program implementation, including the quality of instruction. The contractor shall develop data collection protocols to guide the conduct of site visits in selected states and communities. These protocols shall address the issues identified earlier in Section 4. The protocols shall include:

  • Protocols to guide data collection activities with secondary district and school administrators, teachers, and other staff. These protocols shall focus on issues such as: (1) local policies and procedures to promote vocational education improvement; (2) organization and quality of vocational programs; (3) professional development; (4) coordination and articulation with postsecondary programs; (5) supply of and demand for vocational teachers; (6) role of employers in vocational education planning and implementation; (7) access to vocational education for special populations; (8) linkages with workforce development initiatives; (9) leadership and technical assistance provided to districts and schools from the state; and (10) allocations and priorities for Perkins funding.
  • Protocols to guide interviews with representatives of businesses and business organizations. These protocols shall focus on the extent and ways in which employers participate in vocational education planning and implementation, satisfaction with their involvement, and factors that promote or discourage their participation.
  • Protocols to guide data collection activities with state agency or state- level staff, including those in state Department’s of Education, Community College Boards, Workforce Development Boards, and representatives of employers or teacher organizations. These protocols shall focus on issues such as changes to state standards, curriculum frameworks, policy guidelines, or assessment that affect vocational education; the role of vocational programs in state workforce development planning and implementation; allocations and priorities for Perkins funding.
  • Questioning guides or protocols for focus group discussions with parents, teachers, counselors, and students. Each protocol shall address issues most salient to the specific stakeholder group but include the purpose of vocational education, quality of current vocational instruction, and satisfaction with that instruction.

 

The contractor shall submit the draft protocols to ED for review and approval 7 months after the effective date of the contract.

Deliverable: Secondary Case Study Protocols (draft and final)

 

Subtask 2.5: Develop Postsecondary Case Study Protocols

The contractor shall prepare detailed protocols for interviewing administrators, institutional financial officers, and faculty at selected postsecondary institutions. Questions preferably enable collection of quantifiable and comparable data across sites.

  • Protocols to guide interviews with administration and finance personnel shall include information on the following topics: college mission (primary emphasis on workforce development, transfer function, economic development), extent the institution has pursued and achieved articulation agreements with secondary and 4- year institutions (in practice how much are they used?), view of state’s role/impact on occupational education, view of impact of WIA on occupational programs (connection to one- stop?, actions to become/remain eligible provider, types of participating students), Perkins fund allocation ($ amount) and uses, overall budget ($), overall government funds, and institutional relations with area business (existence of and use of business advisory board?).
  • Protocols to guide focus groups with faculty shall cover the following issues: college mission (primary emphasis on workforce development? transfer function?), extent the institution has pursued and achieved articulation agreements with secondary and 4- yr. institutions (in practice how much are they used?), use of Perkins funds, mechanisms to ensure program and curriculum relevance to the labor market, and linkages with area business and one- stops (business advisory board? Impressions of its impact?).

The contractor shall submit the draft protocols to ED for review and approval 7 months after the effective date of the contract.

Deliverable: Postsecondary Case Study Protocols (draft and final)

Subtask 2.6: Conduct Pretest of Teacher Survey

The contractor shall pretest the teacher survey with no more than nine vocational teachers. To the extent possible, the contractor shall choose teachers that represent a mix of vocational program specialties, geographic areas, and populations served. The contractor shall debrief each of the pretest teachers to get their reaction to the survey instrument, and shall elicit comments on the usefulness and likely accuracy of the information requested as well as the burden associated with providing it.

Based upon the results of the pretest, the contractor shall prepare a memorandum discussing recommended changes to the questionnaire. The contractor shall submit this memo to ED within 9 months after the effective date of the contract.

Deliverable: Memo on Pretest Results

Subtask 2.7: Prepare OMB Clearance Package

The contractor shall prepare the necessary forms required for OMB Clearance of the teacher survey. The clearance package must justify the necessity for collecting the data and comprehensively respond to each required item in the instructions. The forms clearance package shall include brief, concise statements of a) the study mandate and objectives, b) types of information to be requested, c) steps taken to minimize respondent burden, d) plans for tabulating data, e) the data collection schedule, and f) steps taken to have plans reviewed by outside persons, g) estimates of burden for respondents and how such figures were estimated, h) discussion of "sensitive" questions, if any, and i) estimates of the cost of the activity. The contractor shall append copies of the instruments, marked to show the study’s mandate and the voluntary nature of the respondent’s participation, as well as table shells indicating how data will be tabulated and analyzed.

Forms clearance typically requires approximately 120 days. Data cannot be collected without OMB approval and clearance. Therefore in planning and scheduling data collection, the contractor must devote sufficient time and resources to this product to assure timely completion of the clearance process. The contractor shall submit the OMB package to ED within 10 months after the effective date of the contract.

Deliverable: OMB Forms Clearance Package (draft and final)

Task 3: Survey of Secondary Vocational Teachers

In spring 2001, the contractor shall conduct a survey of a nationally representative sample of secondary teachers of vocational and other career preparation courses in comprehensive and vocational high schools and area vocational centers. This survey will be designed to collect information about classroom practice, training, state and local policy, and expectations for students.

Subtask 3.1: Sample Design

The contractor shall design a sampling plan that is representative of all secondary instructors of career preparation courses. Special care will need to be taken to ensure that the sample adequately reflects: (1) teachers not formally classified as vocational educators but who otherwise teach courses with a strong career preparation emphasis, and (2) teachers in area or regional vocational centers. The plan should also describe how the sampling might be conducted to fully reflect the diversity of vocational teachers- - including the range of program specialties and geographic areas where they teach. A sample of about 3,000 teachers is considered adequate to generate reliable estimates and to allow subgroup analysis of the data. The sample design for the teacher survey should be included in the revised overall study design, to be submitted to ED by month 2 after the effective date of the contract.

 

Subtask 3.2: Conduct Survey

The contractor shall conduct the teacher survey in spring 2001. If an Internet survey is planned, the contractor shall offer mail questionnaires for teachers who do not have access to the Internet. Whether a mail or Internet survey administration is pursued, the contractor shall plan to follow- up with non- respondents by telephone. In addition, the contractor shall prepare a letter of introduction to the sample members from ED after receipt of OMB clearance. This letter shall contain the following:

  • Explanation of the purpose of the survey
  • Indication that the appropriate OMB approval has been received and an estimate of the expected burden of the questionnaire
  • Statements stressing the importance of the survey and requesting the cooperation of all respondents; and
  • Provision of the names and phone numbers of both ED staff and contractor employees who should be contacted with questions and comments

The contractor shall submit a draft of this letter to ED within one week after receiving OMB clearance and approval and no later than month 16 after the effective date of the contract. The contractor shall mail the letter to the sample of teachers one- week prior to the beginning of the survey administration.

The contractor shall be responsible for tracking the status of all responses. ED anticipates an 85 percent or higher response rate to the survey. If the response rate is below 85 percent, the contractor shall undertake alternative measures in consultation with the COTR.

Deliverable: Accompanying Letter

Subtask 3.3: Protection of Data

The contractor shall be familiar with, and be prepared to comply with: The Privacy Act of 1974, PL 93- 579, 5 USC 552 a; The "Buckley Amendment," Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1- 1, 45 CFR Part 5b, and 40 FR 44502 (September 26, 1975); and, as appropriate, the Federal common rule or Department final regulations on protection of human research subjects.

The contractor shall be responsible for maintaining the strictest of confidentiality of all individual data collected in this study. The contractor shall under no circumstance release any personally identifiable information about sample members or the schools in which they teach unless such a release is legally required. The contractor shall maintain information which identifies persons or institutions in files that are physically separate from other research data and which are accessible only to authorized agency and contractor personnel. The contractor shall only use individual identifiers for purposes of data collection, establishing sample composition, authenticating data collection, or obtaining missing data. The contractor shall destroy all lists and codes of individual identifiers in any format within 24 months of completing final data analysis.

Subtask 3.4: Analysis of Survey Data

The contractor shall develop quality assurance specifications for the survey data, and data entry specifications if a mail survey is used. To ensure accuracy, the contractor shall verify all data entered by either respondents or data entry clerks, and conduct edit and consistency checks. The contractor shall resolve all problems identified through this process through phone calls to respondents.

The contractor shall prepare an analysis plan that specifies the basic tabulations that will be conducted for the entire sample and any relevant subgroups. The analysis will require simple descriptive statistics and cross- tabulations. The contractor shall attempt to determine typical or average responses and the degree to which there is variation in these responses.

The contractor shall submit a draft analysis plan to ED by month 15 after the effective date of the contract, and the completed data analysis tables by month 21.

Deliverables: Analysis Plan (draft and final)

Data Analysis Tables

Task 4: Case Studies

The case studies of state and local vocational education policy and practice are a key component of the National Assessment. The contractor shall design the case studies to provide an in- depth assessment of: 1) what constitutes high quality vocational programming, 2) the extent to which vocational education seems to contribute to student performance, and 3) how implementation is changing in response to Perkins III.

The contractor shall conduct site visits as the primary data collection activity within the case studies. Site visits shall consist of collecting existing information, observing vocational program activities at schools, and interviewing and conducting focus groups with key stakeholders. The contractor shall also collect information from other sources, where possible, such as state school/student databases.

A significant element of the site visits shall be in- depth examination of vocational instruction in a small number of classrooms. This examination shall include classroom observation, review of lesson plans, textbooks, tests, and other curriculum materials, and interviews with both teachers and students concerning standards of performance expected of students and students’ own objectives for taking the courses. The case studies shall provide qualitative and quantitative information on the many dimensions of vocational education planning and implementation.

Subtask 4.1: Sample Design for Secondary Case Studies

Choosing states, schools, and teachers for the case studies poses several methodological challenges. On the one hand, the study could focus on sites (and teachers) that demonstrate effective vocational education as measured by increases in a broad range of student outcomes. Such a strategy would promote one of the study’s objectives: documenting what constitutes high quality instruction and providing some evidence on how schools can use vocational education to improve student performance. This approach would also be consistent with the emphasis in Perkins III on measuring program performance based on developing students’ academic as well as vocational and technical skills.

On the other hand, including only very effective schools and teachers fails to provide an accurate representation of vocational education as a whole. The lack of contrast in quality also makes it more difficult to accurately identify high- quality practices. In addition, given the relatively short time frame of this study and limited availability of student outcome data, it may be challenging to find enough states for which the former sampling approach could be used.

Several possible sampling strategies are presented here. The contractor may also propose alternatives that can address the analytic priorities described above. The strategy should balance the interest in observing "promising practices" and high quality instruction with a need to examine the true distribution of quality in vocational education.

Based upon a review of state Perkins plans, existing reports and other documents, and conversations with ED staff, the contractor shall propose a total of 10 states for case studies and 5 schools within each state for more in- depth examination. States may be selected purposefully for the case studies. The contractor shall weigh several criteria in making selections, to ensure diversity in the state sample:

C The submission or absence of a unified state Perkins plan

C State emphasis on academic school reforms and/or workforce development initiatives

  • Geographic diversity
  • The presence or absence in the state of area vocational centers

However, the contractor shall give strong preference in choosing states to those that have established accountability systems. These systems typically include performance data on individual high schools or, in some cases, individual students over time. Selecting states with these data systems will allow the contractor to sample schools based on performance reports, even if performance data is not available separately for vocational students or includes only academic achievement and not also technical skills.

The contractor shall select 5 schools within each state for more focused data collection. If school performance data is available, the contractor may:

  • Choose 3 schools with rapidly improving student performance and 2 schools with slower or no improvement in performance.
  • Ensure, through interviews with state vocational staff or other means, that the selected schools have substantial enrollments in vocational education

If school performance data is unavailable, the contractor may seek nominations from state agency staff of 3 schools with strong vocational programs and practices. From among less well- regarded schools, the contractor may randomly sample two schools with substantial enrollments in vocational education.

Regardless of the sampling approach used, the contractor shall stratify the sample to ensure representation of rural and high poverty schools.

Given that states are likely to measure student performance in varying ways, it will also be important for the contractor to ensure that high and low performing sites across states are distinct.

The contractor shall select 3 teachers within each sampled school for more in- depth examination. ED is weighing two alternative approaches:

  1. Randomly choose 3 teachers from among the vocational and other career preparation teachers. This approach focuses on substantiating the prevalence of high quality practices at the classroom level.
  2. Purposefully select 3 teachers who are viewed by school administrators and faculty as exemplifying high quality vocational instruction. Using this approach, the primary purpose of the detailed classroom analysis is to identify and describe high quality practices in vocational education (rather than the prevalence of these practices on a broader scale).

The contractor shall include a discussion of the final case study selection approach in the revised study design report, to be submitted to ED by month 2 after the effective date of the contract.

 

Subtask 4.2: Selection of Sites for Secondary Case Studies

The contractor shall prepare a memorandum listing the proposed sample of states, schools/communities, and teachers for inclusion in the case studies, and describing the rationale for the selection of each site. This memorandum shall be submitted to ED by month 7 after the effective date of the contract.

Deliverable: Memo on Proposed Sites and Teachers

Subtask 4.3: Selection of Sites for Postsecondary Case Studies

The contractor shall prepare a memorandum listing the proposed sample of postsecondary institutions (approximately 20 community colleges and vocational- technical institutes with emphasis on the former representing all 10 states) located within a subset of the communities where secondary site visits are being conducted. The contractor shall ensure that postsecondary sites chosen reflect diversity in the types of higher education institutions that offer postsecondary occupational education, diversity in the demographic characteristics of participating students, region, and urban/rural location across the states. In addition, the contractor shall heavily favor those institutions containing or located near one- stop centers. This memorandum shall be submitted to ED by month 7 after the effective date of the contract.

Deliverable: Memo on Proposed Sites

Subtask 4.4: Train Case Study Staff

The contractor shall train the staff conducting the case studies, either jointly for the secondary and postsecondary components or separately. The contractor shall submit training materials to ED for review within 12 months of the effective date of the contract.

The contractor shall conduct training within 15 months after the effective date of the contract. This staff, whether in- house staff or persons hired specifically for the study, shall have experience in conducting qualitative research and case studies, and be knowledgeable about vocational education. The training shall ensure that study staff are thoroughly familiar with the study objectives, design, and data collection procedures. A major purpose of the training shall be to ensure that site visit data collection is as consistent as possible across site visitors, to increase the ability to make generalizations and comparisons across case study sites.

Deliverables: Site Visit Training Materials (draft and final)

 

Subtask 4.5: Conduct Case Studies

The contractor shall begin conducting the site visits to the states and communities selected to gather information for the case studies within 12 months of the effective date of the contract. The contractor shall collect information from a wide range of sources using the protocols developed under Subtask 2.4 and 2.5, and shall review all extant documentation and reports. Offerors shall assume that visits to local communities (sampled schools, their local education agency, and other relevant institutions and organizations) to collect information about secondary and postsecondary vocational education shall require approximately 6 person days per site for secondary data collection and 1- 2 person days for postsecondary data collection.; visits to state agencies shall require approximately 2 person days per state.

The contractor shall prepare a 10- 15 page memorandum for secondary and a 7- 10 page memorandum for postsecondary, summarizing preliminary findings from the case studies. These memorandums shall be submitted to ED by month 20 after the effective date of the contract.

Deliverables: Memo on Preliminary Case Study Findings

 

Task 5: Report Preparation

The contractor shall prepare two reports as part of this study of vocational education policy and practice.

 

Subtask 5.1: Report on the Quality of Secondary Vocational Education

The contractor shall prepare a report summarizing and integrating findings from the case studies of secondary vocational education and the teacher survey. This report shall address the policy and research issues described earlier, in a format that is easy to read and highlights major points and themes. The contractor shall submit an outline for the report to ED in month 21 after the effective date of the contract. Upon approval of the outline by ED, the contractor shall submit a draft of the report to ED 24 months after the effective date of the contract. Based on ED comments, the final report shall submit the revised final report 26 months after the effective date of the contract.

Deliverable: Secondary Synthesis Report (outline, draft, and final)

 

Subtask 5.2: Report on the Role of Perkins Funding and WIA on Postsecondary Occupational Education

The contractor shall prepare a final report that integrates findings from the various postsecondary site visits and includes documentation of all study procedures and any quantifiable data. In addition to descriptive statistics, the final report shall compare and contrast the primary implications of WIA and the role of Perkins funding on postsecondary vocational programs. Finally, the contractor shall discuss the federal policy relevance of the findings. The contractor shall submit an outline for the report to ED in month 21 after the effective date of the contract. Upon approval of the outline by ED, the contractor shall submit a draft of the report to ED 24 months after the effective date of the contract. Based on ED comments, the final report shall submit the revised final report 26 months after the effective date of the contract.

 

Deliverable: Postsecondary Synthesis Report (outline, draft, and final)

 

Task 6: Contractor Performance Reporting

In addition to the reports described in Task 5, the contractor shall submit one copy of the following reports, monthly, to the contracting officer, with one copy to the COTR:

  • Monthly Progress Reports/Exception Reports. The contractor shall prepare monthly progress reports due within ten workdays after the end of each month. They shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving problems.

 

  • Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due within ten working days after the end of each month. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

 

7. Schedule of Deliverables (Unless otherwise specified, the COTR will notify the contractor within two weeks of receipt of changes required in the deliverable, and the contractor shall have two weeks to make the necessary changes and submit a final deliverable to ED.)

1999

2000

1

2

3

4

5

6

7

8

9

10

11

12

13

November

December

January

February

March

April

May

June

July

August

September

October

November

Revised Study Design (1, 3.1,4.1)

Initial Meeting Memo (1)

Non- Technical Summary (1)

Memo on Quality Indicators(2.1)

Memo on Site Selection (4.2,4.3)

Draft Survey Questionnaire

(2.2)

Draft Observation Guide(2.3)

Case Study Protocols(2.4, 2.5)

 

 

Teacher Survey Pretest Results Memo (2.6)

Draft OMB Package

(2.7)

Site Visit Training Materials (4.4)

2000

2001

14

15

16

17

18

19

20

21

22

23

24

25

26

December

January

February

March

April

May

June

July

August

September

October

November

December

Draft Teacher Survey Analysis Plan (3.4)

 

 

 

 

 

 

Draft Survey Letter (3.2)

Memo on Preliminary Case Study Findings

(4.5)

Data Tables for Teacher Survey (3.4)

Secondary Report Outline(5.1)

Postsecondary Report Outline (5.2)

 

 

Draft Secondary Synthesis Report (5.1)

Draft Post- Secondary Synthesis Report (5.2)

Final Secondary Synthesis Report (5.1)

Final Post- Secondary Synthesis Report (5.2)

 

 



   
Last Modified: 08/23/2003