Data & Research EVALUATION OF PROGRAMS
Evaluation Plan
National Assessment of Vocational Education (NAVE)

Secondary Vocational Education

The role of vocational education at the secondary level is evolving. Many policymakers, educators, and parents remain concerned about whether secondary vocational programs serve students' best interests in the long run. Previous studies suggest that completion of a vocational program has a positive effect on employment, at least in the short run, if students do not pursue postsecondary education[4] and if they obtain training-related jobs.[5] However, the earlier research also indicates that vocational education contributes to neither academic achievement nor postsecondary enrollment—the path desired by most students and their parents. Studies undertaken by the last NAVE confirmed that vocational instruction in the early 1990s was still largely traditional, with little focus on academic skills.[6] These concerns about the quality and outcomes of vocational education have contributed to the decline in participation of high school students since the 1980s. These same concerns have raised the stakes for gaining a clearer understanding of: (1) how participation in and impacts of secondary vocational education have changed since the earlier studies, (2) the extent to which secondary vocational education now reflects high quality practices, and (3) whether promising career education reform strategies are proving to be effective.

A. Participation and Outcomes

Perhaps the most important issue for vocational education, and for the current NAVE, is who participates at the secondary level and how well they fare in school and beyond. At least through the early 1990s, vocational education had come to be stigmatized as a high school "track" for students with poor academic capabilities, special needs, or behavioral problems.[7] This result alone would not be problematic, but vocational programs also appeared to contribute little "value added" to student outcomes. In the current climate, with increasing federal emphasis on program performance and on high academic achievement for all students, vocational education has much to prove.

Over the last five years, however, career education has been given new prominence by several initiatives targeted to students other than those traditionally served by vocational programs. In addition, some states and districts have worked to strengthen vocational courses.[8] Whether any of these efforts have successfully broadened the appeal and improved the impact of occupational programs needs careful examination.


Figure 5

Secondary Vocational Participation and Impacts

Research Issues


  • Enrollment trends
  • Special Populations Participation
  • Course Taking
  • Academic Achievement
  • Postsecondary Enrollment
  • Labor Market Benefits

Study Plans


  • NAEP Transcript Study
  • Analyses using NELS, HS&B, NLSY, NAEP
  • NELS and HS&B Academic Achievement Comparison
  • Analyses of state data

Who participates in secondary vocational education?

Updating the enrollment trends and characteristics of students served by high school-level vocational programs is of interest, particularly in light of growing reports from state directors that enrollment patterns have been changing. Congress and most educators, remain committed to providing students from special populations access to vocational education. However, many believe the quality of vocational programs is unlikely to improve without attracting a broader segment of the student population; or that the participation of a more diverse set of students will signal that quality improvements are being made. Several questions must be addressed:

  • Who enrolls in secondary vocational education and to what extent has this picture changed over the 1990s? How do the characteristics of vocational students compare with those of nonvocational students?

  • Does enrollment vary by high school setting and vocational program area?

  • Has the course taking (academic and technical) of vocational students changed?

PLANNED STUDIES

Descriptive statistics from national data. NAEP transcript data (1998) provide the most recent evidence of secondary vocational participation nationwide. Using NAEP and student transcripts from other data sources, trends in student characteristics, course taking and achievement will be documented.

What is the impact of vocational education at the secondary level?

As a result of changing objectives and educational priorities, it is important to understand the contribution of secondary vocational education to traditional objectives (technical competency, labor market outcomes, and general employability skills) but also to academic achievement and postsecondary enrollment. Among the research issues are

  • To what extent does vocational education contribute to students' academic achievement and chances of attending and succeeding in college?

  • What effect does secondary vocational education have on students' technical preparation for work, employability skills, and ability to meet employers' expectations?

  • What is the impact on wages and earnings? In the short run? In the longer run? For noncollege-bound students? College-bound students?

  • How do these outcomes vary for different groups of students, particularly students from "special populations?"

PLANNED STUDIES

Analyses of national and state databases: NAVE will use several national databases (HS&B, NELS, NLSY, and NAEP) to (1) document trends in student achievement, earnings and other outcomes; and (2) assess the effects of vocational education (on average and for different programs of study and student populations).

NAVE will also explore possible use of state level data systems to provide information on students' technical competencies and employability skills, as well as employer satisfaction with vocational students.

B. Quality of Secondary School Vocational Education

For nearly a decade, federal policy has attempted to improve the quality of vocational programs by strengthening the connection between vocational education and mainstream educational objectives at the high school level. These vocational improvements are intended to keep pace with and complement other reform efforts in high schools. States and local districts have been raising the academic coursework and skills required for graduation, making high academic achievement the paramount marker of a school's success. While other measures of school performance are also important (e.g., placement into higher education or career-oriented employment, reductions in drop-out rates, technical competency), efforts to increase academic attainment are likely to continue as a focus for school improvement. A major policy issue facing vocational education, then, is how it can support this central mission for high schools.

The "quality" of vocational education is clearly critical to this objective. Perkins III builds on prior legislation in emphasizing program improvement. While it may be too early to fully judge the educational system's response to Perkins III, Congress will certainly be interested in the extent to which actual practice is consistent with legislative and other views of what constitutes "effective" vocational programs.


Figure 6

Quality of Secondary Vocational Education

Research Issues


  • Strategies for Academic Improvement
  • Prevalence and Intensity of Basic Reform Strategies
  • contribution to Overall Education Reform
  • Teacher Preparation and Qualifications

Study Plans


  • Large Scale In-depth Case Studies
  • National Vocational Teacher Survey
  • Analyses of Student Surveys
  • Analyses of SASS and PRAXIS Results
  • Analysis of National Employer Survey

How can schools improve the academic performance of vocational students and what, if anything, is the relationship between vocational education and those improvement efforts?

Federal vocational education policy now places priority on ensuring that secondary vocational education students are academically well prepared for careers and success in postsecondary education. For some districts and schools, meeting these objectives requires substantial changes, including new policies or requirements, shifts in instructional methods, or modifications to course content; some schools have already undertaken these reforms. While Perkins III provides guidance on program improvement strategies, identifying the approaches used by schools that have actually raised the academic (and technical) competence of vocational students will be of great benefit to both policymakers and practitioners. Several critical questions need to be addressed, such as

  • Do high schools that have succeeded in improving vocational students' achievement organize vocational education differently than do schools that have been less successful?

  • What vocational education practices and approaches appear most promising in promoting academic achievement and technical competence?

  • What is the relationship between vocational education improvement and school reforms underway in many states and local communities?

  • Does federal vocational education policy, as implemented by states and districts, facilitate or impede implementation of effective programs of study at the secondary level?

PLANNED STUDY

Comparative Case Studies of Schools, Districts, and States: In-depth case studies will document and contrast state and local policy and school practice, and how they might contribute to student outcomes. Through intensive site visits, the case studies will obtain descriptive in-formation about the extent and quality of vocational programs, state and district efforts to support and improve the programs, and the linkages between these efforts and schools reform and workforce development initiatives. Overall employer satisfaction with graduates of vocational programs will also be gauged with data from an ongoing National Employer Survey.

What is the quality of vocational education and to what extent are federal strategies for improving vocational education quality reflected in actual classroom practice?

Perkins III continues to emphasize several basic reform strategies, the genesis of which was in Perkins II; e.g., integrating academic and vocational education, linking secondary and postsecondary vocational programs, and broadening vocational curriculum beyond its traditional emphasis on entry-level job preparation. It is therefore important to examine whether nearly a decade of federal efforts to improve the quality of programs have found their way into teaching approaches and classroom organization. This examination will be guided by several research questions, such as

  • Are states and communities making progress in implementing key reform strategies emphasized in Perkins III? What other approaches reflect quality in vocational education?

  • To what extent has professional development been used to support these strategies? What kind of preparation and qualifications do vocational teachers have that will help them make these improvements?

  • How does the quality of vocational instruction and teaching vary among different institutions (comprehensive high schools, vocational high schools, and area vocational schools), different communities, or the demographic characteristics of students? What factors affect the depth or quality of vocational education?

  • What are the barriers to improving the quality of vocational practice?

PLANNED STUDIES

Several approaches will be used to measure the prevalence of vocational improvement strategies.

Teacher Survey and Other Teacher Data: In the year 2000, the NAVE will survey a nation-ally representative sample of vocational teachers to collect information on curriculum and instructional methods. Information on teacher preparation and qualifications will come from the Schools and Staffing Survey (SASS) and from an analysis of PRAXIS test score results (used for teacher certification in many states).

Student Survey: Analyses of existing student surveys will help identify the frequency of certain practices and the extent to which they are becoming more common for vocational and other students: (1) survey of seniors in eight states in 1996, 1998, and 2000 (Mathematica), (2) National Longitudinal Survey of Youth-97.

Case Studies: Comparative case studies will provide data on the quality of practice, barriers to improvement, and use of professional development. A significant part of the case studies will be a focused examination of selected vocational teacher instruction, including classroom observation and a review of lesson plans, competency tests, planned projects, and student work as concrete evidence of instructional practice.


C. Effectiveness of Promising Vocational Education Strategies

In recent years, a variety of programs and reforms have been introduced to improve the outcomes of secondary vocational students. Some approaches have been narrowly conceived (making minor adjustments to a small set of courses or activities) while others involve broader curriculum and institutional changes. Many of these reform models and practices have become popular, expanding to more schools or becoming institutionalized in state policy. Some have been supported with federal grants.

Until recently, however, few studies have been conducted of sufficient size and rigor to provide evidence on how well and why these reforms work. Schools have to a large extent relied on limited information that such interventions as Tech-Prep, career academies, career clusters, and High Schools that Work are effective in improving student performance. Now that these and other reforms have been in place for some time, it is reasonable for policymakers and educators to seek answers to the following questions:


Figure 7

Effectiveness of Promising Practices

Research Issues


  • Impact of Promising Interventions
  • Key Factors of Successful Programs

Study Plans


  • Career Academies Evaluation
  • longitudinal and Comparative Study of HSTW
  • Comparison Group Study of Tech-Prep Graduates

Do these strategies, when well implemented, improve student outcomes?

Although developers and advocates of particular reform models have long cited student success, in fact there is little concrete evidence of the programs' positive impacts. Studies found that implementation of these initiatives has been uneven, which has in some cases stymied efforts to conduct large-scale evaluations of effectiveness. With longer experience, however, examples of outstanding implementation have emerged that can be used as test cases. Of particular interest are comprehensive program models that can be applied to either vocational students or as part of whole-school reform. Research on these models should address the following questions:

  • To what extent do these promising interventions make a difference in key student outcomes: academic achievement, technical competence, postsecondary enrollment, or employment?

  • What other outcomes are associated with the new strategies (e.g., technical literacy, work readiness, employer satisfaction)?

  • Are there differences in impacts for different groups of students (defined, for example, by demographics, socioeconomic status, skill level, or occupational interest)?

What specific elements do approaches that seem to work have in common?

While knowing which programs or reforms make a difference in student outcomes is important, so is understanding the elements that make some initiatives successful. For policy and technical assistance purposes, it is critical to determine whether key outcomes result primarily from, for example, innovative instructional techniques, changes in administrative structures, or simply raising core requirements. Looking across all of the studies may provide answers to the following key questions:

  • Which features of reform are essential to producing positive outcomes?

  • Do the importance of these features vary across school settings?

PLANNED STUDIES

Evaluations of Promising Programs and Practices: The NAVE will draw primarily on studies already underway to assess the impacts of promising interventions. These studies may include but not be limited to evaluations of

  • Career Academies: a random assignment evaluation conducted by MDRC; High Schools That Work: a longitudinal and comparative study of school-wide re-form being conducted by MPR Associates;

  • Tech-Prep: a matched comparison group study conducted by NCRVE; Career Magnet High Schools: possible extension of this random assignment evaluation to assess postsecondary and employment outcomes; and

  • Other: NAVE may commission shorter investigations of particular, innovative state or local practices that have the potential to improve vocational student outcomes (e.g., technology education curriculum, high stakes end-of-course exams in North Carolina and Oklahoma that measure student competencies, etc.).



4 Ferran Mane, Trends in the Payoff to Academic and Occupational-Specific Skills: The Short and Medium Run Returns to Academic and Vocational High School Courses for Non-College Bound Students (Working paper # 98-07) (Ithaca, NY: Center for Advanced Human Resource Studies, Cornell University).

5 National Assessment of Vocational Education, Final Report to Congress, Volume I, July, 1994.

6 Ibid.

7 Ibid.

8 Alan Hershey, Marsha Silverberg, Joshua Haimson, Paula Hudis, and Russell Jackson, Expanding Options for Students: Report to Congress on the National Evaluation of School-to-Work Implementation (Princeton, NJ: Mathematica Policy Research, October 1998).

Overview of the Evaluation PlanPrevious Table of Contents NextPostsecondary Occupational Education


 
Print this page Printable view Send this page Share this page
Last Modified: 09/16/2003