EVALUATION OF PROGRAMS
Assessing the Quality of Vocational Education in High Schools
Cathleen Stasz, RAND
Archived Information

Downloadable File PDF (49 KB) | MS Word (71 KB)

May 6-7, 1999
Independent Advisory Panel Meeting
National Assessment of Vocational Education

Like1 its predecessor, the Carl D. Perkins Vocational and Technical Education Act (Perkins III) directs state and local education providers to improve vocational and technical education programs. Most of these improvements are aimed directly at changing what teachers do and what and how students learn. The National Assessment of Vocational Education (NAVE) is charged with evaluating the impact of Perkins III and making a report to Congress by July 2002. Of chief interest to Congress is how these recommendations for program improvement are realized in high schools and how they affect what goes on in classrooms.

This paper, written to inform the design of NAVE, argues that to understand the quality and effects of curricular and teaching changes undertaken in response to Perkins III, the evaluation must focus on teachers and classrooms. This paper has three goals: (1) to examine the principal themes and issues related to assessing quality of vocational education (with an emphasis on teachers and teaching); (2) to lay out a research plan to assess quality practice; and (3) to discuss the conceptual, methodological, and practical challenges to conducting the research.


Themes and Issues

Perkins III promotes several program improvements that are intended to raise the overall quality of vocational and technical education programs in high school. The core program improvements address two main areas: (1) the content and delivery of vocational programs and (2) the training and staff development of teachers, administrators, and others in support of program delivery. The improvements in content and delivery emphasize strengthening the ties between school and work by integrating academic and vocational education, teaching "all aspects" of an industry, and providing students with skills to utilize technologies in the information age.

Perkins III also wisely recognizes that those engaged in these integration-related reforms, both in and out of school, may need specialized training and professional development to implement them. Teachers certainly need support for integration, since they bear most of the responsibility for redesigning curricula and classroom activities in support of student learning. Because integration-related reforms require closer connections with the workplace, employers are more likely to become involved in school reform efforts. Teachers and school administrators need to explore ways to effectively engage employers as reform partners; employers need to explore how to partner with schools. Appropriate professional development activities can assist in supporting those involved in the design and delivery of vocational programs to define and develop their new roles.

The program improvements related to content and delivery of vocational education carry over from the 1990 amendments to the Act (Perkins II). The extent and success of integration was assessed in the previous NAVE, which found notable increases in such efforts between 1992 and 1993. However, the evaluation also concluded that reforms varied widely in quality, size, and kind and included everything from simply relabeling existing programs and making ad hoc changes to well-planned restructuring efforts (NAVE 1994). While a few states had adopted comprehensive reforms of work-related education as a matter of policy, most school systems were attempting to fit the Perkins II reforms into existing curricula, rather than making broad changes. At the time of the previous NAVE, systematic evaluations of programs or curriculum in support of integration were few and often flawed, and, since few programs were actually in place at the time, the research focused on implementation processes rather than student outcomes (Stasz, Kaganoff, and Eden 1994).

In current practice, the content and delivery of vocational and technical education goes way beyond the specific issues mentioned in the legislation. Like Perkins II, the guidance provided by Perkins III remains somewhat fuzzy, a condition that encouraged a variety of programs to proliferate, with much experimentation and innovation (Stasz and Grubb 1991). For example, integration can occur at several levels, from relatively straightforward attempts to enhance the academics within vocational courses or to align a sequence of courses linked to an occupational area, to much more complex models that affect how a high school is structured (e.g., career academies with blocked scheduling; organization around career clusters instead of subject-matter departments). With the implementation of the School-to-Work Opportunities Act of 1994 (STWOA), the opportunities for high school students to participate in programs with a dual purpose of teaching academic and work-related skills and competencies has significantly increased. STWOA enhances existing programs or fosters the development of new programs which purposely seek to attract all students, not just those who traditionally enroll in vocational education. In addition to integrating academic and vocational education within school programs (through course sequencing, team teaching, career academies, career clusters, or majors), school programs increasingly promote integration through out-of-school activities, such as job shadowing, mentoring, or work-based learning (Hershey, Hudis, Silverberg, and Haimson 1997).

The expansion of work-related programs and courses through Perkins and related legislation, and the success of these programs in attracting new students create several problems for evaluators. First, it is difficult to identify the intervention. Programs calling themselves "integrated" vary widely in terms the extent of curricular integration they provide and the type of activities that students engage in, both in and outside the classroom. Each program can have multiple purposes and goals that shape the curriculum and instructional offerings. As a result, there is likely to be as much variation within programs of a certain type as between them. Many new programs, especially those developed through STWOA, may not be exclusively supported with Perkins funds and may not even be identified as vocational programs. Likewise, it may be difficult to identify participants as vocational students. However, the extent and success of these programs are surely of interest to Congress and other policymakers whether or not they flow directly from Perkins dollars or serve the traditional vocational student.

As has been mentioned, integration-related curricular reforms are embodied in various types of program models (Grubb, Davis, Lum, Plihal, and Morgaine 1991). It is within these models that changes in instructional practice--what is taught and how it is taught--take place. The bulk of the research on integration-related reforms to date provides descriptive data on the various forms of integration and the various challenges that education providers face in implementing integration-related reforms (e.g., Bodilly, Ramsey, Stasz, and Eden 1993; Schmidt, Finch, Faulkner, Isom, Magee, and Fox 1992a, 1992b). Relatively few studies, however, have investigated integration at the level of instructional practice, either by carefully documenting changes in the curriculum or by describing how teachers organize instructional activities. Few studies attempt to link practices with outcomes. Thus, relatively little is known about the quality of integration-related reforms at the classroom level that enables one to identify "best practices" for promoting desired student attainments.

The failure to identify best practice in vocational education is due more to the state of the research than to lack of understanding. One of the chief reasons given for integrating academic and vocational education is pedagogical: integration should promote instructional practices that emphasize the application of abstract or theoretical concepts (the academic) to real, practical problems (the vocational). The move to contextualized or applied or experiential approaches to teaching and learning has been developing from research on how people learn. Since at least the late 1980s, scholars explicitly use traditional vocational apprenticeship as a model for contextualized learning (e.g., Raizen 1989; Collins, Brown, and Newman 1989; Lave 1988). Others draw insights from studies of learning at work or from problem solving in everyday life to illustrate various shortcomings of school-based learning for helping students make use of formal knowledge (e.g., Resnick 1987; Scribner, 1982). As belief in the value of contextualized or applied pedagogies increases, the effect is to bring this central aspect of vocational education teaching into the mainstream of thought concerning "best" practice. While the theoretical work is suggestive, there is little systematic research on the effectiveness of these approaches, especially with respect to meeting academic standards.

The form and manner of integration-related activities are largely in the hands of school-based practitioners. Perkins III rightly emphasizes the need to support teachers and other school personnel with appropriate professional development. Teacher quality is central to the success of any initiative aimed at instructional practice, so it will be important to consider what preparation they have (preservice education, industry experience) and what kinds of professional development they need. Previous research suggests that few teachers are prepared to meet the challenges of creating curricula or altering their teaching to promote integration (e.g., Bodilly et al. 1993). Even when prepared or motivated, they are not well-supported in their efforts through such actions as shared planning time or other incentives. Studies of integration-related reforms spell out what kinds of professional development teachers need or propose inservice training models that will help teachers use constructivist teaching methods and develop integrated curricula (Stasz, 1997).2Although the opportunities for teachers to participate in staff development geared toward integration are on the rise, they are by no means common practice for most teachers in these programs. While the need for professional development for employers or administrators is recognized, much less work has been done to design or deliver appropriate activities for nonteachers.

The more ambitious attempts to integrate academic and vocational education require collaboration with employers. An important issue for NAVE concerns linkages with the employer community that help improve the quality of vocational education at many levels. The growth of work-based learning connected to school has focused research on employer participation, and the literature describes which employers participate and for what reasons, and how the education community can promote participation (e.g., Bailey, Hughes, and Barr, 1998). By and large, few employers participate in ways that directly affect curriculum and teaching in classrooms, nor is it clear whether such participation is feasible or desirable, at least at the high school level. Employers may have an even more important role to play, however, as industry skill standards continue to be developed and used as quality benchmarks for designing curricula. The federal government is supporting over twenty projects to develop industry skill standards, and NAVE provides an excellent opportunity to determine whether and how these efforts impact on high school programs.3

The legislative directives, coupled with the state of integration in practice, suggest a research approach with the following characteristics:

  • A focus on teachers and teaching that can describe integration at the level of classroom practice and identify relationships between teacher characteristics, instructional practices, and student attainments;

  • An emphasis on program types that have been less studied and potentially serve to restructure the whole school, such as career clusters or career major strategies. Such strategies are especially important because they serve to connect vocational education to other school reforms, rather than remaining on the periphery of the high school program.4

  • Careful attention to previous or ongoing studies on which the current assessment might build or extend.

The following section outlines a research strategy for assessing the quality of vocational education in high schools that incorporates these principles and the main issues discussed thus far.


A Research Strategy

Previous NAVE evaluations primarily utilized survey methods to examine the extent of integration or other specific program improvements discussed in the legislation. This is certainly an acceptable method for understanding whether a particular reform is widespread, but provides little information about how reform policies are specifically realized in everyday curriculum content or teaching methods. Although surveys will certainly have an important place in NAVE, a case study approach is more appropriate for examining how vocational education reforms affect classroom practices. Ideally, case studies of practice should be linked to quality standards and student outcomes, although this may not be possible given time and data limitations (discussed further below). What follows is a sketch of a study design, which can address several questions of interest, such as:

  • What are the general characteristics of programs that integrate academic and vocational education? What are the goals of the program and how do they relate to the school as a whole?

  • How does the curriculum support desired improvements (e.g., integration, "all aspects" of the industry, technology-related skills). Does the curriculum align with appropriate quality benchmarks, such as graduation requirements or academic or industry skill standards?

  • What teaching practices are evident in these programs? What kinds of activities do students engage in? Do the activities enhance contextualized/applied learning of academic and vocational subject matter (e.g., instruction anchored in and learning grows out of authentic problems, which lead to authentic products for authentic audiences5)?

  • What do students learn in these programs? What is the quality of the work they produce? How do they perceive the learning experience? What kinds of student performance data are collected and how are they used?

  • What are the characteristics of participating teachers (e.g., educational background, work experience)? What kinds of staff development activities have they engaged in to support their teaching? What kinds of further support do they need?

  • What are the roles of administrators and employers? How can they be better supported to achieve program improvement goals?

  • How does the program relate to district or statewide reforms, including standards and assessments or core performance indicators?

A replicated case study design is most appropriate because the interest is in examining and interpreting ongoing processes in real-world contexts, especially where the processes to be studied (teaching, learning) are not easily separable from the context and where the variables of interest outnumber the units of study (Yin 1994). In this design, similar sets of criteria are used to select sites and the individuals within them and common data-gathering instruments and procedures are used across sites. Because classroom practices are influenced by the broader institutional context, it is important to gather data at multiple levels (e.g., the classroom, school, district and state) and from a variety of informants (policymakers, school administrators, employers, teachers, and students).

Consider a study design focused on a purposive sample of 20 to 40 teachers and classrooms. Different sampling frames are feasible, but one approach is to select three states, three schools within each, and 2-3 teachers within each school for intensive study. The states and schools should represent serious efforts to recast vocational education along the lines recommended in the legislation. In addition, states which can provide data on student attainments (e.g., achievement tests, UI data for tracking post-high school attainments) or teacher quality (e.g., personnel records), and which are agreeable to sharing the data, are preferable. The school sample should include program forms that have been less studied, such as career majors or cluster arrangements. Within schools, programs and teachers should be selected with some thought to subject area, perhaps focusing on two career areas (e.g., health and technology/electronics).6 Participating teachers can be randomly selected from the targeted school/program. The overall intent is to draw a purposive sample where the conditions for creating and sustaining desirable practice are in place.

By nesting the sample of teachers and classrooms within programs, schools, and states, it will be possible to examine study questions at several levels and to track the relationships to classroom practices (both curriculum and instruction) or to teacher quality. The study design should utilize multiple data gathering activities and sources of data. For example:

  • Semi-structured interviews of state, district, school level administrators and policymakers, employers to examine various policy and implementation issues. Semi-structured interviews can gather core data from different respondents, yet also permit tailoring interview questions to specific situations and respondents and allow the interviewer to pursue useful lines of inquiry that may not be known prior to site visits.

  • Teacher surveys to include target teachers, plus a larger sample of teachers within the same school. The survey would focus on teacher background and staff development activities, curriculum, and teaching practices.7

  • Classroom observations. Assuming a two-year time frame, two observations per year, at least one of which should be videotaped8. Observations should be completely open-ended, with fieldnotes transcribed. Several computer-based programs for working with text-based data are available for coding and analysis. Observations are the best means to gather information on teaching practice.

  • Teacher activity logs for selected time periods can gather information (through closed and open-ended items) about specific lessons, including teacher objectives, student activities, curriculum content and teaching behaviors. Teacher assessments, quizzes, or homework assignments in connection with these lessons can also be used to gauge quality.

  • Student transcripts and surveys would gather information on student coursetaking and perceptions of curriculum and teaching.

  • Student work. Examples of student work in the case-study classrooms would provide important information about the quality of the activities, level of difficulty, and extent of contextualized learning.

  • Student records. Ideally, the study would include some measure of student attainment, either in school or post-school. Although the time frame is short, it will be important for NAVE to explore possible options, e.g., pre-post testing or analysis of available test data from school, district, or state.

  • Documents. Existing documents, such as state curriculum frameworks, course syllabi, industry skill standards, or policy documents will complement interviews or other data.

A general approach to analyzing the qualitative data is to begin with an a priori set of theoretical or conceptual themes that define key questions, variables, and expected relationships. The data-gathering methods discussed above should align with study questions and themes. Data analysis would follow an iterative process that begins by coding/organizing data by themes, but permits new themes or issues to arise. For example, the analysis of observational/video and log data might follow current theoretical and conceptual frameworks suited to describing contextualized teaching and learning approaches. The curriculum can be described along several dimensions, such as cognitive complexity, academic and vocational content, and adherence to accepted curriculum or industry standards. The analysis might proceed case by case, followed by cross-case analysis to identify any general findings.


Connections to Other Work

NAVE should explore ways to relate the study of teaching practices sketched here with recent or ongoing studies. For example, Bob Crain and his colleagues at teachers' college recently completed a longitudinal study of career magnet high schools (Crain, Allen, Little, Sullivan, Thaler, Quigley, and Zellman 1999). It may be possible to extend this work by following up on student attainments post-high school. A study of career academy programs being conducted by Jim Kemple and others at MDRC (Kemple, Poglinco, and Snipes 1999) may provide information about successful programs that could be further studied in NAVE. At LRDC, Lauren Resnick is leading a study on contextualized learning approaches for learning academic subjects. This project will likely yield both a literature review and some useful conceptual frameworks or study sites for future work.

The last NAVE commissioned a literature review on integrating academic and vocational education (Stasz, Kaganoff, and Eden 1994). Given the proliferation of programs and studies in the last five years, a review which focuses on quality and effectiveness of integration-related programs (rather than descriptions of implementation processes) would be most useful at this time. This review should be quite broad and include case studies of teaching practices as well as those using longitudinal data. David Stern and others, for example, are currently looking at participation in vocational and school-to-work programs and various student outcomes, using the NLSY97.


Difficulties and Limitations

Any evaluation of the quality of vocational education in high schools that is sensitive to issues of teachers and teaching practice faces a number of significant challenges.

  • How to define "quality?" Program quality can be identified along several dimensions, such as student outcomes, adherence to a set of accepted curriculum or industry standards, or adherence to some accepted model of contextualized learning. It is unlikely that student outcome data can be used to identify "successful" sites, because comparable data suitable for selecting programs in multiple states are simply not available. Therefore, decisions about "quality" (either for selecting programs/classrooms to study or for characterizing the teaching and learning that is observed in them) must be left to expert judgment following some conceptual or theoretical framework.

  • How to measure student outcomes or learning? A pre-post design within sampled classes and with nonparticipating students might be possible, if technical issues could be overcome.9 It may be possible to administer a standardized test such as Work Keys, which assesses applied academics (e.g., applied math, reading for information, applied technology) and other employability skills (e.g., teamwork, motivation, managing resources)10. Developing these types of skills are likely to be a concern of most integration-related programs. Within-school comparisons of selected student outcomes for participating and nonparticipating students are also feasible (e.g., GPA, attendance, credit pace, and other data regularly collected by high schools), but are problematic due to selection effects or other potential sources of bias.11

  • How to define "vocational" programs and students? The past decade has seen many important innovations in the design of high school programs that link school and work. The aim of Perkins II and other federal reforms has been to blur the lines between academic and vocational courses, students, or teachers. Thus, it is difficult to identify strictly vocational programs or students or to determine what vocational education's unique contribution to student learning.

  • How to measure instructional practice? The difficulties inherent in assessing instructional practice are well documented.12 Generally, survey data are most useful for gathering information about curriculum or course taking if the standard is knowing whether or not a topic has been taught and if it has been taught over several periods or weeks. Surveys are insufficient for gathering reliable data on teaching practice, especially when practice may be in transition. This is certainly the case for identifying contextualized teaching practices. When practice is in flux, nonsurvey data (teacher logs, interviews, observations) and the use of exams, quizzes, assignments, and student work provide a richer picture of classroom practices. These nonsurvey methods are, of course, more expensive to gather and analyze and present other difficulties in interpretation and inference.

  • How to measure teacher quality? Studies of teacher quality typically relate specific background characteristics (e.g., undergraduate training, certification, and scores on ability tests) with various student outcomes (e.g., Ehrenberg, Goldhaber, and Brewer 1995; Goldhaber and Brewer 1997). This line of research finds that some characteristics, such as teachers' educational background, are associated with higher student attainments. These studies do not link quality measures with teaching practices, however, so how practice relates to teacher characteristics is largely unknown. Within the NAVE time frame, it will be possible to gather information on teacher characteristics through self-report surveys and to relate some background characteristics with practices in a limited way. However, the study outlined here could not address supply and demand of vocational teachers or other labor market issues that are currently of great interest to policymakers. As mentioned earlier, the latter type of analysis requires detailed personnel data that few states systematically gather.

  • Are relationships between teaching practice and student outcomes generalizable? Determining statistical generalizibility requires data on a large number of teachers and students. This creates an obvious dilemma, because survey data, which would provide information on enough teachers, is by itself inadequate for assessing teaching practice. A case study approach, as described above, will provide more valid information, but is limited for determining statistical generalizibility. However, in order to get a sense of how integration-related reforms are realized in high schools at a level of detail that is useful to teachers, it is most sensible to pursue an approach that yields specific information about teaching practice--e.g., about the teaching strategies and tactics, approaches for designing integrated curricula and complimentary assessments, and the principles for creating effective learning activities.

The approach outlined here also has several important advantages. First, the inclusion of video observations in the case study design will yield an important contribution to research on teaching practice. Newer technologies make video data and analysis affordable, and, most importantly, make it possible to create a database that has use beyond NAVE. It is also quite feasible to obtain the necessary releases to use the videos for further research.13 The study would also help advance understanding of contextualized or applied teaching approaches that many believe will have most significant impact on student learning.

Video examples of effective teaching in integration-related programs would be very useful for staff development purposes, as they would provide specific models for teachers to observe.14 Presently, research suggests that there are many significant barriers to implementing integration-related reforms, several of which relate to teacher staff development. Some of these barriers are administrative, such as lack of time during the school day for teachers to collaborate to develop appropriate curricula. Many teachers lack explicit training in contextualized learning approaches and typical short-term staff development activities are often inadequate for providing the skills they need. Most teachers lack the training and skills to develop assessments that are appropriate for use in contextualized learning environments (Stasz 1997).

In sum, due primarily to time and data limitations, it is not feasible for NAVE to reliably assess teaching practice linked to quality standards and student outcomes. However, a replicated case study design that closely aligns multiple data sources and methods with selected questions of interest can provide new information about teachers, curriculum and instructional practices in programs that explicitly connect school and work. If the evaluation includes another wave of survey data, it will be possible to gather information on curriculum, teaching practice and student characteristics from a larger sample of teachers, as well as track the extent of implementation since the last NAVE. An updated literature review can both inform the analysis of the case studies, as well as summarize findings on the extent and success of integration-related reforms. By coordinating case study and survey methods, NAVE should be able to provide a useful picture of the quality of vocational education in high schools today.

Return to NAVE Home Page


   1 | 2 | 3
Print this page Printable view Send this page Share this page
Last Modified: 11/14/2005