Data & Research EVALUATION OF PROGRAMS
 

Edging Toward Effectiveness: Examining Postsecondary Occupational Education

II. Beyond Perkins III: Five Principles for Effective PSOE Programs

Since the early 1960s, one of the central purposes of federal aid for vocational education has been program improvement. Most recently, in Perkins II and now in Perkins III, improvement has principally been defined as the integration of academic and occupational education, the creation of tech-prep programs, and the development of performance measures to reflect outcomes. However, there are many more dimensions to effectiveness than these three particular approaches, and NAVE ought to consider research related to effectiveness and program improvement that goes beyond the conception embedded in Perkins III.

In examining effectiveness, there is often no substitute for analyzing outcomes, and I propose several ways to research outcomes in Section II.6. However, often it is impossible to measure outcomes. In other cases, changes need to be considered before long-run outcome measures can be available, and in any event evidence of poor outcomes is literally useless without some sense of what kinds of improvements might enhance outcomes. Therefore it is almost always necessary to have some idea of what elements of PSOE programs are necessary for greater effectiveness, in addition to (and sometimes instead of) information about outcomes. Over the last several years I have developed five criteria to judge the effectiveness of many types of occupational education and training programs, based on my reading of the available evaluation and econometric literature and on the characteristics of particularly effective programs.13 Others may differ about whether these are precisely the right criteria and about how important each of them is, but they at least provide a starting point for thinking about effectiveness — and the potential research that might illuminate improved practice. Each of these five criteria therefore serves as a starting point for at least two kinds of research: determining what conventional or modal practices in PSOE are, and identifying promising practices for others to emulate.

II.1: Labor Market Information and Connections

Effective programs understand the local labor market, and target those jobs that are likely to employ individuals with community college credentials, with relatively high earnings, strong employment growth, and opportunities for individual advancement. Some community colleges do this through careful local needs assessments; others, particularly those with extensive cooperative programs, accomplish this by maintaining stable links with local employers. Programs in high employment areas, and many programs during the current economic boom, can get away with ignoring this recommendation, of course, though probably at the expense of long-run effectiveness. But those colleges that fail to consider the hiring requirements and the quality of jobs for which they provide occupational preparation are likely to place individuals in minimum-wage positions with few prospects for advancement, with dismal results over the long run.

There are various mechanisms by which local institutions can keep abreast of labor market patterns and trends, including formal labor market assessments, advisory committees, the local contacts of individual instructors, placement office, follow-up studies, and — most impressive of all — the contacts afforded by high-quality cooperative education programs. However, the extent of these efforts varies substantially, and some-for example, advisory committees that meet once a year for ceremonial functions, or placement offices that stress low-quality "stay-in-school" jobs, or instructor contacts that are uneven and idiosyncratic-are not much use at all (Grubb 1996b, Ch. 6). Thus, contact with local labor market conditions is far from automatic in PSOE, and may need improvement in many cases.

Two kinds of research are possible. One is to ascertain how the mechanisms that potentially connect local programs with employers and labor markets actually function, through local case studies (see IV.4). An adjunct to such a study would try to identify best practices from around the country, to provide guidance to community colleges and other PSOE providers about the most effective ways of maintaining connections with local employers.

The second and complementary study would question employers about their practices-particularly about their proclivity to hire from local PSOE programs rather than other sources, their preferences for experience over formal schooling, the kinds of promotion practices they use and the role of formal PSOE in promotion. Some observers have concluded that many employers — at least those at the sub-professional level-hire on the basis of skills attained regardless of the source of those skills, so that community colleges students compete with those trained in the military, those who have learned through experience, and others who have picked up skills in informal or self-taught ways. In other cases, licensing requirements, company policy, and formal or informal agreements with local colleges create tighter linkages between formal PSOE and employment. Further analysis of employer practices — through either survey methods, interviews (see IV.8), or the analysis of several local labor markets (see IV.5) — would help clarify where employment practices encourage rather than discourage hiring from local PSOE programs.

II.2: Curriculum and Pedagogy

Effective programs contain an appropriate mix of academic (or remedial, or basic) education, occupational skills, and work-based learning, in the best cases integrated with one another. In addition, while flexible and innovative scheduling may be necessary, the intensity of both academic and vocational education must be appropriate to the jobs being targeted. Finally, effective programs pay attention to the pedagogy of everything they teach, whether classroom-based or work-based.

This principle, which really embodies three interrelated conditions for effectiveness, clarifies the obvious point that PSOE is an educational program. If the fundamental conditions for learning are not present, then PSOE can at best serve as a signal of motivation, ability, or some other valued characteristic but cannot make students more productive. Research under this principle is potentially vast, since there is virtually no analysis in this country of the effectiveness of different approaches to curriculum and instruction (Achtenhagen and Grubb 1999). One strand includes the research on the incorporation of academic competencies proposed above in Section I.2.

Another strand could include more focused research on work-based learning, and its integration with other components of instruction. Despite the recent interest in this country in work-based learning, and the small amounts of funding from the School-to-Work Opportunities Act, work-based learning is comparatively rare in most community college (except in certain fields like nursing). The questionnaire-based surveys of community colleges provide one opportunity to ask about the extent of work-based learning, though such results would no doubt simply replicate those available from earlier studies (e.g., the Bragg studies cited earlier). The case studies (see IV.4) are probably more effective ways to ascertain the role of work-based learning, including the resources available to support it, the nature of its integration (if any) with the rest of the curriculum, and (potentially) measures of effectiveness if the case study institutions have adequate data on student outcomes.

The other topic that has been virtually unexplored in the area of curriculum and pedagogy is the nature of teaching in PSOE. Aside from one chapter describing occupational classes in Richardson, Fisk, and Okun (1983) and an analysis of occupational teaching in several chapters of my recent book (Grubb and Associates 1999), there's been nothing written about the nature and challenges of teaching in postsecondary occupational subjects. (For a broader review of the international literature, see Achtenhagen and Grubb 1999.) From the work that has been done, I have argued that occupational teaching is in many ways more complex than academic teaching: it incorporates a greater variety of competencies or "intelligences"; more settings, typically including classrooms, workshops, and sometimes job placements; different and often more demanding forms of reading, writing, and mathematics than are taught in conventional academic classes; a greater number of literacies including specialized diagrams and charts as well as conventional writing and mathematics; and distinctly different approaches to teaching that I have labeled "skills" versus "systems" approaches, paralleling the distinctions between behaviorist and constructivist teaching in other subjects. Yet for all this complexity, PSOE instructors receive virtually no preparation in teaching (save in Iowa and a few isolated community colleges), and most postsecondary instructors learn how to teach through trial and error. The classroom observations I have recommended as part of institutional case studies (see IV.4) could readily be extended into a special study of its own. This special study might, for example, concentrate on PSOE teaching within selected occupational areas, or within occupational areas subject to licensing or to industry-generated standards (see IV.1). or within institutions where instructor training is practiced. In any of these forms, its central purpose would be to further investigate the nature of pedagogy in PSOE, the origin of different instructors' approaches to teaching, and the potential ways of improving instruction.

Finally, I should acknowledge the difficulty of addressing questions of the appropriate intensity for PSOE programs. The dismal outcomes of job training programs are due in substantial part to their limited duration and intensity; within PSOE, the economic returns are clearly higher to those completing associate's degrees than to individuals with certificates taking shorter amounts of time or those with varying amounts of coursework. If more is better, then how much is enough? And when is more too much? One way to examine this question is to examine the industry-generated standards and licensing requirements described in Section IV.1, since these specify from the employer's perspective how much education is enough for various positions. Another is to examine the debates within occupational fields about the intensity of postsecondary education, though this discussion can take place only in occupations where there is some kind of organization-either an industry association, a licensing body, or an association of instructors-to convene such a discussion. For the moment, the former seems an easier starting point.

II.3: Support Services

Effective programs provide a variety of support services, as appropriate given the needs of their students. While PSOE may be principally an educational program, an institution that fails to consider the need for support services may have low completion rates and fail for that reason alone. In community colleges and other forms of PSOE, five kinds of support services seem particularly important for students in occupational education:

  • Guidance and counseling mechanisms: Particularly for "experimenters" unsure of what they want to do, or dislocated workers looking for alternative occupation, or welfare recipients seeking a way back into the labor force, guidance and counseling is particularly important. Guidance and counseling mechanisms could include introductory courses or project-based efforts to get students to learn about occupational alternatives and the efforts of occupational instructors themselves, and not simply the efforts of guidance counselors in their offices. The Puente program in California, which creates learning communities of instructors along with counselors, provides another interesting model of integrating counseling into instructional programs.

However, as in high schools, there are many complaints about the adequacy and type of counseling available in community colleges and other postsecondary institutions; counselors are often criticized for providing information solely about academic issues and transfer, neglecting career-oriented counseling. While instructors and administrators can describe what happens within their own institution, there seems to be no central source of information about postsecondary guidance and counseling, no established conception of the developmental processes involved in adult decisions about careers, no conventions about good practice. NAVE could provide a substantial service to PSOE by carrying out, perhaps as part of case studies (IV.4), an analysis of what's now available in postsecondary guidance and counseling and some method of developing conceptions of good practice.

  • Remedial/developmental education: The problem of underprepared students in all postsecondary programs has grown substantially, and PSOE is no exception. Community colleges and technical institutes typically provide some forms of remedial or developmental education, though the type and quality of these programs varies enormously, and almost nothing is known about the effectiveness of these programs in helping students complete their goals. In the best cases, individual instructors have developed their own approaches (often constructivist) through trial and error, or a developmental studies department has devised a coherent policy, or instructors have linked developmental courses with occupational courses in learning communities. (See also I.2 above, on integrating academic and occupational education for remedial purposes). In others forms of PSOE, remedial education may be missing, or students may be directed to adult education programs of varying (and mostly abysmal) quality.14

The nature and effectiveness of remedial/developmental education is a large subject, applicable to most postsecondary institutions, and this is not an issue that NAVE can readily research on its own. As with many other questions, the case studies of local institutions (see IV.4) provide one opportunity to examine the nature of these efforts, their relationships (if any) with occupational programs, and (if institutions have good enough data) their effectiveness. However, NAVE might also explore the possibilities for joint research with other government agencies and centers concerned with the quality of postsecondary education.

  • Retention programs: The issue of who is a completer, and whether completion as defined by certificates and associate's degrees is important, is one of the most vexing questions in PSOE, one that I examine more fully in Section IV.1. For the moment, however, it is sufficient to point out that community colleges have made various efforts to increase retention and completion. They can be categorized as sorting, supporting, connecting, and transforming strategies (Beatty-Gunther 1994), and the best community colleges use a variety of them. Again, institutional case studies (see IV.4) provide the easiest ways of learning more about the nature of these efforts and their interactions with occupational programs more specifically.
  • Placement efforts: In occupation-specific forms of education, the benefits may be lost unless an individual finds employment related to his or her field of study. Results from national data confirm that the economic returns to PSOE are much higher when individuals find related employment, especially among women (Grubb 1997 or 1999); and some preliminary California results based on UI data corroborate these findings. Therefore, occupational education by itself may be necessary but not sufficient to gain access to well-paid employment, and placement efforts may be necessary as well. Again, local case studies provide low-cost ways of examining what community colleges and other local PSOE programs now offer.

  • Services for students with disabilities: Because community colleges are open-access institutions with relatively low spending levels, the diagnostic procedures to identify students with learning disabilities are often weak, and services to help such students are often inadequate. Community college instructors report that they suspect having students with learning disabilities, but do not know how to help them and have nowhere to refer them. Such students often end up in remedial classes, which may not be appropriate. While I am not the right person to recommend how to address this issue, a fuller analysis of support services would consider carefully the availability of such disabilities and their effects on the progress of the students themselves as well as their effects on other students.

II.4: Connections to Other Programs

Effective programs provide their clients or students with pathways or "ladders" of further education opportunities, so they can continue their education and training when they are able to. One issue of creating such pathways in PSOE involves the link between high school and community college, in the tech-prep programs described in Section I.3 above. The other common linkage is that between community colleges and four-year colleges, in articulation mechanisms to smooth the process of transfer. In many occupational areas-e.g., business, computers and information technology, engineering technology, some health occupations-the possibilities of transfer are substantial, and both instructors and institutions provide encouragement and support (like transfer centers); in other areas — automotive areas, metal trades, construction trades, cosmetology — there are no comparable baccalaureate-level programs and so articulation mechanisms are scarce. The possibility of and support for transfer should be one of the issues incorporated into local case studies (see IV.4).

However, the most important kinds of connections for NAVE to investigate are probably the connections among different types of occupational education and training programs. These connections — the efforts to create more coherent systems of work force development encompassing short-term job training, adult education, welfare-to-work programs, training for incumbent workers, and occupational education in postsecondary institutions — have been the subject of substantial research by NCRVE (see Grubb et al. 1999, for the latest effort) as well as research by the earlier NAVE (e.g., Hollinger and Harvey 1994). However, changes in federal legislation — particularly the Workforce Investment Act (WIA) of 1998 and the changes in welfare legislation passed in 1996 — have substantially changed the conditions within which local programs operate, and therefore new research may be warranted. In particular, the emphasis of both welfare legislation and WIA on "work first" conceptions-in which individuals are first placed in employment without worrying about their competencies — contradicts the human capital emphasis of PSOE, and the resolution of this conflict is a central issue.

The issues involved in relations with other programs include the provision of services and special programs for welfare recipients (if any); the ways PSOE institutions participate in state and local Workforce Investment Boards; the incorporation of PSOE into information and services provided by local one-stop centers, which play a central role under WIA; the eligibility of PSOE for the Individual Training Accounts (ITAs) provided some individuals, the numbers and types of individuals who actually use ITAs to attend community colleges and other forms of PSOE; and the difficulty PSOE institutions have in meeting WIA accountability measures (see I.1 above). The period within which NAVE will operate-with data being collected roughly between summer of 1999 and summer of 2001 — will be one of substantial turmoil, as states create Workforce Investment Boards under WIA and begin planning local efforts, and therefore it may be possible only to understand how PSOE is participating in this process and not what the outcomes are. 15

One way to carry out this research is again to incorporate questions about participation in WIA into the questionnaire surveys (see IV.3) and local case studies (IV.4). However, because these issues involve interactions between PSOE and other programs, it would be far better to conduct research examining WIA efforts and welfare programs as well as PSOE. This can be accomplished with community case studies, where several communities are identified and all the programs within these communities providing PSOE, job training under WIA and welfare, adult education, and other forms of work force development are examined to learn about their interactions (see IV.5).16 In this way the perspectives of each program about the others can be ascertained, rather than viewing the cooperation among programs solely from the vantage of community colleges and other PSOE institutions. Of course, the communities selected for such studies could be a subset of the communities in which the institutional case studies (see IV.3) are conducted, so that the community cases studies are effectively extensions of the institutional case studies.

II.5: Data and Improvement

Effective programs collect appropriate information about their results and use these to improve their quality. In the case of PSOE, these informational issues include the development of accountability measures, examined above in Section I.1, and the development of state and local data on employment outcomes, examined in the next section.

II.6: Outcome Measures: National and State Sources

To many researchers and policymakers in particular, outcome measures-rather than the elements of program structure I have covered in Sections II.1 to II.5-are the only reliable indicators of effectiveness. The good news is that better data have become available in the last ten years, both from national data sets and from state data sets based on UI wage record data.

In terms of national studies, I have recently surveyed the available literature for the Community College Research Center (Grubb 1999), and I see no reason for replicating that survey. There are, however, some topics that have not been extensively studied, and if NAVE wants to invest in further analyses of national data, it should concentrate on these issues. One of them is the problem (see also IV.1) of whether students who complete small amounts of PSOE benefit or not; this requires disentangling the effects of field of study, the relatedness of education and subsequent employment, and student intentions (i.e., for upgrade training) on economic returns, since it's likely that all of these influence the value of small amounts of education. A second issue that merits more exploration is the effect of finding education-related employment, since this has been investigated in only one study (Grubb 1997); there are obvious selection and self-selection effects to consider, as well as alternative ways to define education and employment as being related or not. Third, the effect of local economic conditions and of the business cycle on the employment benefits of sub-baccalaureate education have not been studied, though they are arguably quite important. A study reanalyzing certain data sets — particularly the Survey of Income and Program Participation and the National Longitudinal Study of Youth — could address these unanswered questions (see IV.8).

States have made great progress in developing data based on UI data. A few states have been leaders in these efforts, including California, Washington, Texas, Florida, and North Carolina. However, the analyses they have carried out so far are relatively unsophisticated (partly for political reasons), and have generally been confined to a few cross-tabulations. One valuable NAVE project would therefore be to take one or more of these state data sets and carry out more sophisticated multivariate analyses (see IV.10). In addition to carrying out such analyses, such a study should consider how to translate complex findings into simpler terms, since policymakers and administrators are unable to read regression coefficients. In addition, I propose in Section IV.1 using these state data, in conjunction with other information, to carry out further analysis of the complex issue of completion.

One other use of state UI data would be to conduct an outlier study. In the school effectiveness literature, particularly effective (and ineffective) schools were identified with regression analysis of test scores controlling for family background and school resources; schools with high positive residuals were then considered especially effective, and case studies were conducted to see how they differed from other schools. Similarly, in the realm of job training, Chris King and his colleagues (King, Lawson, Olson, Trott, and Baj 1998) selected especially effective JTPA programs in Texas and Illinois based on earnings over 155 percent of poverty and on continuous employment. The outliers they visited were committed to a structured process for determining high demand and emerging occupations; they emphasized occupational skills training rather than short-term job search assistance and on-the-job training; they stressed certain support services, including intake, assessment, counseling, and case management continuing throughout the program and afterwards; and they constrained the choices of training activities, referring individuals only to approved providers and only in high-demand or emerging occupations. (These characteristics correspond to the principles articulated in Sections II.1, II.2, and II.3 above.) A similar outlier study could be conducted of community colleges, for example, by using regression analyses of a state's UI data, identifying programs with high placement rates and earnings controlling for student characteristics and local economic conditions, and visiting those programs to see what characteristics of effectiveness are present. Such a study would build on the reanalysis of a state's data (see IV.10), and then add a small number of additional case studies.


[ Go to ED Home Page ]

This page last modified August 11, 1999 (glc).

 
Print this page Printable view Send this page Share this page
Last Modified: 10/24/2003