Data & Research EVALUATION OF PROGRAMS
National Assessment of Vocational Education

What Can We Learn About Postsecondary Vocational
Education From Existing Data?

Thomas Bailey and Gregory Kienzl
Community College Research Center
Teachers College, Columbia University

May 6 - 7, 1999
Independent Advisory Panel Meeting
National Assessment of Vocational Education


This paper outlines an agenda for research on postsecondary vocational education that can be carried out with existing data. Although we take a broad view of this mandate, we are particularly interested in encouraging research that can provide insight into the effects of the Carl D. Perkins Vocational and Technical Education Act of 1998 (the Act).

The Act prescribes a set of performance indicators designed to promote continuous program improvement, yet some performance measures may not accurately nor completely measure the educational and skills training mission performed by community colleges. In fact, certain measures may adversely affect the two-year postsecondary education sector. We argue that this issue becomes especially important in light of the emerging trend of nontraditional pathways taken by students through higher education. Although fewer and fewer students proceed through higher education in an uninterrupted linear path at a single institution, this image continues to dominate postsecondary education assessment. Helping to resolve the problems of assessment and institutional accountability within the rapidly changing landscape of postsecondary vocational education is one of the most important roles that projects using existing data could play for the National Assessment of Vocational Education (NAVE).

We propose a set of six projects. Most of these can be done as stand-alone projects but some of them would benefit from coordination. One of our central goals is to propose a portfolio of projects that can help define what is meant by "success" in postsecondary vocational education. That is, what is the standard against which to assess the performance of these institutions? Where possible, we also suggest some analyses that will begin to evaluate the performance of individuals, groups of institutions, and community colleges as a whole against those standards.

We begin with a brief review of relevant sections of the Perkins Act, followed by a discussion of the changing pathways through postsecondary education. This is followed by the descriptions of our proposed projects.

In discussing postsecondary vocational education, we focus primarily on public community colleges. The definitions within the Perkins Act call for an emphasis on community colleges rather than four-year institutions, and these are the postsecondary institutions influenced most heavily by the legislation.

One of our objectives is also to provide some background information on available data that can help the NAVE staff judge what other issues might be successfully addressed with available data. We do this first by listing, throughout the paper, possible data sources to address relevant issues. We also include three appendices. The first lists the dates for the next waves of data collection for five of the most important educational data sets. Appendix B reproduces some of the questions raised in the "Key Issues and Strategies" planning document and specifies the data sets that could be used to address those questions. Lastly, Appendix C provides descriptions of ten data sets that have been, or can be, used to analyze postsecondary vocational education.

Description of Relevant Sections of the Perkins Act

The two major focus areas of the new law are to (1) increase accountability and (2) provide states with more flexibility to use these federal funds.

Accountability

The state must make progress on continually improving the performance of vocational and technical education students. Therefore, each state will negotiate expected levels of performance with the Secretary of Education. The eligible community college, or "agency," must establish state performance indicators that, at a minimum, must include measures of:

  1. Attainment of academic and vocational-technical proficiencies;

  2. Attainment of a secondary school degree, GED, or postsecondary degree or credential;

  3. Placement, retention, and completion of postsecondary education, placement in military service or employment; and

  4. Participation in and completion of programs that lead to nontraditional training and employment.

The levels of performance must be objective, quantifiable, and measurable. Along with the core indicators of performance mentioned above, the eligible agency may choose to include additional indicators of performance. How can available data contribute to an assessment based on these performance indicators?

Available data can be used to assess the performance on each of these indicators, except the first, of the community college system as a whole. (We return to this later in our discussion of new pathways through higher education.) In some cases, the performance of individual institutions can be assessed using the degree completions file of the Integrated Postsecondary Education Data System (IPEDS).

Second, in addition to assessing the performance of the system, available data can be particularly helpful in evaluating the system of performance indicators. We argue that the concepts of placement, retention, completion, and degree attainment are increasingly complex and ambiguous. In other words, simple measures designed to assess "success" may be misleading. Thus research using available data can help clarify the shifting meaning of these concepts and help understand the implications of different performance indicators.

Flexible Use of Federal Funds

Perkins focuses the federal investment in vocational and technical education on high-quality programs that:

  • integrate academic and vocational education

  • promote student attainment of challenging academic and vocational and technical standards

  • provide students with strong experience in, and understanding of, all aspects of an industry

  • address the needs of individuals who are members of special populations and involve parents and employers

  • provide strong linkages between secondary and postsecondary education

  • develop, improve and expand the use of technology

  • provide professional development for teachers, counselors, and administrators

The extent to which community colleges are addressing these issues cannot be determined with available data. Most of these issues can be more fruitfully analyzed through case studies or new data collection. Rather than proposing special projects designed to evaluate service to special populations (which could be addressed using available data) we suggest that all relevant studies of access, retention, completion, and so forth take into account differences among different population groups. If we take a broad view of the meaning of the mandate to promote flexibility of funds for use in high quality and relevant programs, we do think that available data could be of use in assessing the extent to which community colleges adjust to the changing needs of their local economies, so we do propose a project in this area.

Accountability and Emerging Pathways Through Higher Education

We suggest that one crucial role that NAVE can play is to help redefine the commonly used accountability measures in light of changing patterns and purposes of postsecondary education. Measuring accountability is most straightforward when it involves traditional pathways through college–continuous attendance at one institution through graduation followed by enrollment in additional education or employment. This has been disrupted by two broad trends. The first is the increasing importance of interrupted attendance and multi-institutional enrollment (either sequential or simultaneous). The second is a growing use of training and education that does not lead to an associate's degree or certificate–"Competence without Credentials" as a recent Department of Education publication put it.

Traditional Educational Pathways

Generally speaking, there are five points on the traditional educational pathway of a typical community college student. The first is access, followed by transfer, dropout or stopout, degree or certificate completion, and posteducational work status. The traditional linear model of postsecondary attendance allows these stages to be studied in isolation and indeed there is a large literature on each one. These will be briefly reviewed below, indicating some of the most important conclusions as well as data sources that have been or can be used to address the issues.

Access: The first step on the traditional educational pathway–access to a postsecondary institution–is a familiar ground covered by national reports. For instance, a National Center for Education Statistics (NCES) study conducted in 1997 used the National Educational Longitudinal Study of 1988 (NELS) data set to examine access to postsecondary education of 1992 high school graduates two years out. The study also profiled these students by their socioeconomic status, race and ethnicity, and parental levels of education (U.S. Department of Education, 1997a). Supplemental data from the National Postsecondary Student Aid Study (NPSAS) and High School and Beyond (HS&B) can also be used to answer questions about how college costs and financial aid affects attendance status.

Transfer: A common view of a community college is as a "transfer-readying" institution–where students prepare to transfer to a four-year institution. Moreover, transfer has been a primary focus of much community college research over the last two decades. Data on transfer are primarily collected in Beginning Postsecondary Study (BPS) and HS&B. Student transcripts are available for the BPS and HS&B cohort, which greatly improves the accuracy of the data. Some of the highlights of existing studies on transfer are summarized below:

  • One out of four community college students indicated in 1989—90 that they were working toward a bachelor's degree (prospective transfers). Of this group, 39 percent transferred directly to a 4-year institution by 1994.

  • Among community college students identified as prospective transfers, those who enrolled full time in their first year were about twice as likely as those who enrolled part time to transfer to a 4-year institution within 5 years (50 percent of full-timers transferred, compared with 26 percent of part-timers).

  • Among community college beginners who transferred to a 4-year institution, 65 percent transferred without a degree. About one out of three completed an associate's degree before transferring.

  • On average, community college beginners who transferred to a 4-year institution spent about 20 months at the first institution. They often took a considerable amount of time off between institutions, averaging 21 months.

  • While one out of four community college transfers had received a bachelor's degree by 1994, another 44 percent were still enrolled at a 4-year institution, for an overall persistence rate of 70 percent. This is comparable to the persistence rate among students who began at 4-year institutions and among 4-year horizontal transfers.

  • The bachelor's degree attainment rate was much higher among the minority of community college transfers who completed an associate's degree before transferring: 43 percent of associate's degree completers had received a bachelor's degree by 1994, compared with 17 percent among those who transferred without any credential. (U.S. Department of Education 1997b).

Dropouts and stopouts: Dropouts, stopouts, and student persistence can be determined using NELS, HS&B, and BPS, respectively. For example, in a recent NCES report using the BPS data set, the educational experiences of students who leave college in their first year. The report tracks the path of those who return (stopouts) to determine where and when they enrolled. The report also compares the background and school experience of stopouts with those who did not return (stayouts) to college. The analysis reveals that nearly 30 percent of 1989—90 beginning students left postsecondary education in their first year. While a majority of stopouts in the four- year sector transferred elsewhere, in the public two-year sector, the opposite pattern occurred. Fifty-seven percent of students at public community college returned to the same institution, while 43 percent transferred (U.S. Department of Education 1998).

The important point to keep in mind is that institutions only "retain" students whereas the students themselves "persist" (Adelman 1998, p. 3). Students start and stop, attend multiple institutions simultaneously or sequentially, leave without degrees or certificates, or enroll with no intention of earning a degree. This topic will be addressed in more detail below.

Degree attainment: As mentioned earlier, IPEDS can be used to determine the number of degree and/or certificate conferred by two- and four-year postsecondary institutions. On the other hand, studies of degree attainment using BPS, HS&B, NPSAS, and NELS data sets will have students as the unit of analysis.

Work status: Depending on whether an individual perceives himself/herself mainly as a student who works or as a worker who occasionally takes class has a tremendous influence on the educational trajectory chosen. Studies using BPS, HS&B, and NPSAS can show how the amount of hours worked while enrolled affects student completion and persistence. Other national data sets, particularly the Current Population Survey (CPS) and the Survey on Income and Program Participation (SIPP) can give employment status and wage data for individuals who completed a certain level of education.

Nontraditional Pathways: What is Success?

In order to arrive at an appropriate set of accountability measures, we need a clear mapping of emerging nontraditional pathways through higher education. A great deal of progress can be made on this using national longitudinal data sets, albeit with considerably more time and energy. Such factors as degree objective, type of institution attended, timing of enrollment, enrollment intensity and continuity, transfer, financial aid receipt, and student employment should be taken into account in these analyses. A handful of studies have already started this work.

A recent study conducted by Adelman, for example, finds that of all starting community college students who earned at least 10 credits, only 47 percent attended only one institution (1999, table 20). To derive this figure, Adelman used the HS&B data set which collected student information primarily from the 1980s to early 1990s. He points out that more recent data suggest that the 1990s saw an increase in multi-institutional attendance, with particularly strong increases among those who attended at least three colleges. His point is that institutionally based retention or dropout data give a much more pessimistic picture of college completion than individual longitudinal data do.

Students use community colleges for a variety of reasons and obtaining a credential may not necessarily be their primary objective. Using student transcripts from the HS&B data set, Adelman writes, "nearly one out of six [of all students who entered a community college] never earns even a semester's worth of credits" (1998, p. 3). In other words, these students are classified as "experimenters" (Grubb 1999), and, according to Adelman (1998) including them in a universe with which to judge institutional performance can be misleading.

Even for students who attend a single institution and pass the stage of experimenting, dropping out, or "stopping out," may not be an indication of failure. Community college advocates point out that many students attend community college to acquire particular skills with no intention of completing a degree. Indeed from this point of view, a distribution of credit completion short of graduation may indeed be an indication of the flexibility of the institution rather than of failure.

For example, a student accumulates 36 credits but does not earn a credential. The student's transcript shows that half of the courses are in finite mathematics, electronics, computer programming, computer organization, and architecture. Few would disagree that a dominant tone of this record exists–this student is prepared to enter the labor market in the general area of computer technologies. Although this does not guarantee labor market success, it does mean that the student has derived something from the community college experience and this something is valuable to employers. Some support for this general argument that community college students can have success is provided by research that shows the accumulation of credits without a degree does have economic value, although earning the degree does also have an additional value (Grubb 1999).

Adelman (1998) composes a new system "completion rate," where students who have a classifiable cluster of coursework are added to those who obtained an associate's degree and certificate. Using this calculation, community colleges achieve nearly 90 percent "success." This is also equivalent to the bachelor's degree completion rate of four-year college students who entered directly from higher school and earned more than 60 credits (Adelman 1998).

At the same time, policymakers should be careful not to conclude that focusing on new and varied pathways through higher education renders the concept of completion as meaningless as an indication of success. After all wage analysis does show that many students who attend community college without earning a degree do not enjoy wage increases (even though on average such students do experience increased earnings). Our point is that changing pathways and uses of higher education by an increasingly diverse student population complicates assessment, and we suggest that the NAVE can make a crucial contribution to sorting out this issue.

Proposed Projects

(1) Mapping Emerging Pathways in Higher Education

We have emphasized throughout this paper the importance of understanding the new and varied pathways through higher education. This is a central issue for NAVE since a better understanding of these pathways will have a profound influence on the standard against which postsecondary vocational education is assessed. Therefore, a project that maps these pathways and increases our understanding of them will be a fundamental contribution of the NAVE. Indeed, as we shall see, the conceptualization and measurements developed by this project could also serve to strengthen other projects listed below.

Adelman's recent work (1999) traces the progress of students from high school into higher education using the HS&B data set and is the most ambitious attempt to understand the new pathways. Although this work is instructive from a methodological standpoint, its focus is almost exclusively on bachelor degree attainment. In addition, because it is based on the HS&B data set, it is useful primarily for illustrating the educational developments in the mid- to late 1980s. Trends in multi-institutional attendance appear to have accelerated in the 1990s. Moreover, his study does not examine the growth in noncredit offerings and customized training, which can be on either the credit or noncredit side.

A number of national data sets can be useful for this proposed project. For example, IPEDS completions file can be used to determine the number of degree and/or certificate conferred by a postsecondary institution. But the most important part of this analysis will be based on the longitudinal data sets, such as BPS and HS&B, that tracks the educational attainment as well as the employment status of students who attended a postsecondary institution during a 5- and 13-year period, respectively. Additional analyses using these data sets have been conducted on the educational attainment of racial and ethnic minorities, first-generation students, at-risk students, and students with disabilities. The latest waves of the BPS and NELS data sets (see Appendix A) would be particularly useful.

Thus the outcome of this project will be a mapping of the changing pathways through community college, especially looking at credit accumulation, degree attainment, multi-institutional attendance, and interruptions of education. When possible the use of noncredit education and training should be addressed. Initially this should basically be a descriptive study, but the project should work towards a categorization or typology of educational pathways. But once a categorization is developed, the project can also analyze the determinants of different pathways. Are there differences by race, income level, occupational area, or local economic conditions? This project would probably rely primarily on some combination of HS&B, NELS, and BPS.

(2) Economic Returns to Community College Education

In the past, available data have been used widely to evaluate the economic benefit to community college degrees. This research measures how much more individuals can earn if they complete community college degrees or certificates or if they complete a given amount of course work with or without degrees. Extensive work has also been done on measuring the benefits of particular occupational areas and on comparing the benefits of community college degrees to bachelor's degrees. The general consensus is that students who attain a subbaccalaureate degree earn more than those with a high school diploma do. Bachelor's degree holders earn more than those with community college degrees, but this is to be expected since community college degrees take less time to earn. Students do earn more as they accumulate credits even if they do not complete a degree, although the degree or certificate gives an additional boost to earnings. Finally, there is strong variation in earning depending on the field of study and those with community college education earn more if they find a job in the field for which they were prepared. Grubb (1999) has recently completed a review of this research.

Since this has been a widely researched area, a foundation has already been established on which to build NAVE research. We suggest that NAVE at least commission a review paper that will update the most recent reviews. A larger project that analyzes the data from the latest wave of NELS might be useful. Those data, however, will not be collected until 2000, and will probably not be available in time for the NAVE deadline.

We also suggest a reanalysis of past work based on the emerging pathways conceptualization. Using HS&B and NELS, the project would try to measure the earnings effects of interrupted and multi-institutional attendance. The conceptualizations developed in the above project would be useful for this project as well.

(3) Review of State Accountability Policy

Performance-based accountability systems are still relatively new in education, so there is little concrete evidence about what mix of incentives, assistance, and sanctions produces the best performance results. Accordingly, states are continually working to find the optimal balance between support and penalty. As we have argued, accountability is most straightforward for traditional-aged, degree-oriented students in higher education who attend only one institution. A more informative project would examine accountability measures in the context of nontraditional student pathways.

We suggest that the NAVE conduct a survey of state accountability measures for community colleges. This could be done for a sample of states including interviews with state officials and a representative sample of colleges within the state. The sample of states should be chosen so as to allow the examination of examples of different approaches to accountability. This could be done in the context of a follow-up to National Center for Research on Vocational Education (NCRVE) studies on accountability systems in vocational education (Rahn et al. 1992). The proposed project should summarize the performance measures and standards that have been implemented nationwide as of fall 1998 and highlight examples of states' measures and standards that are clearly and precisely defined for various outcome areas.

Once again we emphasize that this project should focus on the implications of the emerging pathways conceptualizations. Given what we are beginning to learn about these pathways, what are the implications of the various state accountability measures? Do they take into account these new patterns?

(4) Analysis Based on State Unemployment Insurance (UI) Data

Over the last 20 years, most analyses of the economic value of community college education has been measured using national longitudinal data bases such as HS&B and the National Longitudinal Study of the Class of 1972 (NLS-72). During the 1990s, several states, most notably California, Florida, North Carolina, Texas, Indiana, Illinois, Colorado, Alaska, and Washington have been conducted statewide studies using state UI wage record data to determine postcollege earnings.1 Using individual social security numbers, states can match students with their earnings, which are recorded for purposes associated with state unemployment insurance. Thus the SSI data offer unprecedented opportunities to use posteducation wage data for evaluation and accountability measures of individual institutions.

On the other hand, analysts using the UI/SSI data face several serious problems. The UI data do not include hours worked. They do not include people who are self-employed, work for the federal government, are unemployed, or have left the state. In most cases, individual students cannot be tracked due to confidentiality requirements, so data are usually available only in aggregate categories.

For example, California has been creating files on the total universe of students rather than just samples of students extracted from administrative records. In short, as students frequently move in and out of postsecondary education, it becomes more difficult to track them with less than a universal approach. Also when students transfer from a public to a private postsecondary institution or move out of state, then it becomes increasingly difficult track them and their educational and wage record in essence is dropped. As of this writing, only Alaska and Washington have arranged to retrieve employment earnings from neighboring states.

While most UI wage studies account for students who enlist in the military, are employed in public sector, or are incarcerated, few have addressed the need to account for those students who are self-employed. Other concerns related to the matching of a student's social security number to his/her employment and UI wage record center on confidentiality and privacy issues (Sanchez and Laanan 1998).

Thus the UI/SSI data would be most useful for a student taking a traditional path through postsecondary school—full-time schooling at one institutions, followed by full time work. Yet, as we have emphasized, a growing number of community college students attend part time, work while enrolled, attend sporadically, or attend multiple institutions before graduating. All of these trends complicate the use of UI/SSI data for accountability measures for individual schools.

We propose a project that would first update reviews of the use of SSI/UI data to measure the earnings effects of community college education. Recent reviews have been done by Grubb (1999) and Sanchez and Laanan (1998), but states are continuing to analyze their data so additional studies have been completed since those reviews have been conducted. While a general review will be useful, it should focus particularly on the use of the SSI/UI data for accountability purposes. Moreover, most of the studies included in these reviews were simple comparisons between the average earnings of different groups. Thus the next step of this project should be to reanalyze some of these data using more sophisticated multivariate statistical techniques.

This project should also include a discussion of the implications of the new pathways for the use of SSI/UI data for accountability. The unemployment insurance data may be particularly crude for taking into account these new developments, especially if the data are only made available in aggregate form (in different cells). Thus we need to understand whether a growing reliance on these data will distort our understanding of community college effectiveness.

(5) Noncredit and Customized Training

Customized training, which includes contract training, is one of the fastest growing community college activities. Community college educators and advocates argue that customized training and other noncredit activities represent important contributions to their local economies. Indeed, some states claim that they have more noncredit than credit students. Noncredit training and education is also an important contributor to the growing importance of nontraditional pathways through higher education. Although there are some examples of impressive programs, but we have no overall sense of the extent of these activities to say nothing of their effectiveness. Given the growing emphasis on these activities in the community college, no overall assessment of the effectiveness of community colleges will be complete without a better understanding of this phenomenon.

At present existing national data sets are probably too small for a full and definitive study of these types of activities. Most of our systematic knowledge about customized training comes from surveys by individuals or groups of researchers on one institution or a set of institutions. Some states do have data on categories of noncredit training, but we know of no systematic review of what states know about this. Last year, the American Association of Community Colleges (AACC) asked directors of state postsecondary education agencies for information on noncredit education. Responses from the surveys showed that at least a dozen states were able to provide some basic information on noncredit at community colleges. Thus these states do have relevant databases.

Also last year, a Postsecondary Education Quick Information Survey2 (PEQIS) on noncredit and customized training was planned by NCES. The survey, with the help of AACC, was piloted at several two-year and four-year institutions. This particular effort, however, was cancelled primarily because of the ubiquitous phrasing of the questions on the survey. The institutions also noted that the survey placed an undue burden on them, partly due to the length of the survey.

We suggest a project that will at least lay the foundation for better knowledge and understanding of customized and noncredit training. This would involve building on the work of the AACC to systematically survey of state directors to develop a catalogue of state data that could be used to study noncredit training. This project should also evaluate the problems with the PEQIS survey and try to develop an improved plan. While this project will probably not result in any definitive measures before the NAVE report is due, we do think that some preliminary review of state data will be possible. The product for this project would be a statement of the availability of such data and descriptive and comparative analysis of the data that are available. The report would present the nature and relative importance of noncredit education, a plan for improving the information available, and a discussion of accountability measures for noncredit and customized training.

(6) Responsiveness of Community Colleges to Local Labor Market Needs

Community colleges are increasingly seen as playing a vital role in economic and workforce development. But we have no easy way to assess their effectiveness in carrying out this function. How can we measure whether a community college's activities have contributed to economic growth in a particular area?

One possibility would be to analyze the supply response of vocational education institutions to rapidly growing occupations in the local area or state. We suggest that a project select a sample of regions or cities in which the shifting occupational distribution within those regions can be compared to trends in the occupational training and education provided by the local community colleges. Particular emphasis should be paid to the occupations for which community colleges have traditionally be an important source of trained labor (such as nursing and information technology). By tracking these changes over time, this project could discuss how responsive the community colleges have been to developments in their local labor markets. Occupational distributions could be measured using data from the CPS and Bureau of Economic Analysis, while the distribution of occupational training can developed from IPEDS. This project would be particularly interesting if initial findings from this quantitative study could be followed up by case studies of particular schools that were found either particularly unresponsive or responsive.


Footnotes

1 For a review of literature, see Grubb (1999) and Sanchez and Laanan (1998).

2As its name suggests, PEQIS surveys obtain information on emerging issues quickly and can be used to assess the feasibility of developing large-scale data collection efforts on a given topic.


[ Go to ED Home Page ]

This page last modified August 11, 1999 (glc).

 
Print this page Printable view Send this page Share this page
Last Modified: 10/24/2003