EVALUATION OF PROGRAMS
Evaluation Plan
National Assessment of Vocational Education (NAVE)

Program Management

Federal vocational legislation has for some time used funding and accountability provisions to achieve specific policy goals. Perkins III is no exception. Perhaps the most significant changes in the new legislation involve these program management tools. Understanding how the new provisions are implemented, and the consequences of them, will be an important issue for Congress and for the NAVE.

A. Accountability

Increased accountability is a major component of Perkins III. While states have been expected to gather information on student outcomes since 1990, few states have been able to consistently and systematically do so. The 1998 Act raises the requirements for state reporting of student outcome data, and the potential rewards and consequences for states that can and cannot do so.

Perkins III requires each state to develop a system of measurements and to establish expected levels of performance in four categories:

  1. Student attainment of academic, vocational, and technical skill proficiencies.

  2. Completion of a secondary or postsecondary degree or credential.

  3. Placement and retention in postsecondary education, advanced training, employment or the military.

  4. Participation and completion of programs that lead to nontraditional employment.

In addition, each state is required to report on the progress of special populations with respect to each of these categories.


Figure 10

Perkins III Accountability

Research Issues


  • Quality of Measures and Standards
  • Effects on Program Management
  • Best Practice
  • Comparability Across States
  • Relationship to WIA Accountability

Study Plans


  • Secondary vocational education—analyses of state and local reporting systems
  • Postsecondary vocational education—analyses of state and local reporting systems

The major changes in Perkins III accountability have less to do with the types of measures that are required than they do with how performance data, once collected, is to be used. Perkins III requires states to negotiate quantifiable levels of performance[14] with the Department of Education and to report yearly to the Secretary on progress made in meeting these standards. Moreover, the Secretary is to make this information available publicly and to compile state-by state comparisons. Finally, Perkins III raises the potential stakes associated with performance. Failure to meet state-level performance standards could eventually result in loss of Perkins funds. As a reward, however, section 503 of the WIA provides incentive grants to states that exceed performance levels under the Perkins Act, Adult Education and Family Literacy Act, and WIA Title I.

Given the significance of these provisions, NAVE intends to address two main issues regarding accountability:

Have states, school systems and colleges developed appropriate measures, standards, and data-gathering systems?

Studies undertaken in the early 1990s of initial efforts to implement Perkins II accountability requirements raised an important concern that remains relevant today. States were generally able to meet (or exceed) legislative requirements, but the quality of their systems was uneven. A more recent article suggests that "states have made dramatic progress toward meaningful accountability systems," perhaps spurred by the realization that accountability is not a passing fad.[15] Anecdotal information indicates that Perkins III accountability requirements have been taken much more seriously by federal and state officials than was the case under the previous Act. However, several critical questions need to be addressed:

  • Are data systems in place to produce reliable information about student performance?

  • Are local entities in a state using similar measures and assessing outcomes for the same population of students?

  • Have challenging, yet realistic, standards of performance been established?

  • Why have some states made greater progress in developing accountability systems for vocational education than others?

  • To what extent are states and local entities able to integrate WIA and Perkins performance accountability requirements?

How are Perkins performance report results used to manage and improve programs?

In addition to highlighting data quality concerns, the earlier accountability studies of Perkins III also questioned the utility of the data collected. Local educators reported seeing little value in the information they were asked to gather and report. They therefore failed to use the data as intended for program improvement, providing even less incentive for them to ensure high data quality. Given this history, it will be important to examine several questions about the impact of the new requirements:

  • Are states and communities using the data gathered to manage and improve their programs (i.e., are one-stop clients given access to this information)? Have accountability requirements resulted in any unintended effects?

  • Are the incentive payment provisions credible?

  • Can state reported performance data be aggregated to produce a national profile? Can it be used to compare state performance?

PLANNED STUDIES

NAVE will examine accountability practices at both secondary and postsecondary levels through:

  • Surveys of state directors.

  • Review of state reported performance data.

  • In-depth case studies of state and local accountability practices.

B. Funding Issues

It has been frequently observed that federal policy is a "blunt instrument" for change. However, nowhere is the impact of federal legislation more direct and immediate than the rules governing allocation of funds. The Congress that enacted Perkins III was guided by two major principles regarding funds (1) "drive" more money to local grantees, and (2) provide greater flexibility regarding the use of funds.


Figure 11

Perkins III Funding Issues

Research Issues


  • Changes in Local Grants as a Result of Legislative Changes
  • Effects of Eliminating Set-asides for Special Populations
  • Secondary/Postsecondary Split
  • Use of Perkins Funds at the State and Local Level

Study Plans


  • Secondary and Postsecondary State Director Surveys
  • Secondary and Postsecondary Case Studies

Toward these goals, the Act (1) eliminated a 10.5 percent set-aside for single parents and displaced homemakers, enabling Congress to direct that 85 percent (up from 75 percent) of funds should go to local programs; (2) created a 10 percent reserve fund which may be distributed outside of the intra-state formula; (3) eliminated the requirement for a state sex equity coordinator while mandating that $60-150,000 of state leadership funds be used for sex equity services; (4) eliminated rules regarding prioritizing local funds to serve programs with the highest concentration of special populations; and (5) changed the secondary school intra-state formula to simplify but not significantly alter funding allocations. Also relevant was the elimination of an 8 percent set-aside for school programs as Congress replaced the Job Training Partnership Act with WIA. NAVE will examine both the allocation and use of funds under the new Perkins provisions, including:

To what extent have the new provisions affected how funds are allocated?

  • How have changes in the intra-state formula, and reserve fund, altered the amount and distribution of funds at the local level?

  • Have states changed the allocation of funds between secondary and postsecondary education?

How has the use of funds changed, if at all?

  • What has been the effect of eliminating set-asides and other provisions for special populations?

  • How do states use "state leadership" funds?

  • What priorities are emphasized by local education agencies and postsecondary institutions in their use of Perkins funds? To what extent does the use of funds support priorities in Perkins III?

PLANNED STUDIES

Surveys of state directors (secondary and postsecondary)

In-depth case studies of state and local priorities.


14 States can renegotiate levels of performance prior to the third program year covered by the state plan. Timelines for the NAVE report will allow it to describe any changes in standards that may occur, but not the resulting outcomes.

15 Mikala Rahn and Patricia Holmes, "Accountability Systems: Performance Standards and Assessment," NCRVE Centerpoint (March 1999).

Postsecondary Occupational EducationPrevious Table of Contents NextOther Mandated Issues


 
Print this page Printable view Send this page Share this page
Last Modified: 09/16/2003