October 15, 1999
TO: State Directors of Vocational Education
State Directors of Community, Technical and Junior Colleges State Tech-Prep Coordinators
FROM: Patricia W. McNeil
SUBJECT: Accountability Systems Development for the Carl D. Perkins
Vocational and Technical Education Act of 1998
Among the most important provisions of the Carl D. Perkins Vocational and Technical Education Act of 1998 (Perkins III) are those creating a new State performance accountability system to assess the impact of vocational and technical education on students (section 113). Since the Act was passed, the Department of Education has worked with the State vocational education agencies, the Department of Labor, and other representatives of the vocational education community to design a system that will produce valid, reliable, and useful information about students' participation in vocational education and their post–education experiences.
The purpose of this memorandum is to provide answers to questions about the accountability system that arose from the Department's consultation with the vocational education community. Along with the Perkins State Plan Guide and the OVAE Core Indicator Framework for State Performance Accountability Systems, this memorandum provides initial guidance you need to establish your State's accountability system. The draft Core Indicator Framework is attached.
The State Plan Guide is available at http://www.ed.gov/about/offices/list/ovae/pi/memorandums.html .
Is the State required to use the performance indicators in section 113 (a)(2)(A) to annually evaluate the effectiveness of its vocational programs?
States are required to use the core indicators described in section 113(a)(2)(A) to evaluate program effectiveness (section 113(c)(1)(A), section 122(c)(6), and sections 123(a) and (b)). Being based on the core indicators, State evaluations must now focus on the impact of programs on students.
In what format will States be required to report the results of their evaluations?
The Department will issue an OMB–approved annual performance report form to collect this information. Instructions for reporting accountability data will be established in the Annual Performance Report (Education Department General Administrative Regulations 34 CFR 80.40), which will be available for public comment in September 1999. That document will include additional instructions. This memorandum does not impose information collection requirements that are subject to the Paperwork Reduction Act of 1995.
Which students should be included in the accountability system?
The accountability system should include the secondary and postsecondary students participating in public vocational and technical education in the State. It should also include students participating in private vocational and technical education at agencies or institutions that receive Perkins III funds in the State. The purpose of the accountability system is to assess the effective of the State in achieving statewide progress in vocational and technical education (section 113(a)).
The State Plan Guide requested that States report on vocational education "concentrators." Since the Guide was published, the Department and the States have extensively discussed how to define a "concentrator." Through these consultations, the States and the Department developed the concept of a "threshold" to identify "concentrators," who are students who have a sufficiently high level of participation in vocational and technical education to be in the accountability system. A State should establish a threshold level consistent with the definition of "vocational and technical education" found in section 3(29) of Perkins III, which means organized educational activities that offer "a sequence of courses (or equivalent instructional units) that provides an individual with the academic and technical knowledge and skills the individuals need to prepare for further education and careers." Therefore, the Department recommends that States establish a threshold that includes only "concentrators" or students who have participated in this type of sequence of courses or instructional units.
Can samples be used to collect information for the accountability system?
A primary purpose of the new accountability system is to assess the effectiveness of the State in achieving statewide progress in vocational and technical education (section 133(a)) and, through the establishment of performance levels, ensure States continually improve the performance of vocational technical education students (section 113(b)(3)(B)). Evaluations are linked to the core indicators (section 113(c)(1)). The emphasis on statewide performance, student outcomes, and continuous improvement mean that States will need detailed information about local performance on indicators to target improvement strategies. In our discussions with the States, States have recognized that a preferred approach to collecting this information is to include all students, so that student achievement can be linked to instructional approaches as the basis for continuous program improvement.
The Department recognizes that a number of States are now using sample studies to gather information about students, especially to track the employment and further education of students after they leave a vocational education program. To help States make the transition to the new accountability system, the Department will consider data collected through sampling for the first two years of the accountability plan (section 1113(b)(3)(A)(iii)).
The State agency is responsible for ensuring that data reported by its local education agencies and institutions, and the data the State reports to the Secretary, are complete, accurate, and reliable (section 122(c)(20)). Therefore, the use of sampling should adhere to generally accepted principles for statistical data collection and analysis. States wishing the Department to set adjusted levels for performance based on samples should submit a statewide sampling and analysis plan as part of the State plan for accountability (or an amendment to the State plan). The sampling plan should address methodological issues such as how the sampling will result in:
A true random sample of the students in vocational and technical education.
A sample size that is sufficiently large to ensure that findings will be statistically significant for each local education agency and each postsecondary institution, and for each special populations group.
A response rate that will produce unbiased results (generally not less than a 75% return rate), through aggressive follow–up to contact individuals in the sample.
States should contact the Division of Vocational and Technical Education to indicate their interest in using sampling.
Is Tech–Prep Education included in the accountability system?
States should include students participating in Tech–Prep Education in the determination and reporting of the core indicators and the State adjusted levels of performance. Section 113(a) requires a State to establish a State performance accountability system to assess the effectiveness of the State in achieving statewide progress in vocational and technical education. A "Tech–Prep Program" is vocational education by definition (sections 3(26)(B) and (C), and 202(a)(3)). States should report for Tech–Prep students by each core indicator separately from all vocational education students. Please see the State Plan Guide for additional information.
Will States set and report levels in numbers or percentages?
States may set target performance levels in terms of either percentages or numbers. The State should report its actual performance (on the annual performance report) in the same form (percentage or number) as its negotiated performance levels are set. When the indicator is expressed by a percentage, the State should also report the raw numbers (numerator and denominator) to make clear the universe of students included in the calculation and the size of the student population meeting the standard.
What are the requirements for reporting on special population groups?
Each eligible agency must report to the Secretary annually on the progress of members of special populations in achieving the State adjusted levels of performance (section 113(c)(2)). The purpose of measuring and reporting the progress of special populations is to ensure that the "special populations meet or exceed State adjusted levels of performance" (section 122(c)(8)(C)). To determine whether they are meeting this goal and assess the impact of instructional and program strategies on special populations, States should track the progress of each special population group separately. Therefore, States should report the levels for each core indicator, for each special population group, not for all special populations groups in aggregate. States that encounter problems establishing systems that will allow them to measure results for specific special populations groups should consult their Division of Vocational and Technical Education representative for assistance.
For the related State planning requirements, see the State Plan Guide and section 122(c)(12) of Perkins III.
What academic content areas should the State include in its analysis of secondary student attainment of State academic standards?
For the first core indicator of performance, States must measure "student attainment of challenging State established academic skill proficiencies" (section 113(2)(A)(i)). Perkins III also requires that States ensure that students who participate in vocational and technical education programs are taught to the same challenging academic proficiencies as are taught to all other students (section 122(c)(4)(B)). To keep the vocational and technical education standards and accountability system consistent with what the State uses to assess academic achievement for all secondary students, States should include all academic content areas for which there are State–established assessments.
What if the State academic standards or assessments change over time?
Many States are still developing or just implementing State academic assessments. If the State assessments or standards change during the period of the Perkins Act State Plan, students participating in vocational and technical education should continue to be assessed the same as all other students. The Department will work with States to set adjusted performance levels that take into account changes in the academic assessment systems in the State. For example, the State adjusted level of performance for vocational student achievement could be expressed in relation to performance level of all students in the State (i.e., as a ratio of scores), rather than setting a level in terms of a test score.
If State academic assessments change, the State may need to amend its plan to be consistent with the new assessment strategy. Plan revisions may be submitted annually, as deemed necessary by the State (section 122(a)(2)(A)). After the second year of the five–year State plan, the State is required to review Perkins activities and submit any necessary revisions (section 122(a)(2)(B)). Finally, the State may request that its State adjusted levels of performance be revised in response to changes that affect the extent to which the adjusted levels of performance promote continuous improvement (section 113 (b)(3)(A)(vii)).
Must States measure "retention in postsecondary education" when following the progress of students leaving secondary vocational education programs?
Most States have observed that there is no practicable way to measure retention in education. To measure the retention in postsecondary education of postsecondary students (for section 113(b)(2)(iii)) States would have to track postsecondary student completion of their course of study. For secondary students, States must measure placement in postsecondary education, advanced training, military service, and employment; and retention in employment/military service. The draft Core Indicator Framework provides further guidance on how to measure these outcomes. States that have the capacity to assess secondary student retention in postsecondary education may include this as a State additional indicator of performance (section 113(b)(2)(B)).
What are the options for determining non–traditional occupations?
To determine State performance on the core indicator, "Student participation in and completion of vocational and technical education programs that lead to nontraditional training and employment," States would have to identify non–traditional occupations and the corresponding vocational education programs. Nontraditional training and employment is defined in Perkins III as "occupations or fields of work for which individuals from one gender comprise less than 25 percent of the individuals employed in each such occupation or field of work" (section 3(17)). Because our nation's labor force is increasingly mobile and the labor market internationally competitive, the Department encourages States to use national employment statistics to identify non–traditional occupations. However, States may use State data on employment to identify nontraditional occupations if it improves targeting of educational improvement activities for nontraditional employment and service to students enrolled in those programs.
Once non–traditional occupations are identified based on employment data, States would have to determine which of their education programs prepare students for these occupations by cross–walking the occupations to education programs. States should do this analysis at a level of detail that best reveals the male–to–female employment ratios in different occupations. The draft Core Indicator Framework provides additional guidance on identifying the education programs that prepare students to enter non–traditional occupations.
States may get additional information on employment statistics and non–traditional occupations from the National Occupational Information Coordinating Council, State Occupational Information Coordinating Councils, or the Bureau of Labor Statistics of the U.S. Department of Labor (BLS). National data on gender of individuals employed in various occupations is derived from the Current Population Survey, which is conducted by the Census Bureau for BLS. The most recent data, the national household average for 1998, was published in the January 1999 edition of the BLS publication Employment and Earnings. A chart that provides a detailed breakdown of various occupations by gender, race, and Hispanic origin in 1998 is available at the BLS website at http://stats.bls.gov/home.htm.
May the State include only high skill, high wage occupations in its data for the indicator of "non–traditional occupations"?
States should include all identified nontraditional occupations in their data for this indicator. The definition of "nontraditional training and employment" (section 3(17)) comprises all occupations for which individuals from one gender comprise less than 25 percent of the individuals employed in each such occupation or field of work. These occupations must "include careers in computer science, technology and other emerging high skill occupations," but there is no statutory basis for excluding from the indicator any occupations that meet the definition.
A State that has identified high skill, high wage occupations may focus its activities to reduce gender bias in those education programs that prepare students for high skill occupations. A State may direct its program improvement activities to a subset of its non–traditional education programs, rather than trying to improve the gender equity in the enrollment of every program that prepares students for non–traditional occupations. A State may wish to have its proposed improvement activities be a factor in establishing the adjusted levels of performance for the non–traditional occupations indicator.