A r c h i v e d  I n f o r m a t i o n

FY 1999 Annual Plan - Volume 1. Objective Performance Plans and Data Quality - February 27, 1998

Quality of Performance Data: How Data Will Be Verified and Validated

Exhibit 4
Strategies for Ensuring High Quality Information for Strategic Plan and Program Performance Indicators
  • Performance indicator standards
    • Timeliness. Ensure performance information is collected on a regular and timely basis (e.g., a minimum of two data collections per indicator over the Strategic Plan period; quick turnaround customer surveys).
    • Validity. Ensure performance indicators are valid (e.g., measures align with research-based findings on effective practices). Conduct state-of-the-art reviews of measures by objective and major program area.
    • Reliability. Ensure performance indicators are reliable (e.g., measures meet acceptable sample size criteria for confidence levels, samples are appropriately representative; definitions are consistent over time.)
    • Disaggregation. Ensure performance indicators appropriate disaggregate information down to key operational units (e.g., states for state grant programs).
  • Employee training in performance measurement and use
    • Materials and tools. Develop guides and training programs around performance management and measurement
    • Training:
      • Tailored training to particular offices needs and offer ongoing performance measurement assistance.
      • Develop certificated courses in performance measurement and utilization.
      • Evaluate employee capabilities in different offices to manage information for performance.
      • Offer on-line training
  • Monitor data quality
    • Internal monitoring and reviews. Develop an indicator ranking system that assesses indicator quality by objective and major program area.
    • Staff performance agreements. Require program managers, as part of their performance agreement, to validate the quality of their program performance information.
    • Evaluation of data quality.
      • Ensure independence and objectivity of performance information by relying on program evaluations to collect and process performance reports and to conduct independent evaluations of performance for major programs.
      • Conduct indicator quality evaluations to assess the accuracy of key indicator information.
    • Inspector General. Support Inspector General reviews of data quality (see exhibit 5).
  • Accountability for data quality
    • Attest to data quality. "ED program managers assert that the data used for their program's performance measurement are reliable and valid or will have plans for improvement." (Objective 4.7: Indicator 31)
    • External validation. Evaluate program managers assertions through IG audits and program evaluations.

Effectively reporting the performance of the Education Department in achieving its Strategic Plan goals and objectives requires developing some new data systems and fixing old ones. The new systems will seek to redirect data collections toward gathering performance information on the accomplishment of Department-wide and program goals. Existing data systems need to be strengthened to ensure the Department receives high quality and timely data on its programs and their effects.

The Department is undertaking a comprehensive set of data improvement activities built around the following two strategies.

Actions being taken to achieve these strategies are described below followed by highlights of the strategies being taken to improve elementary/secondary and postsecondary data quality.

Strengthening data quality

The quality of the Department's performance measures can be no better than the quality of the data from which they are generated. Inadequate attention to data quality produces inaccurate information and misleading results. The lack among many program staff of formal training in information processing, evaluation and reporting is a further impediment to obtaining high quality information. .

To ensure the quality of performance indicator information the Department is proceeding forward on a four-part improvement strategy (Exhibit 4).

During 1998 and 1999, the Department will be strengthening its two major performance indicator systems, one for the elementary and secondary system and a second for student aid. Collectively, these systems account for about three-quarters of Departmental funds.

Developing an integrated data system for elementary and secondary state grant programs

A key component of the Department's information improvement strategy is creating integrated federal/state indicator systems for the elementary and secondary state grant programs. This redesign will cover all federal funds disbursed through the state grant process. These include programs to support challenging standards, at-risk students, professional development, education technology, safe and drug-free schools, and general improvement support.

A joint redesign project with the Council of Chief State School Officers will begin in spring 1998. Initial tasks will include agreement on a set of data standards, key performance indicators and pilot testing of systems beginning in the fall of school year 1998-99. Our target is to begin working with a least six exemplary states on developing aligned, exemplary integrated indicator systems.

Key steps to developing the redesigned system include:

Exhibit 5
Inspector General Support for
Data Verification and Validation

The Office of Inspector General (OIG) has participated in an advisory capacity in the development of the Department's Strategic Plan and Annual Plan and will continue to provide this service to Department managers.

The OIG is currently performing the first in a series of audits covering the Department's implementation of the Government Performance and Results Act. The objectives of the first audit are; (1) to assess the Department's process for institutionalizing the results-oriented management envisioned by the Act; and (2) to assess the development of the system for the accurate and timely collection and reporting of performance data.

For the Strategic Plan, the OIG recommended that Department program managers assert that the data used for their program's performance measurement are reliable and if not reliable, detail plans for improving the data or finding alternative sources. The Department agreed with the OIG's recommendation and included it as a performance indicator in the Strategic Plan. The OIG plans to perform a series of audits on select performance measurement data to assess the reliability of that data. The OIG also plans to assess how the Department is using the performance data to improve programs.

Improving postsecondary data quality

Validity and accuracy of postsecondary performance measures. Data used to measure progress toward achievement of the performance indicators come from several sources including program data, surveys conducted by the National Center for Education Statistics (NCES), and evaluation studies. Steps being taken in 1998 and 1999 to strengthen the quality of these data include:

Accuracy and efficiency of program data systems. In FY 1999 the Department of Education will provide nearly $51 billion in federal student aid funds. To properly distribute and account for these funds, the Department of Education needs to process and store data from over: 8 million student aid applications; 93 million individual student loans with a value of more than $150 billion; 6,000 postsecondary institutions; 4,800 lenders; and 36 state guarantee agencies. Ensuring the accurate and efficient collection of these data is a key component in the successful delivery of the student aid programs and achievement of Goal 3 in the Department's Strategic Plan to, "ensure access to postsecondary education and lifelong learning."

The student aid delivery system has suffered from data quality problems which are sufficiently severe to cause the Department to fail to receive an unqualified audit opinion. Steps being taken to improve the efficiency and quality of the student aid delivery system include:

Ensuring the quality of performance information on internal management systems

The Department must have solid information on the performance of its internal management systems if it is to achieve its Goal 4 objective of a high performance organization. This means obtaining adequate coverage of internal system performance and ensuring the reliability of such data.

The Department plans on extending its independent evaluations, now used mainly for program evaluations, to management evaluations. A priority area will be evaluation of the quality of performance data, including data for customer surveys, performance contracting and employee performance ratings and awards. Also, informational technology systems are the bulwark of productivity growth and evaluations are planned to examine the quality of data on the performance of key information systems and of employee use of these systems.



Hatry, Harry and Kopczynski, Mary. Guide to Program Outcome Measurement for the U.S. Department of Education. Washington, DC: U.S. Department of Education's Planning and Evaluation Service, 1997.


[Relationship of program ...] [Table of Contents] [Program Performance Plans]