Data Quality and Timeliness | Data Improvement Strategies
The work of the National Center for Education Statistics (NCES) has been guided by written standards since 1988. As part of the congressionally mandated governmentwide Information Quality Guidelines activity of 2002, NCES standards underwent a substantial revision. NCES requires that all of its products meet the specifications identified in the 2002 Statistical Standards. The Statistical Standards are built on the basic tenets of quality, utility, and integrity. They cover the range of activities for which NCES has been given responsibility-from research and procurement plans, sampling, data collection, and processing to evaluation of the data, protecting confidentiality, analysis, report writing, data set preparation, and dissemination. NCES reports and data products must also pass an extensive technical quality review, including both internal review and a "blind" external review conducted by the Office of the Institute of Education Sciences Deputy Director for Science.
NCES continues to monitor the implementation of the statistical standards and center-sponsored data collections by documenting the use of incentives, response rates, timeliness, and utility of all data collections. These efforts contribute to the ongoing improvement of NCES data. By documenting the interaction between the use of incentives and the resulting response rates, NCES staff can make more effective decisions concerning where incentives are most useful in improving response rates and what types and levels of incentives are most effective with different types of respondents. Monitoring the use of incentives provides a basis for promulgating shared best practices across data collections. Finally, the data collected on timeliness serve as a baseline for newly developed performance standards intended to decrease the time period from the end of data collection to the release of the first report.
NCES is also charged with the responsibility of working with other components of the Department and with state and local educational institutions to improve the quality of education data. To this end, NCES conducted research that provided the basis for the Department's selection of the average freshman graduation rate, and anticipates publishing that report and related data this fall. This rate is the most reliable among a number of possible alternatives and will be used by the Department on an interim basis until the data required for individual states to compute a true cohort graduation rate are available. Related to this, NCES is responsible for a grant program that provides funding to states for the development of high-quality longitudinal student information systems that are needed to compute a true cohort graduation rate.
Through its Cooperative Systems, NCES encourages improved quality in the data collected at the elementary/secondary and postsecondary levels. At the elementary/secondary level, NCES recently released a Cooperative System Guide to Building a Culture of Data Quality, aimed at schools and school districts. At the postsecondary level, NCES has redesigned the Integrated Postsecondary Education Data System from a paper system to an online data collection. The combination of built-in edits and the shortened time to release has served to improve the quality of these data, while at the same time increasing their utility.
Federal and state officials are in agreement that all states must have valid and reliable accountability systems. Through a number of activities, the Office of Vocational and Adult Education (OVAE) and states have significantly improved the quality of state adult education performance data over the last several years as states have implemented the National Reporting System for adult education. Our joint efforts have paid dividends: state adult education programs' performance is now measured on the basis of participant learning gains that are determined by standardized assessments.
OVAE has enhanced states' capacity to provide high-quality assessment data by developing state data quality standards that identify the policies, processes, and materials that states and local programs should have in place to collect valid and reliable data. OVAE uses these standards when conducting onsite monitoring of states. If a state does not meet one or more of the standards, OVAE identifies the deficiencies in the monitoring report and requires the state to address them through a corrective action plan.
OVAE also provides ongoing training and technical assistance to states to promote compliance with the NRS requirements, the collection of valid and reliable data, and the use of data to improve program performance. OVAE also provides guidebooks and training to state teams, as well as materials states can use in training local providers on the following topics:
- Developing an individual student record system that can be used for accountability and program improvement purposes.
- Setting local program performance standards, monitoring local program performance, and using performance data to help local programs improve student outcomes.
- Using NRS data for program management and improvement.
- Data collection guidelines and procedures.
OVAE also offers online tutorials on these topics for local program staff.
To improve the quality of outcome data, OVAE has promoted the use of unemployment insurance wage records to identify the employment outcomes of adult education participants, consistent with the requirements of the Family Educational Rights and Privacy Act. Unemployment insurance wage records are a more accurate and less expensive way of identifying employment outcomes than follow-up student surveys. Nearly half of the states are now using state unemployment insurance wage records, and more are planning to begin using them in the next year.
For the past eight years, the Office of Special Education Programs (OSEP) has implemented a data-dependent accountability system (Continuous Improvement and Focused Monitoring System-CIFMS) that has focused on state performance on a number of performance measures and regulation-based compliance requirements. In support of CIFMS, the office has provided the following ongoing technical assistance and data reviews to support states' efforts to provide valid, reliable and accurate data related to the implementation of the Individuals with Disabilities Education Act:
- Conducting onsite verification visits to examine state systems for the collection of valid, reliable, and accurate data and providing follow-up technical assistance as needed.
- Providing individual state-tailored technical assistance through Westat, OSEP's data collection contractor.
- Providing ongoing technical assistance and support to states on sampling and the development of measurable and rigorous targets.
- Creating the General Supervision Enhancement Grant program, using Part D (National Activities) of the act's technical assistance funds to provide funds to states for the development of outcome indicators and measurement systems. Projects have included such activities as creating web-based data systems, creating individual student identifier record systems, and creating new systems to allow data from different agencies to "talk" to each other.
Section 616 of the reauthorized Individuals with Disabilities Education Act requires state educational agencies and lead agencies for Part C (Infants and Toddlers Program) to develop State Performance Plans with indicators established by the Department and targets set by the states, and to use those targets and indicators in reporting annually to the Secretary on the performance of the state. States must also use those targets and indicators to report annually to the public on the performance of each local educational agency and local early intervention service provider in the state.
The Department recognized that many state and local educational agencies lacked methods to collect and analyze outcome indicator data and, therefore, the capacity to collect sufficient data to determine the impact of early intervention and special education services. Section 616(i) of the reauthorized Individuals with Disabilities Education Act created a new funding authority and required the Secretary to provide technical assistance, where needed, to improve the capacity of states to meet the data collection requirements for all children, including infants and toddlers with disabilities.
The General Supervision Enhancement Grant Program, funded under Special Education Technical Assistance on State Data Collection at $4.5 million, has as its focus developing or enhancing state outcome indicators and methods to collect and analyze outcome indicator data.
Project proposals are expected to support states in their efforts to ensure that required data submissions are valid, reliable, and accurate. OSEP expects to make grant awards in January 2006.
Improving Data Systems-
Office of Safe and Drug-Free
The Department's Office of Safe and Drug-Free Schools is improving state data systems and linking those improvement activities to the Performance-Based Data Management Initiative. The No Child Left Behind Act requires that each state collect certain school crime and safety data elements and report the data collected to the public. Under the statute, states must create a system for collecting and disseminating information for several data elements, including truancy and the incidence of violence and drug-related offenses leading to suspension and expulsion.
The Grants to States to Improve Management of Drug and Violence Prevention Program provides support to states to explore strategies that will address the challenges they face in collecting and using data, including the following:
- Lack of standardized collection instruments and definitions both within and across states.
- Lack of expertise related to collecting data about youth drug use and violence.
- Lack of time and other resources to support high-quality data collection and analysis in these areas.
Improved data collection systems that result from this program will allow state, district, and school administrators to develop, expand, and/or enhance the capacity of state and local educational agencies to collect, analyze, and use data to improve the quality of drug and violence prevention programs. In addition, these systems will be able to identify the needs of students and assess progress in addressing these important problems.
Project proposals are expected to adhere to the following requirements:
- Be consistent with the state's Performance-Based Data Management Initiative strategy and produce data that can be transmitted to the Department via its Education Data Exchange Network.
- Include validation and verification activities at the state and substate recipient levels designed to ensure the accuracy of data collected and reported.
The first projects funded under this program were awarded in September 2004. Additional projects were awarded funds in September 2005.