Skip main navigation.
 U.S. Department of Education: Promoting Educational Excellence for all Americans - Link to ED.gov Home Page
ESRA: Research, Development and Dissemination - FY 2005


CFDA Number: 84.305 - Education Research

Program Goal: Transform education into an evidence-based field.

Objective 1 of 2: Raise the quality of research funded or conducted by the Department.
Indicator 1.1 of 4: The percentage of newly funded research proposals funded by IES that receive an average panel review score of excellent.
 
Measure 1.1.1 of 1: The percentage of new research proposals funded by the Department's National Center for Education Research that receive an average score of excellent or higher from an independent review panel of qualified scientists.
Year Actual Performance Performance Targets
2003 88  
2004 97  
2005 100 100

Source: U. S. Department of Education, National Center for Education Research, review panel. The average panel review score for each newly funded research proposal will be calculated.

Frequency: Annually.

Next Data Available: March 2006

Data Validated By: No Formal Verification.
Evaluations are only as good as the qualifications of the peer review panel. Inclusion of only senior scientists and leading researchers in their fields helps assure the quality of these data.
 
 
Indicator 1.2 of 4: Whether or not the modal rating (most common judgment) of an independent review panel of qualified scientists is that new research and evaluation publications by IES are of high quality.
 
Measure 1.2.1 of 1: Whether or not the modal rating (most common judgment) of an independent review panel of qualified scientists is that new research and evaluation publications by IES are of high quality. (Data tables will indicate ''2'' for ''Yes'', ''1'' for ''No'',''0'' for ''No new publications/evaluations issued'').
Year Actual Performance Performance Targets
2003
0
 
2004
0
 
2005
 
2
2006
 
2

Source: U. S. Department of Education, National Center for Education Research, review panel. Staff from the National Center for Education Research selects a random sample of new research and evaluation publications. Publications are distributed to senior scientists in the field for review.

Frequency: Annually.

Next Data Available: March 2006

Data Validated By: No Formal Verification.
Evaluations are only as good as the qualifications of the external review panel. Inclusion of only eminent senior scientists who are distinguished professors in their institutions, editors of premier research journals, and leading researchers in education and special education assures the quality of the data.
 
Explanation: No new research/evaluation publications were issued in 2003 or 2004. The indicator was changed from focusing on percentages of publications deemed to be of high quality to focusing on whether or not the modal response (most common judgment) of the review panel is that new IES publications are of high quality. This is because the number of IES research and evaluation publications is currently quite small. With very small numbers, percentages are not very meaningful, because changes in one or two reports can translate into large changes in percentages. In this case, focusing on whether the most common rating is that publications are of high quality is a more meaningful indication of the overall judgment of the review panel.
 
Indicator 1.3 of 4: Of new research and evaluation projects funded by the IES that address causal questions, the percentage of projects that employ randomized experimental designs.
 
Measure 1.3.1 of 1: Of new research and evaluation projects funded by the IES that address causal questions, the percentage of projects that employ randomized experimental designs.
Year Actual Performance Performance Targets
2001
32
32
2002
100
75
2003
97
75
2004
90
75
2005
 
75

Source: U. S. Department of Education, National Center for Education Research, review panel. National Center for Education Research researchers evaluate all newly funded research and evaluation proposals to identify projects that address causal questions and of those projects, those that use randomized experimental designs to answer those questions. Data will be collected annually. The 75% target for 2002-2005 recognizes that some high-quality research addressing causal questions will not be able to employ randomized experimental designs.

Frequency: Annually.

Next Data Available: March 2006

Data Validated By: No Formal Verification.
Evaluations are only as good as the qualifications of the proposal reviewers. Having qualified researchers conduct the reviews, as well as a check of inter-rater agreement in which the two IES researchers independently evaluate a subset of proposals (with minimum inter-rater agreement of 90%), minimizes threats to the validity and reliability of data. Presence of a causal question is defined as instances in which the investigation is designed to examine the effects of one variable on a second variable. A causal relation might be expressed as one variable influencing, affecting, or changing another variable. A randomized experimental design is defined as instances in which there are (a) an experimental (treatment) group and one or more comparison groups, and (b) random assignment of participants to treatment and comparison groups, or random assignment of groups (e.g., classrooms or schools) to treatment and comparison conditions. If a proposal includes a design in which two or more groups of participants are compared, but the PI does not explicitly indicate that random assignment procedures will be used, the proposal is recorded as not using a randomized experimental design.
 
 
Indicator 1.4 of 4: Of new research and evaluation publications funded by IES that address causal questions, the percentage of publications that employ randomized experimental designs.
 
Measure 1.4.1 of 1: Of new research and evaluation publications funded by IES that address causal questions, the percentage of publications that employ randomized experimental designs.
Year Actual Performance Performance Targets
2002
100
75
2003
0
75
2004
0
75
2005
 
75

Source: U. S. Department of Education, National Center for Education Research, review panel. National Center for Education Research staff evaluate all newly funded research and evaluation publications by the Center to identify projects that address causal questions and of those projects, those that use randomized experimental designs to answer those questions. Data will be collected annually. The 75% target recognizes that some high quality studies will not be able to employ randomized experimental designs.

Frequency: Annually.

Next Data Available: March 2006

Data Validated By: No Formal Verification.
Evaluations are only as good as the qualifications of the proposal reviewers. Having qualified researchers conduct the reviews, as well as a check of inter-rater agreement in which the two IES researchers independently evaluate a subset of publications (with minimum inter-rater agreement of 90%), minimizes threats to the validity and reliability of data. Presence of a causal question is defined as instances in which the investigation is designed to examine the effects of one variable on a second variable. A causal relation might be expressed as one variable influencing, affecting, or changing another variable. A randomized experimental design is defined as instances in which there are (a) an experimental (treatment) group and one or more comparison groups, and (b) random assignment of participants to treatment and comparison groups, or random assignment of groups (e.g., classrooms or schools) to treatment and comparison conditions. If a publication includes a design in which two or more groups of participants are compared, but does not explicitly indicate that random assignment procedures will be used, the publication is recorded as not using a randomized experimental design.
 
Explanation: No new research/evaluation publications were issued in 2003 or 2004.
 
Objective 2 of 2: Increase the relevance of our research in order to meet the needs of our customers.
Indicator 2.1 of 3: The percentage of new research projects funded by IES that are deemed to be of high relevance to education practice as determined by an independent review panel of qualified practitioners.
 
Measure 2.1.1 of 1: The percentage of new research projects funded by IES that are deemed to be of high relevance to education practice as determined by an independent review panel of qualified practitioners.
Year Actual Performance Performance Targets
2001
21
 
2002
25
25
2003
60
37
2004
50
50
2005
 
65
2006
 
75

Source: U. S. Department of Education, National Center for Education Research, external review panel. An external panel of qualified practitioners will evaluate the relevance of a random sample of newly funded research proposals. Data will be collected annually. The final target of 75% recognizes that some important research may not seem immediately relevant, but will make important contributions over the long term.

Frequency: Annually.

Next Data Available: March 2006

Data Validated By: No Formal Verification.
Evaluations are only as good as the qualifications of the external review panel. Inclusion of only experienced practitioners and administrators in education and special education assures the quality of the data.
 
Explanation: We met our FY 2004 target of 50.
 
Indicator 2.2 of 3: The number of annual hits on the What Works Clearinghouse Web site.
 
Measure 2.2.1 of 1: The number of annual hits on the What Works Clearinghouse Web site.
Year Actual Performance Performance Targets
2003
1,522,922
1,000,000
2004
4,249,668
2,000,000
2005
4,734,767
4,500,000
2006
 
5,000,000

Source: U. S. Department of Education, National Center for Education Research, What Works Clearinghouse, program report.

Frequency: Annually.

Next Data Available: October 2005

Data Validated By: No Formal Verification.
A Web-based program will automatically count the hits on this Web site.
 
Explanation: The FY 2005 target was exceeded. FY05 actual hits through 7/27/05 was 4,734,767 so we exceeded our target of 4.5 million with over two months remaining in FY05. Final figures should be available in early October.
 
Indicator 2.3 of 3: The percentage of WWC Web site users surveyed randomly who responded to the following statement, ''Evidence provided on the WWC Web site is useful in making decisions about education programs and practices,'' by checking ''agree'' or ''strongly agree.''
 
Measure 2.3.1 of 1: The percentage of WWC Web site users surveyed randomly who responded to the following statement, ''Evidence provided on the WWC Web site is useful in making decisions about education programs and practices,'' by checking ''agree'' or ''strongly agree.''
Year Actual Performance Performance Targets
2005
67
30

Source: U. S. Department of Education, National Center for Education Research, What Works Clearinghouse, program report. Data collected in 2005 will be the baseline data. Subsequent targets will be adjusted after we have the baseline data. There were no available data in 2003 or 2004.

Frequency: Annually.

Next Data Available: October 2005

Data Validated By: No Formal Verification.
 
Explanation: The FY 2005 target was exceeded based upon partial data. Through August 4, with about 2 months remaining in FY 2005, we excceeded our target levels by more than double.
 

Return to table of contents