U.S. Department of Education: Promoting Educational Excellence for all Americans

A r c h i v e d  I n f o r m a t i o n

ESRA: Research, Development and Dissemination - 2005

CFDA Number: 84.305 - Education Research


Program Goal: Transform education into an evidence-based field.
Objective 8.1 of 2: Raise the quality of research funded or conducted by the Department.
Indicator 8.1.1 of 4: The percentage of newly funded research proposals funded by IES that receive an average panel review score of excellent.
Targets and Performance Data Assessment of Progress Sources and Data Quality
The percentage of new research proposals funded by the Department's National Center for Education Research that receive an average score of excellent or higher from an independent review panel of qualified scientists.
Year Actual Performance Performance Targets
2003 88  
2004 97  
2005   100


 
Additional Source Information: The average panel review score for each newly funded IES research proposal will be calculated.

Frequency: Annually.
Collection Period: 2004 - 2005
Data Available: March 2006
Validated By: No Formal Verification.
Evaluations are only as good as the qualifications of the peer review panel. Inclusion of only senior scientists leading researchers in their fields assures the quality of the data.

 
Indicator 8.1.2 of 4: Whether or not the modal rating (most common judgment) of an independent review panel of qualified scientists is that new research and evaluation publications by IES are of high quality.
Targets and Performance Data Assessment of Progress Sources and Data Quality
Whether or not the modal rating (most common judgment) of an independent review panel of qualified scientists is that new research and evaluation publications by IES are of high quality. (Data tables will indicate “2” for “Yes”, “1” for “No”,''0'' for ''No new publications/evaluations issued'').
Year Actual Performance Performance Targets
2003
0
 
2004
0
 
2005
 
2
2006
 
2


Explanation: No new research/evaluation publications were issued in 2003 or 2004. The indicator was changed from focusing on percentages of publications deemed to be of high quality to focusing on whether or not the modal response (most common judgment) of the review panel is that new IES publications are of high quality. This is because the number of IES research and evaluation publications is currently quite small. With very small numbers, percentages are not very meaningful, because changes in one or two reports can translate into large changes in percentages. In this case, focusing on whether the most common rating is that publications are of high quality is a more meaningful indication of the overall judgment of the review panel.  
Additional Source Information: IES selects a random sample of new research and evaluation publications from IES. Publications are distributed to senior scientists in the field for review.

Frequency: Annually.
Collection Period: 2004 - 2005
Data Available: March 2006
Validated By: No Formal Verification.
Evaluations are only as good as the qualifications of the external review panel. Inclusion of only eminent senior scientists who are distinguished professors in their institutions, editors of premier research journals, and leading researchers in education and special education assures the quality of the data.

 
Indicator 8.1.3 of 4: Of new research and evaluation projects funded by the IES that address causal questions, the percentage of projects that employ randomized experimental designs.
Targets and Performance Data Assessment of Progress Sources and Data Quality
Of new research and evaluation projects funded by the IES that address causal questions, the percentage of projects that employ randomized experimental designs.
Year Actual Performance Performance Targets
2001
32
32
2002
100
75
2003
97
75
2004
90
75
2005
 
75


 
Additional Source Information: IES researchers evaluate all newly funded research and evaluation proposals by IES to identify projects that address causal questions and of those projects, those that use randomized experimental designs to answer those questions. Data will be collected annually. The 75% target for 2002-2005 recognizes that some high-quality research addressing causal questions will not be able to employ randomized experimental designs.

Frequency: Annually.
Collection Period: 2004 - 2005
Data Available: March 2006
Validated By: No Formal Verification.
Evaluations are only as good as the qualifications of the proposal reviewers. Having qualified researchers conduct the reviews, as well as a check of inter-rater agreement in which the two IES researchers independently evaluate a subset of proposals (with minimum inter-rater agreement of 90%), minimizes threats to the validity and reliability of data. Presence of a causal question is defined as instances in which the investigation is designed to examine the effects of one variable on a second variable. A causal relation might be expressed as one variable influencing, affecting, or changing another variable. A randomized experimental design is defined as instances in which there are (a) an experimental (treatment) group and one or more comparison groups, and (b) random assignment of participants to treatment and comparison groups, or random assignment of groups (e.g., classrooms or schools) to treatment and comparison conditions. If a proposal includes a design in which two or more groups of participants are compared, but the PI does not explicitly indicate that random assignment procedures will be used, the proposal is recorded as not using a randomized experimental design.

 
Indicator 8.1.4 of 4: Of new research and evaluation publications funded by IES that address causal questions, the percentage of publications that employ randomized experimental designs.
Targets and Performance Data Assessment of Progress Sources and Data Quality
Of new research and evaluation publications funded by IES that address causal questions, the percentage of publications that employ randomized experimental designs.
Year Actual Performance Performance Targets
2002
100
75
2003
0
75
2004
0
75
2005
 
75


Explanation: No new research/evaluation publications were issued in 2003 or 2004.  
Additional Source Information: IES researchers evaluate all newly funded research and evaluation publications by IES to identify projects that address causal questions and of those projects, those that use randomized experimental designs to answer those questions. Data will be collected annually. The 75% target recognizes that some high quality studies will not be able to employ randomized experimental designs.

Frequency: Annually.
Collection Period: 2004 - 2005
Data Available: March 2006
Validated By: No Formal Verification.
Evaluations are only as good as the qualifications of the proposal reviewers. Having qualified researchers conduct the reviews, as well as a check of inter-rater agreement in which the two IES researchers independently evaluate a subset of publications (with minimum inter-rater agreement of 90%), minimizes threats to the validity and reliability of data. Presence of a causal question is defined as instances in which the investigation is designed to examine the effects of one variable on a second variable. A causal relation might be expressed as one variable influencing, affecting, or changing another variable. A randomized experimental design is defined as instances in which there are (a) an experimental (treatment) group and one or more comparison groups, and (b) random assignment of participants to treatment and comparison groups, or random assignment of groups (e.g., classrooms or schools) to treatment and comparison conditions. If a publication includes a design in which two or more groups of participants are compared, but does not explicitly indicate that random assignment procedures will be used, the publication is recorded as not using a randomized experimental design.

 

Objective 8.2 of 2: Increase the relevance of our research in order to meet the needs of our customers.
Indicator 8.2.1 of 3: The percentage of new research projects funded by IES that are deemed to be of high relevance to education practice as determined by an independent review panel of qualified practitioners.
Targets and Performance Data Assessment of Progress Sources and Data Quality
The percentage of new research projects funded by IES that are deemed to be of high relevance to education practice as determined by an independent review panel of qualified practitioners.
Year Actual Performance Performance Targets
2001
21
 
2002
25
25
2003
60
37
2004
 
50
2005
 
65
2006
 
75


 
Additional Source Information: External panel of qualified practitioners will evaluate the relevance of a random sample of newly funded research proposals. Data will be collected annually. The final target of 75% recognizes that some important research may not seem immediately relevant, but will make important contributions over the long term.

Frequency: Annually.
Collection Period: 2004 - 2005
Data Available: March 2006
Validated By: No Formal Verification.
Evaluations are only as good as the qualifications of the external review panel. Inclusion of only experienced practitioners and administrators in education and special education assures the quality of the data.

 
Indicator 8.2.2 of 3: The number of annual hits on the What Works Clearinghouse Web site.
Targets and Performance Data Assessment of Progress Sources and Data Quality
The number of annual hits on the What Works Clearinghouse Web site.
Year Actual Performance Performance Targets
2003
1,522,922
1,000,000
2004
4,249,668
2,000,000
2005
 
4,500,000
2006
 
5,000,000


 
Additional Source Information: What Works Clearinghouse. Baseline data for number of annual hits is FY 2003.

Frequency: Annually.
Collection Period: 2004 - 2005
Data Available: March 2006
Validated By: No Formal Verification.
A Web-based program will automatically count the hits on this Web site.

 
Indicator 8.2.3 of 3: The percentage of WWC Web site users surveyed randomly who responded to the following statement, ''Evidence provided on the WWC Web site is useful in making decisions about education programs and practices,'' by checking ''agree'' or ''strongly agree.''
Targets and Performance Data Assessment of Progress Sources and Data Quality
The percentage of WWC Web site users surveyed randomly who responded to the following statement, ''Evidence provided on the WWC Web site is useful in making decisions about education programs and practices,'' by checking ''agree'' or ''strongly agree.''
Year Actual Performance Performance Targets
2005
 
30


 
Additional Source Information: Data collected in 2005 will be the baseline data. Subsequent targets will be adjusted after we have the baseline data. There were no available data in 2003 or 2004.

Frequency: Annually.
Collection Period: 2004 - 2005
Data Available: March 2006
Validated By: No Formal Verification.

 

Return to table of contents