U.S. Department of Education: Promoting Educational Excellence for all Americans

A r c h i v e d  I n f o r m a t i o n

National Institute on Disability and Rehabilitation Research (NIDRR) - 2003

CFDA Number: 84.133 - National Institute on Disability and Rehabilitation Research


Goal 8: To conduct high-quality research that leads to high quality research products
Objective 8.1 of 4: Conduct high-quality research
Indicator 8.1.1 of 3: The percentage of grantee research that is deemed to be good to excellent as reflected in the appropriateness of the designs used and the rigor with which accepted standards of scientific and/or engineering methods are applied.
Targets and Performance Data Assessment of Progress Sources and Data Quality
Percentage of research is deemed to be good or excellent in the appropriateness and rigor of experiment design and the rigor with which accepted standards of scientific and/or engineering methods are applied.
Year Actual Performance Performance Targets
2002
68
65
2003
 
70
2004
 
70
2005
 
75
2006
 
75
2007
 
80
Status: Target exceeded

Progress: Actual performance for FY 2002 exceeded the revised performance target of 65% by 3%, indicating improvement in expert reviewers' ratings of ''quality of research This target was revised from 70% to 65% in response to refinements in measurement aimed at eliminating the previous ''double-barreled” measure, based on the average of ratings for ''scientific rigor'''and ''usefulness of research,'' which tended to result in inflated scores. This occurred because ''usefulnesss'' typically was rated higher than ''rigor.'' Given this improvement in measurement, and additional changes to the program review process focused on emphasizing accountability for results (i.e., outcomes), it is difficult to compare performance on this indicator for 2002 to previous years.

Explanation: Data are from the FY 2002 series of summative program reviews conducted with a total of 28 centers (9 RERCs and 19 RRTCs). To satisfy this indicator, grantees must receive an average rating of 4 or greater, indicating ''good to excellent,'' based on a five-point Likert-type scale. Ratings were completed by expert panels of 10-12 reviewers selected from key stakeholder groups, including other researchers and engineers, practitioners, policy analysts, industry representatives, disability advocates and consumers with disabilities. Specific areas of performance are taken from NIDRR's ''centers of excellence'' model and include the rigor and feasibility of scientific methods, appropriateness of research tools, and adequacy and diversity of sample size. Among the centers reviewed, the % rated good to excellent in research varied from a high of 86% and 78% for Health & Function RRTCs and RERCs, respectively, to 50% for Employment and Community Integration RRTCs. This variability, in part, reflects differences in the research designs and methods utilized by sub-areas of disability and rehabilitation resear  
Source: Other
Other: Expert Panel.
Sponsor: NIDRR.
Date Sponsored: 09/30/2002.

Additional Source Information: Data are based upon ratings obtained from expert panels during reverse site visits.

Frequency: Annually.
Collection Period: 2003
Data Available: December 2003
Validated By: On-Site Monitoring By ED.

Limitations: The data for this indicator are limited to expert panelists' ratings from the two largest program mechanisms within the NIDRR portfolio -- i.e., RERCs and RRTCs. Within these two programs, the data are further limited to the subset of centers who were funded initially in 1998 and scheduled for summative program review in 2002.

Improvements: Extensive efforts are being made to ensure that centers being rated and experts serving as reviewers are conversant with the evidence based and outcomes oriented approaches to the review process. To improve the measurement of this indicator in future years NIDRR is planning to: (1) augment the data source to include information from the web-based annual performance reporting (APR) system and expand the sample of centers and projects assessed for ''quality of research;'' and (2) refine the ''centers of excellence' (CoE) criteria upon which the ratings of research quality are based. These improvement are scheduled to go into effect in FY 2005 based upon a redesign of the web-based performance reporting system and psychometric analysis of the CoE criteria.

 
Indicator 8.1.2 of 3: A significant percentage of new studies funded by NIDRR assess the effectiveness of interventions using rigorous and appropriate methods.
Targets and Performance Data Assessment of Progress Sources and Data Quality
Percentage of new studies funded by NIDRR assess the effectiveness of interventions using rigorous and appropriate methods.
Year Actual Performance Performance Targets
2003
 
999


Explanation: In FY 2004 NIDRR will set a baseline for this new indicator based on analyses of FY 2003 data from the web-based annual reporting (APR) system and judgements of expert panelists. The 2005 Target will be the baseline + 5%.  
Source: Performance Report
Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utillization Projects).
Program: NIDRR.

Additional Source Information: Triangulation of data from the RTI web-based annual performance reporting (APR) system & program review -type meetings with expert panels.

Frequency: Annually.
Collection Period: 2003
Data Available: June 2004
Validated By: On-Site Monitoring By ED.
Review of expert panel representing key stakeholder groups

Improvements: To reduce the costs and improve the efficiency of collecting qualitative judgements from expert panels, in 2004 NIDRR will experiment with using Internet-based alternatives to face-to-face program review-type meetings.

 
Indicator 8.1.3 of 3: The number of publications based on NIDRR-funded research in refereed journals
Targets and Performance Data Assessment of Progress Sources and Data Quality
The number of publications based on NIDRR-funded research in refereed journals
Year Actual Performance Performance Targets
2003
 
999
2004
 
8
2005
 
10
2006
 
10
2007
 
10


Progress: See explanation below.

Explanation: The performance target for 2003 was converted to Baseline to allow NIDRR to conduct a comprehensive empirical analysis of what the targets should be for grantees from large centers as well as from other program mechanisms. This analysis will be conducted in 2004 using FY 2003 data from the web-based Annual Performance Report (APR) system and expert judgements of outside reviewers. The analysis will also evaluate the merits of developing sub-measures of this indicator to reflect different expectations for publication in peer-reviewed journals associated with different subfields of rehabilitation and disability research. Targets for 2005 will be the Baseline + 5%.  
Source: Performance Report
Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utillization Projects).
Program: NIDRR.

Additional Source Information: Triangulation of data from RTI web-based annual reporting (APR) system and program review-type meetings with expert panels

Frequency: Annually.
Collection Period: 2003
Data Available: June 2004
Validated By: On-Site Monitoring By ED.
Verified by scrutiny of reported publications by Dept. of Education staff.

Limitations: At present, data on numbers of publications in refereed journals are based upon self-reports of grantees from NIDRR's large centers only -- i.e., RRTCs, RERCs and model systems. Concerns have been raised about the representativeness of these centers for all of NIDRR's portfolio and the potential for under reporting. Methods to expand the sample of grantees to include other program mechanisms and to independently confirm publications are planned. The number of publications using the strict definitions of refereed journals are likely to fairly represent the productivity of grantees in areas related to engineering and medicine. However, these definitions may not fully represent the productivity of grantees in other areas.

Improvements: NIDRR is evaluating methods of assessing productivity that fairly represent all parts of the NIDRR grant portfolio (for more details see Explanation above).

 

Objective 8.2 of 4: Disseminate and promote use of information on research findings, in accessible formats, to improve rehabilitation services and outcomes.
Indicator 8.2.1 of 1: Grantees deemed to be implementing a plan for widespread dissemination and utilization of validated research findings, developed with stakeholder input and based on measurable objectives, that is producing products and services at sufficient levels and in accessible formats and reaching targeted customers in sufficient numbers, including those from diverse and underserved populations
Targets and Performance Data Assessment of Progress Sources and Data Quality
The percentage of grantees deemed to be implementing a plan for widespread dissemination and utilization of validated research findings, developed with stakeholder input and based on measurable objectives, that is producing products and services at sufficient levels and in accessible formats and reaching targeted customers in sufficient numbers, including those from diverse and underserved populations
Year Actual Performance Performance Targets
2002 68 50
2003   50
2004   55
2005   60
2006   65
2007   70
Status: Target exceeded

Progress: Actual performance for FY 2002 exceeded targeted performance by 18%, indicating significant improvement in expert reviewers' ratings of dissemination activities and potential outcomes. However, given major refinements to the program review process between aimed at clarifying the ''centers of excellence'' model and emphasizing accountability for results (i.e.outcomes), it is difficult to directly compare performance for 2002 to previous years.

Explanation: Data are from the FY 2002 series of summative program reviews conducted with a total of 28 centers (9 RERCs and 19 RRTCs). To satisfy this indicator, grantees must receive an average rating of 4 or greater, indicating ''good to excellent,'' based on a five-point Likert-type scale. Ratings were completed by expert panels of 10-12 reviewers selected from key stakeholder groups, including other researchers and engineers, practitioners and service providers, policy analysts, industry representatives, disability advocates and consumers with disabilities. Specific areas of performance rated are taken from NIDRR's ''centers of excellence'' (CoE) criteria for Relevance, Productivity & Dissemination and include ''implementation of systematic dissemination plan to reach diverse audiences'' and ''demonstrates that research results are contributing to improved rehabilitation outcomes' The % of centers rated good to excellent in this domain varied from a high of 74% for RERCs to 65% for RRTCs. This variability, in part, reflects differences in the nature of the dissemination products and services produced by RERCs and RRTCs.  
Source: Other
Other: Expert Panel.
Sponsor: NIDRR.
Date Sponsored: 09/30/2002.

Additional Source Information: Data are based upon ratings obtained from expert panels during reverse site visits.

Frequency: Annually.
Collection Period: 2003
Data Available: December 2003
Validated By: On-Site Monitoring By ED.
Review by expert panels representing key stakeholder groups

Limitations: The data for this indicator are limited to the two largest program mechanisms within the NIDRR portfolio -- i.e., RERCs and RRTCs. Within these two programs, the data are further limited to the subset of centers who were funded initially in 1998 and scheduled for summative program review in 2002.

Improvements: To improve the future measurement of this indicator NIDRR is planning to: (1) augment the data source to include information from the web-based annual performance reporting (APR) system and an expanded sample of centers and projects from a range of funding mechanisms; and (2) refine the ''centers of excellence' (CoE) criteria upon which the ratings of dissemination are based. These improvement are scheduled to go into effect in FY 2005 based upon a redesign of the APR and psychometric analysis of the CoE criteria.

 

Objective 8.3 of 4: Ensure Utility of Research Problems and Products to End-Users
Indicator 8.3.1 of 2: Research and development projects conducted by NIDRR grantees deemed to be addressing problems or issues of “high relevance” to consumers and other end-users
Targets and Performance Data Assessment of Progress Sources and Data Quality
Published Papers and Presentations by NIDRR trainees and fellows that contribute to the study of rehabilitation
Year Actual Performance Performance Targets
2003   999


Explanation: A baseline will be set in FY 2004 based on analyses of data from the FY 2003 web-based annual performance reporting (APR) system and judgements of expert panels. The FY 2005 target is 5 percent over the baseline. Out year targets will increase by five percentage points up to 80 percent.  
Source: Performance Report
Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utillization Projects).
Program: NIDRR.

Additional Source Information: Triangulation of RTI web-based annual performance reporting (APR) system and program review-type meetings with expert panels

Frequency: Annually.
Collection Period: 2004
Data Available: December 2004
Validated By: On-Site Monitoring By ED.
Review by expert panel representing key stakeholder groups

Improvements: Extensive efforts are being made to ensure that centers being rated and experts serving as reviewers are conversant with the evidence based and outcomes oriented approaches to the review process. To reduce the costs and improve the efficiency of collecting qualitative judgements from expert panels, in 2004 NIDRR will experiment with using Internet-based alternatives to face-to-face program review-type meetings.

 
Indicator 8.3.2 of 2: Consumer-oriented products and information disseminated by grantees based on NIDRR-funded research that is deemed to be of “high utility” by individuals with disabilities and other end-users
Targets and Performance Data Assessment of Progress Sources and Data Quality
Successful completion of planning tasks and conduct of capacity building and outreach conference. Participation of at least 25 individuals from currently funded entities and individuals from other eligible entities.
Year Actual Performance Performance Targets
2003
 
999


Explanation: A baseline will be set for FY 2004 using FY 2003 data from the web-based annual performance reporting (APR) system and judgements of expert panels. The FY 2005 target will be 5 percent over the baseline. Out year targets will increase by five percentage points up to 80 percent.  
Source: Other
Other: Other.

Additional Source Information: Qualitative ratings by a panel of consumers and other end-users of consumer-oriented products and materials developed by grantees for dissemination.

Frequency: Annually.
Collection Period: 2004
Data Available: December 2004
Validated By: On-Site Monitoring By ED.
Review by expert panel representing key stakeholder groups.

Improvements: Extensive efforts are being made to ensure that centers being rated and experts serving as reviewers are conversant with the evidence based and outcomes oriented approaches to the review process. To reduce the costs and improve the efficiency of collecting qualitative judgements from expert panels, in 2004 NIDRR will experiment with using Internet-based alternatives to face-to-face program review-type meetings.

 

Objective 8.4 of 4: Conduct performance evaluation to ensure program improvement and accountability for results
Indicator 8.4.1 of 1: The percentage of projects that are deemed to have an evaluation plan that is conducted on an ongoing basis and is tied to measurable objectives for assuring quality of implementation and efficient project management, and for assessing the relevance of products and services produced and the extent to which anticipated outcomes are being achieved
Targets and Performance Data Assessment of Progress Sources and Data Quality
The percentage of projects that are deemed to have an evaluation plan that is conducted on an ongoing basis and is tied to measurable objectives for assuring quality of implementation and efficient project management, and for assessing the relevance of products and services produced and the extent to which anticipated outcomes are being achieved.
Year Actual Performance Performance Targets
2003
 
999


Explanation: A baseline will be set in FY 2004 based on analyses of data from the FY 2004 series of Formative Program Reviews and the judgements of expert panelists. The targets for FY 2005 will be 5 percent over the baseline. Out year targets will increase by five percentage points up to 70 percent.  
Source: Performance Report
Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utillization Projects).
Program: NIDRR.

Additional Source Information: Triangulation of data from RTI web-based annual performance reporting (APR) system and program review-type meetings with expert panels.

Frequency: Annually.
Collection Period: 2004
Data Available: December 2004
Validated By: On-Site Monitoring By ED.
Review by expert panels representing key stakeholder groups

Improvements: Extensive efforts are being made to ensure that centers being rated and experts serving as reviewers are conversant with the evidence based and outcomes oriented approaches to the review process.

 

Return to table of contents