U.S. Department of Education: Promoting Educational Excellence for all Americans

A r c h i v e d  I n f o r m a t i o n

RA: National Institute on Disability and Rehabilitation Research - 2004

CFDA Number: 84.133 - National Institute on Disability and Rehabilitation Research

Goal 8: To conduct high-quality research that leads to high quality research products
Objective 8.1 of 3: Conduct high-quality research
Indicator 8.1.1 of 3: The percentage of grantee research that is deemed to be good to excellent as reflected in the appropriateness of the designs used and the rigor with which accepted standards of scientific and/or engineering methods are applied.
Targets and Performance Data Assessment of Progress Sources and Data Quality
Percentage of grantee research and development activity rated 4 or greater in appropriateness of study designs, the rigor with which accepted standards of scientific and/or engineering methods are applied, and the degree to which the research and development activity builds on and contributes to the level of knowledge in the field, based on a 5-point Likert-type scale.
Year Actual Performance Performance Targets
2002
54
65
2003
67
70
2004
 
70
2005
 
75
2006
 
75
2007
 
80


Progress: Data for 2002 and 2003 were re-analyzed to be consistent with refinements made to this measure in the 2005 PPMD. It is important to point out that this measure is based in large part on data from NIDRR's summative program reviews and only two of these reviews are scheduled for 2004. This means that the next data collection period for this measure will be 2005 with results available in 2006.

Explanation: Data for 2003 are based on ratings from summative program reviews conducted with 9 RERCs and RRTCs between October and November of 2003. Ratings were performed by expert panelists selected from key stakeholder groups, including other researchers, practitioners and service providers, policy analysts, industry representatives, and individual with disabilities. Five specific areas of performance were rated taken from NIDRR's ''centers of excellence'' model for scientific research and include conducting an innovative program of applied R&D, use of appropriate and rigorous methods, appropriateness of research tools, adequacy and diversity of sample size, and potential contribution to advancement of knowledge and/or product development. Unlike previous years, there were no differences by type of center in the percentage who met the criteria for ''good to excellent'' (i.e., scored 4 or higher) in this area. Both Rehabilitation Engineering Research and Rehabilitation Research and Training Centers scored 67 percent on this measure.   		Source: Other
Other: Other.

Additional Source Information: Triangulation of data from the web-based annual project performance reporting (APPR) system and the planned GPRA-related Expert Panels.

Frequency: Annually.
Collection Period: 2005
Data Available: January 2006
Validated By: On-Site Monitoring By ED.

Improvements: Data are based upon ratings obtained from expert panels during reverse site visits. Extensive efforts have been made to ensure that centers being rated and experts serving as reviewers are conversant with the evidence based and outcomes oriented approaches to the review process.

 
Indicator 8.1.2 of 3: A significant percentage of new studies funded by NIDRR assess the effectiveness of interventions using rigorous and appropriate methods.
Targets and Performance Data Assessment of Progress Sources and Data Quality
Percentage of new studies funded by NIDRR assess the effectiveness of interventions using rigorous and appropriate methods.
Year Actual Performance Performance Targets
2003
 
999


Progress: No data were collected for this measure in 2003. The measure was revised for the FY 2005PM and re-numbered as 7.1.1. It is also important to point out that implementation of NIDRR's planned GPRA-related Expert Panels upon which this measure depends, and which will replace the current system of summative review, has been delayed until the beginning of 2005. This means that the next data collection period for performance measure 7.1.1 will be 2005 with results available in 2006.

Explanation: In 2004 and 2005 NIDRR will develop and test strategies for deriving this measure using information from the web-based annual project performance reporting (APPR) system and preliminary data from the initial round of GPRA Expert Panels. A baseline will be set in 2006 and performance targets for out years will be baseline +5%.   		Additional Source Information: Triangulation of data from the web-based annual project performance reporting (APPR) system and the planned GPRA-related Expert Panels.

Frequency: Annually.
Collection Period: 2005
Data Available: January 2006
Validated By: On-Site Monitoring By ED.

 
Indicator 8.1.3 of 3: The number of publications based on NIDRR-funded research in refereed journals
Targets and Performance Data Assessment of Progress Sources and Data Quality
The average number of publications per award based on NIDRR-funded research and development activities in refereed journals.
Year Actual Performance Performance Targets
2002
2.74
 
2003
 
8
2004
 
5
2005
 
5
2006
 
10
2007
 
10


Progress: The 2002 performance target for this measure was converted to Baseline in the 2005PM to give NIDRR time work out significant data entry and compilation problems associated with the web-based annual project performance reporting system (APPR) used to collect information on publications. These problems were resolved in July 2004 allowing NIDRR to report an accurate and verifiable average of 2.74 peer reviewed publications per award for the three program mechanisms (i.e., RRTCs, RERCs and SCI, TBI and Burn Model systems) required to provide citation data in the existing APPR. Despite improvements in accuracy, however, data for this measure are not reflective of the performance of all l eight of NIDRR's program mechanisms and do not include peer reviewed publications cited in the final reports of grantees whose funding cycle ended in 2002, since these reports are not part of the APPR. NIDRR plans to correct these limitations with the new version of the APPR, which will be complete the end of 2005.

Explanation: The average number of peer reviewed publications for 2002 differs across reporting program mechanisms from a high of 3.48 for Model Systems to 2.89 for RRTCs and 1.1 for RERCs. This variation is due in part to differences in the nature of the research conducted (e.g., medical rehabilitation vs. engineering), the size of the award, and the amount of institutional support provided for publications. Because of this, additional analyses are being conducted to explore the merits of creating sub-measures of this indicator for future reporting. It is also important to explain that actual performance on this measure lags one year behind the collection period. This is necessary in order to capture all the publications published in a given calendar year, but which may not have come out in time to be included in the APPR for that year. The performance target for 2003 publications will be baseline + 5%.   		Source: Performance Report
Contractor Performance Report

Program: NIDRR.
Contractor: Research Triangle Institute, North Carolina.

Additional Source Information: The web-based annual project performance reporting (APPR) system.

Frequency: Annually.
Collection Period: 2003
Data Available: December 2004

Limitations: Data are based upon reports by the funded centers. Concerns have been raised about the potenital for under reporting. Methods to independently confirm publications are planned. The number of publications using the strict definitions employed are likely to fairly represent the productivity of centers in areas related to engineering and medicine. However, these definitions may not fully represent the productivity of centers in other areas.

Improvements: NIDRR is evaluating methods of assessing productivity that fairly represent all parts of the NIDRR grant portfolio.

 

Objective 8.2 of 3: Disseminate and promote use of information on research findings, in accessible formats, to improve rehabilitation services and outcomes.
Indicator 8.2.1 of 1: Grantees deemed to be implementing a plan for widespread dissemination and utilization of validated research findings, developed with stakeholder input and based on measurable objectives, that is producing products and services at sufficient levels and in accessible formats and reaching targeted customers in sufficient numbers, including those from diverse and underserved populations
Targets and Performance Data Assessment of Progress Sources and Data Quality
The percentage of grantees deemed to be implementing a plan for widespread dissemination and utilization of validated research findings, developed with stakeholder input and based on measurable objectives, that is producing products and services at sufficient levels and in accessible formats and reaching targeted customers in sufficient numbers, including those from diverse and undeserved populations
Year Actual Performance Performance Targets
2003 55.50 50
2004   55
2005   60
2006   65
2007   70


Progress: Data for 2003 were re-analyzed to be more consistent with the new measure developed for this area of performance in the FY 2005PM. The number of the new measure also was changed in the FY 2005PM to 7.2.1. Results from 2003 data indicate that 55.5% of centers receiving summative program review were rated 4 or higher on dissemination and promotion of research findings by expert panels. This figure slightly exceeds the original performance target set under the previous version of the dissemination indicator. It is important to point out that only two summative program reviews are scheduled for 2004. This means that the next data collection period for measure 7.2.1 will be 2005 with results available in 2006.

Explanation: Data for 2003 are based on ratings from summative program reviews conducted with 9 RERCs and RRTCs between October and November of 2003. Ratings were performed by expert panelists selected from key stakeholder groups, including other researchers, practitioners, service providers, policy analysts, industry representatives, and individual with disabilities. Five specific areas of performance were rated from NIDRR's ''centers of excellence'' model for Dissemination, Relevance and Productivity and include evidence that the Center is implementing a systematic dissemination plan with clearly stated and appropriate goals, dissemination outputs are being evaluated for quality and relevance to end-users, and consumer-oriented products and programs are accessible to persons with disabilities. The percentage of Centers rated 4 or higher on these CoE indicators differs by type of center with 67% (2 out of 3) of RERCs meeting the criteria compared ot 50% (3 out of 6) of RRTCs.   		Additional Source Information: Triangulation of data from the web-based annual project performance reporting (APPR) system and the planned GPRA-related Expert Panels.

Frequency: Annually.
Collection Period: 2005
Data Available: January 2006
Validated By: On-Site Monitoring By ED.

 

Objective 8.3 of 3: Ensure Utility of Research Problems and Products to End-Users
Indicator 8.3.1 of 1: Outcomes-Oriented Measure of Results of R&D Investment: The number of new or improved assistive and universally-designed technologies, devices and systems developed by grantees that are deemed to improve rehabilitation services and outcomes and/or enhance opportunities for full participation, and are successfully transferred to industry for potential commercialization.
Targets and Performance Data Assessment of Progress Sources and Data Quality
Number of new or improved assistive and universally-designed technologies, devices and systems developed by grantees that are rated ''good to excellent'' in ability to improve rehabilitation services and outcomes and/or to enhance opportunities for full participation, and are successfully transferred to industry for potential commercialization.
Year Actual Performance Performance Targets
2004
 
999
2005
 
5


Explanation: A baseline will be set in FY 2004 based on analysis of data from the web-based annual project performance reporting (APPR) system and judgments of expert panels. For FY 2005 the target will be 5 percent over the baseline.   		Source: Performance Report
Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utillization Projects).
Program: National Institute on Disability and Rehabilitation Research..

Additional Source Information: Triangulation of data from the web-based annual project performance reporting (APPR) system and program review-type meetings with expert panels.

Frequency: Annually.
Collection Period: 2004
Data Available: December 2004
Validated By: On-Site Monitoring By ED.
Review by expert panel

Improvements: To reduce the costs and improve the efficiency of collecting qualitative judgements from experts panels, in 2004 NIDRR will experiment with using Internet-based alternatives to face to face program-review-type meetings.

 

Return to table of contents