National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR)
In 2009, NIDILRR contracted with the National Academy of Sciences (NAS) to conduct an external evaluation of NIDILRR and its grantees. The resulting report was released in 2012 and titled, Review of Disability and Rehabilitation Research: NIDILRR Grant making Processes and Products. The study interviewed NIDILRR staff and management, surveyed customers, stakeholders and sampled past peer reviewers.
The evaluation report resulted in a number of recommendations for process improvement which NIDILRR is working to implement. One of the recommendations made by NAS was that NIDILRR should develop a more systematic and ongoing evaluation process.
With this in mind, NIDILRR recently contracted with Abt. Associates, Inc. in 2012 to develop a ten year evaluation plan and corresponding tools for monitoring and assessment of the NIDILRR research portfolio. Abt. Associates, Inc. has recently begun working on this evaluation plan and related tools.
In 2013, Abt. Associates, Inc. completed its ten year evaluation plan. The plan includes a set of research questions aimed at assessing the effectiveness and efficiency of NIDILRR’s operations as well as the quality and impacts of NIDILRR-funded activities and products. These questions are grouped around four domains which are presented below:
Domain 1 questions:
explores the quality and impacts of NIDILRR funded activities and products.
Domain 2 questions:
explores the effectiveness of NIDILRR’s process for determining grant priorities.
Domain 3 questions:
explores the effectiveness of NIDILRR’s funding decision process.
Domain 4 questions:
explores the effectiveness of NIDILRR’s process of grants monitoring.
Each of these domains has a subset of questions to be addressed as part of the overall evaluation. Abt. Associates then identified tools that needed to be developed to answer these questions.
A broad listing of potential tools is listed below:
- Analysis of the APR data
- Grantee and trainee survey
- Bibliometric analysis
- NIDILRR staff survey
- Expert panel review of grantee products
- NARIC data abstract
- NIDILRR partner survey
- KT feasibility study
- Advisory board survey
- Peer review survey
As part of the plan, Abt. Associates, Inc. also developed a schedule of evaluation activities which identifies the programmatic focus for each year’s analysis, and identifies specific questions to be addressed and the data source. During 2013-2014, Abt. Associates, Inc. begins the second phase of this contract by developing the tools for collecting the data to address key questions.