ATTACHMENT C DRAFT OERI Expert Panel System: Overview of Review Process Overview As described in the Background section of this Statement of Work, OERI has been charged with establishing a system of Expert Panels to review K 12 educational programs and to recommend to the Secretary of Education those programs that should be designated as promising and exemplary. Program submissions will be evaluated against criteria set forth in regulations established by OERI, to which Expert Panels may make additions or modifications, as appropriate. The evaluation criteria cover four broad categories: educational significance, quality of the program, usefulness to others, and evidence of success. After programs have been reviewed, recommended, and approved by the Secretary of Education, information regarding designated programs will be disseminated widely through the National Dissemination System and other forums to provide the public with access to programs and practices that "work" to improve teaching and student learning. The system will consist of a number of different Expert Panels. A possible structure may be to establish five or six Expert Panels, each representing a "major" educational subject (for example, science, mathematics, history, or reading). Each year, Expert Panels may decide to focus on a specific topic within their fields. For example, a Reading Expert Panel may focus on early reading in one year and using technology for reading in another. To date, two pilot Expert Panels have been established one on mathematics and science education and one on gender equity. These pilot panels have established tentative procedures for reviewing programs. Because the panels were pilots, the procedures that they established may not be adopted ultimately by the entire Expert Panel system; however, the tentative procedures are described here to provide background for the work requested under this RFP. Review Process The Expert Panel review process, as currently conceptualized, consists of a number of different steps and parties. The steps of the process from soliciting programs to disseminating information as well as the parties involved in each step are described briefly below. Program Solicitation. The first step in the Expert Panel activities is to solicit programs for review. OERI has been encouraged to actively solicit programs for review. To that end, the Expert Panels, OERI, and OERI's contractor will publicize the Expert Panel process using a number of different methods, including, but not limited to: postings on the Internet, announcements in professional journals, presentations and brochures at professional meetings, and solicitations from colleagues. Program Submission. Individuals interested in submitting programs for review may respond to solicitations by contacting OERI for the submission guidelines. The guidelines will provide instructions for what materials to include in a program submission, as well as the specific evaluation criteria upon which the reviews will be based. Although the submission forms for each of the Expert Panels will follow similar formats, the Expert Panels may decide to customize their own submission forms. Distribution to Expert Panels. After program submissions are received by OERI, they will be distributed to the appropriate Expert Panels for review. Expert Panels may decide to conduct program reviews entirely internally, or they may choose to establish a system of subpanels to assist in the process. Subpanels may be comprised of individuals outside of the 15-member Expert Panel who are approved by the Assistant Secretary to assist in the review process. To facilitate the system of subpanels, the contractor will maintain a data base of reviewers who have been approved by the Assistant Secretary to participate in program reviews. Expert Panel Review. Each program submitted will be judged against the evaluation criteria established by the particular Expert Panel. As described above, the evaluation criteria cover four broad categories: educational significance, quality of program, usefulness to others, and evidence of success. Each Expert Panel may, however, choose to adopt additional evaluation criteria, as appropriate. Detailed evaluation rubrics and scoring schema will be established for each Expert Panel's criteria to ensure that reviewers rate programs consistently, both internally and across the different Expert Panels. Expert Panel members will receive training on interpreting and applying the evaluation rubrics. Evaluation Validation. After the Expert Panels have completed their reviews, all programs recommended as exemplary will be reviewed further by a nine-member evaluation team. This group will review the evaluation data provided by each program recommended as exemplary, to ensure that the data provided support the evidence of success claims made in the submission. In addition, since this group of evaluators will review the programs identified as exemplary by each Expert Panel, it will ensure that programs designated as exemplary by different Expert Panels are of similarly high quality. Secretary of Education Approval. The Secretary of Education will review Expert Panel recommendations and formally confirm the promising and exemplary designations. The Secretary of Education, therefore, has final approval over which programs are designated. Programs that are not designated may be resubmitted for consideration in a subsequent review cycle. Information Dissemination. After programs have been recommended by the Expert Panels, validated by the evaluation team, and approved by the Secretary, information regarding the designated programs will be disseminated through a number of vehicles. As stipulated in the authorizing legislation, these vehicles may include: the Educational Resources Information Clearinghouse; the regional education laboratories; the National Diffusion Network; and a number of other private and non-profit entities.