The Department of Education and its predecessors, the U.S. Office of Education and the Education Division of the Department of Health, Education and Welfare, have a long history of supporting panel processes for the purpose of reviewing replicable programs before investing in their dissemination. Following is a review of the major activities over almost three decades that have brought us to the current System of Expert Panels. Most features of review-panel operations have continued to evolve over time in response to federal funding for development, the level of innovation in the field, the state of the art of program evaluation, and the federal panel leaders.
As shown in the following "Timeline of the History of Department of Education Review Panels," two Department-wide, multi-topic program review panel efforts (the Joint Dissemination Review Panel and the Program Effectiveness Panel) preceded the System of Expert Panels. Concurrently, various Offices within the Department operate their own more specialized program-review activities. Additional information on both the multi-topic review panels and these more specialized panels and activities are described in Disseminating Promising and Exemplary Programs: Planning a System of Expert Panels, (U.S. Department of Education, December 1997, ED Pubs ORAD 97-1216 or ERIC ED415575); Learning from Consumer-Oriented Review Efforts to Guide the Development of a System of Expert Panels to Identify and Share Promising and Exemplary Products and Programs (U.S. Department of Education, May 1998, ED Pubs ORAD 1999-1203-I & II) and Special Feature: Sharing the Best: Finding Better Ways for the Federal Government to Use Evaluation to Guide the Dissemination of Promising and Exemplary Education Solutions (Klein, S. 1993, Evaluation and Program Planning 16-3, ERIC EJ 471921-29). Almost all of these model program review efforts have had substantial emphasis on judging the evidence of effectiveness. [The ED Pubs documents are free while the limited supplies last. See http://www.ed.gov/pubs/index.html or for ERIC database searches go to http://www.eric.ed.gov/ERICWebPortal/Home.portal ]
The U.S. Office of Education under the leadership of Assistant Commissioner John Evans started its multi-topic Dissemination Review Panel (DRP) in 1972. It became the Joint Dissemination Review Panel (JDRP) when the recently created National Institute of Education, (NIE) began to participate in 1975 to help implement its dissemination responsibilities. Its key purpose was to enable the federal staff to make responsible recommendations about effective programs that went beyond previous practices of basing judgments on anecdotes, site visit reports, favorable ratings from users, experts' judgments of educational soundness, and popularity with adopting schools. Only programs approved by the JDRP were to be endorsed by or disseminated as exemplary using federal Education Division funds.
In the late 1960s and 1970s the federal government supported the development of many innovative demonstration or model programs under the Elementary and Secondary Education Act Titles III, IV, special education, bilingual education, women's education, and career education as well as by the NIE National Research Centers and Regional Educational Laboratories. Some of these federal efforts required or encouraged evaluations, and thus some evidence became available. For example, many of the NIE sponsored programs used a field test and revision strategy, and the National Diffusion Network supported some evaluation of adoptions of approved programs that received dissemination funding.
With the creation of the National Diffusion Network (NDN) in 1974, which provided funding for Developer/Demonstrators and State Facilitators to help disseminate JDRP-approved programs, the popularity of national validation efforts grew. Starting in 1984, JDRP recertification reviews were also required after four years for previously approved programs to remain eligible for NDN funding.
Title III-IV funding was state administered, and many states set up state-level review panels to determine worthiness for in-state and/or regional dissemination of innovations. The state Identification-Validation-Dissemination (IVD) reviews used a variety of criteria, including evidence of effectiveness, and typically involved site visits that somewhat resembled accreditation visits. Programs that had received IVD approval at the state level frequently went on for national approval by JDRP. Thus, until the demise of Title III-IV, there was an active, multi-level review-panel effort supported by federal dollars.
The JDRP was composed of over 20 in-house federal staff with evaluation and dissemination expertise who were drawn from many Education Division offices. Additional federal staff from a variety of program offices selected and encouraged submissions from developers funded through grants and contracts. Sometimes the federal offices obtained submissions from State IVD reviews. Sometimes they paid for a contractor to conduct a national search. Many offices worked closely with the JDRP Chairperson to make sure the submissions were as responsive to JDRP expectations as possible. The offices were required to transmit their submissions to JDRP with a memorandum assuring that they had reviewed them. Education project officers accompanied the developers to the Panel meetings at the Office/Department of Education in Washington. Since the JDRP saw its role as reviewing the evidence of effectiveness, they relied on the submitting offices to pre-review the submissions "for social fairness of content and procedures, and for honesty, completeness, and accuracy of the evidence supporting the intervention's effectiveness." (The Joint Dissemination Review Panel Ideabook, 1977, p. 1, ERIC ED148329.htm).
A JDRP Executive Secretary kept the files and coordinated many of the Panel functions. The JDRP Chairs worked with the Panel members to develop and revise submission guidelines. Suggestions for preparing for the JDRP were included in The JDRP Ideabook, developed under the leadership of Chair Lois-ellin Datta to help submitters think about both different ways to provide relevant evidence and the pitfalls. It also contained submission guidelines and model submissions.
Developers, and often their program evaluators, came to the public JDRP meetings for an in-person presentation and question-and-answer session with members of the Panel who had reviewed the 10 page submissions before the meeting. During some years the panelists would prepare written reviews in advance. They would make changes as needed after the Panel discussion. Although the Panel members often followed up on each other's questions, there was little public or private deliberation among them on specific submissions. After discussing a submission, the panelists voted on the spot, with simple yes/no votes on secret ballots. Approval required a majority of the typically seven panelists.
The Panel Chairs provided the submitters and education office staff with explanations of what the Panel saw as the strengths and weaknesses and the reasons for the Panel decisions. If resubmission was encouraged, the Panel Chair would explain the kind of evidence that would more convincingly make the case. In addition to the oral summaries, Panel Chairs sent the submitters and program offices letters with detailed explanations after the Panel meeting.
There was a formal policy that JDRP be used as a prerequisite for dissemination of exemplary programs throughout the Education Division. This was followed by some offices more faithfully than others. Its most faithful implementer was the National Diffusion Network which required JDRP approval prior to providing dissemination funding or even a listing in the NDN Educational Programs that Work catalogs. Panelists noted improvements in submissions over the years despite limited federal support of evaluations of effectiveness or impact to learn what development or demonstration programs were worth disseminating.
The Program Effectiveness Panel Years, 1987-96.During 1987 the JDRP was replaced by the Program Effectiveness Panel (PEP). After the termination of Title III-IV and its substantial pool of innovative projects, there were fewer other projects that could meet the evidence of effectiveness requirements of the JDRP. Additionally complaints about the perceived narrowness of the JDRP's approach were taken seriously by Chester Finn, the Assistant Secretary of the Office of Educational Research and Improvement (OERI) when he directed staff to revise the NDN regulations to create the Program Effectiveness Panel (PEP) under the initial leadership of John Ralph. The key regulatory changes from JDRP to PEP included:
Like the JDRP, the PEP chair was initially a federal Panel member, and Department Assistant Secretaries had to sign-off on submissions. Similarly, the PEP criteria made it very clear that voting by six Panel members was focused on examining evidence of effectiveness. A heavy emphasis was placed on evidence to support important results, since in the late 1980s, Department officials were concerned that some program objectives were trivial, inaccurate, or inappropriate. As with JDRP, submitters were asked to present evidence to support claims of positive impact rather than the often idealistic project goals or objectives. To broaden the treatment of evidence of effectiveness, PEP recognized that:
Key PEP procedural changes included:
PEP criteria, guidelines, and examples of claims were documented in Making the Case: Evidence of Program Effectiveness in Schools and Classrooms -- Criteria and Guidelines for the U.S. Department of Education's Program Effectiveness Panel (1988, ERIC ED306706).
The PEP reviewed 30-35 submittals plus revalidation requests each year. Unlike the early JDRP policy, programs submitted to PEP did not need to be developed with federal funds. Candidate programs came from an increasing variety of sources including foundations, professional associations, advocacy groups, and universities. Like the JDRP, PEP had about a 60% approval rate. As previously mentioned some perceived the JDRP as narrow because it had the reputation of demanding rigorous evidence with classical evaluation designs and instruments and attention to statistical significance. However, examination of early JDRP approved submittals suggests there was actually much wider latitude in the type of evidence than most realized. Conrad Katzenmeyer, one of the PEP Chairs notes that the PEP got tougher than JDRP especially related to evaluation design adequacy. This change was especially visible in criticisms of revalidation submissions. He also noted that only a few PEP candidates took advantage of the broader perspective on evaluation evidence promoted by the PEP guidelines.
As was true with the JDRP, PEP approval remained essential for participation in NDN but was not used uniformly for all Department of Education dissemination support or recommendations. However, some Departmental initiatives such as Even Start, Tech Prep, and several other vocational education funding programs included PEP evaluation criteria in their requirements for demonstration funding.
Transition to the System of Expert Panels, 1994-1999As discussions began in the early to mid 1990s about how to revamp the overall National Diffusion Network, the redesign of the PEP review process was discussed in many forums. Some National Diffusion Network participants pressed for a less rigorous approval process that did not depend as much on systematic evidence of effectiveness especially related to replication, because they saw great value in adaptation rather than adoption. Others felt that it was especially important for reviewers to have more contact with candidate programs and understand their history of results in multiple sites. An old notion, developing graded distinctions between effective and "promising" projects, was resurrected as a way to entice more "cutting edge" programs into the NDN. And the notion of paying attention to the content and research base of the programs re-emerged as central. As a result, some desired a change in the composition of reviewers to be subject matter experts rather than evaluation experts.
With the passage of the 1994 OERI re-authorization which included NDN, Expert Panels, and related guidance for the National Research Institutes to identify and share what works, deliberations about redesign of the NDN focused on how the NDN and the PEP might be coordinated with the new Expert Panel requirements. This led to piloting a system of topic focused Expert Panels before the NDN funding and the contract supporting the PEP work were eliminated in 1996.
Many individuals and organizations were involved in planning the System of Expert Panels. The OERI National Educational Research Policy and Priorities Board commissioned papers from a wide variety of individuals to examine all aspects of review panels, ranging far beyond the experiences of the Department of Education, and this Board approved both the 1996 proposed rules and the 1997 final rules for the System of Expert Panels. The OERI Office of Reform Assistance and Dissemination developed conceptual and planning papers (see previously cited documents available from ED Pubs), established a listserv to help individuals share information and advice about the system, and provided staff and contractor support for most of the Expert Panels.
Constituency groups of the pilot Gender Equity Expert Panel and the pilot Mathematics and Science Education Expert Panel played a major role in suggesting Panel members and topics. These pilot Panels (designated operational Panels in late 1997) were major steps forward in bringing subject matter, evaluation, and dissemination expertise to the forefront of the review process. The pilot Panel members, their various support contractors, and ORAD staff developed and shared many creative approaches to implement the regulations. The newer Expert Panels in the areas of educational technology and safe, disciplined, and drug-free schools have built on and learned from the pilot efforts. They are also contributing to overall thinking about Expert Panels. For example, the familiarity of many of the Safe, Disciplined, and Drug-free Schools Panelists with medical evaluation models has influenced that Panel's interpretation of criteria of effectiveness, and the Technology Panel is demonstrating how technology such as obtaining submissions via the Internet can be used to expedite Expert Panel reviews.
In addition to a change from one multi-topic-focused panel to separate topic-focused panels with very interested constituency groups, the current Expert Panels differ from the PEP in the use of:
The current four Expert Panels continue the effort to design a panel review process that is responsive, flexible, yet credible. Challenges remain about best ways to sort out the distinctions between promising and exemplary, figure out how to entice program submissions, determine the appropriate balance between evidence of effectiveness and other positive qualities of programs, obtain a complete picture of positive and negative impact evidence with the least burdensome approaches, and find ways for panelists to become knowledgeable about programs while respecting the reasonable amount of time that a volunteer can afford. Additionally, the System of Expert Panels is exploring ways to coordinate its efforts with funding for the evaluation, revision, and dissemination of worthy submissions.
The Value of Strong Historical Roots and Consistent Long-term VisionsThe visions for all of these federal education review panels have been more comprehensive than their actual implementation and impact. It is important to keep these visions in mind and to reshape them as roots grow and as horizons broaden.
Vision 1: Conduct the reviews to help education consumers make well informed selections of programs that are likely to work for them. All the panels focused on helping the Department make sound program endorsements and dissemination investments. None of the panels believed they were identifying the single best solution for the education challenges. They all saw their purpose as identifying multiple interventions that worked and had other positive attributes. The panels wanted the potential users to understand the strengths and weaknesses of these interventions to make wise selections from multiple exemplary programs.
Vision 2: Support impact evaluations especially for programs that look good on other criteria. All panels saw the need for federal education offices and others to support evaluations of development and demonstration interventions that were worthwhile on various quality and replicability criteria. For example, John Evans, the first JDRP chair urged program developers to seek funds for improved evaluations of submissions that were not approved. In the PEP years, NDN often arranged for a contractor to provide assistance in the design and reporting of the evaluations. The "promising" designation by the Expert Panels was developed with this evaluation support in mind.
Vision 3: Attain Department-wide use of panel criteria and review procedures. All of the panels wanted their criteria and procedures to be used consistently in all Department dissemination-funding and endorsement decisions. Much of this happened when there was NDN funding only for programs approved as exemplary by JDRP or PEP, but these panels were never the exclusive Department-wide screen their leaders envisioned. The combined numbers of submissions for the four Expert Panels in the various topic areas is much larger than for any of the years of the JDRP or PEP. As submitters gain trust in the fairness and advantages of participating in this process, the number of submissions should increase. Similarly, the System of Expert Panels is being developed to allow ED program offices flexibility to operate panels to meet their own needs while maintaining some consistency, particularly in designations of promising and exemplary.
Vision 4:Use review panels as a catalyst for helping the nation learn what works by building evaluation capacity of developers and users and by guiding funding of new R&D. All the panels, as shown in the JDRP Ideabook and the PEP Making the Case, saw their role as encouraging both developers and users to learn how to do better evaluations of replicable model programs. The Expert Panels have been increasing evaluation capacity in their topic areas as they focus on a variety of ways of using evidence to support claims of positive impact as well as systematic ways to assess the other three criteria categories. Also, since the Expert Panels are focused on specific topics, they hope to influence funding of new R&D in their topic areas that addresses specific needs they have been able to identify over several review cycles. Over the past 27 years, there has been a growing awareness among educators who have an opportunity to select programs, that they should ask "Does it work?" These federal review panels have not only encouraged asking this question, but in many cases they have made it more likely that there will be some answers.
Vision 5: Encourage complementary high quality review efforts of other organizations especially if they include effectiveness criteria. As in the JDRP and PEP years, federal staff don't want their panels to be the "only program review game in town." JDRP worked closely with the state IVD panels and in some cases with the more specialized efforts in federal offices such as Bilingual Education's Academic Excellence Program and the Vocational Education programs. The Department established a separate category to fund Dissemination Process Projects that managed their own review efforts. In the future, ED might use its web site to provide links to other high quality review efforts and their databases of practices, products, and programs.
SummaryThe long and productive history of federal education program review panels provides many lessons from which to move forward with the System of Expert Panels. The historical roots and emphasis on encouraging the use of federal and other organizations to identify and disseminate replicable model programs that have credible evidence to support important claims of positive impact are strong. The current System of Expert Panels should remember these roots and maintain and build on the long-term visions as it takes advantage of an increased emphasis on improving educational performance and advances in technology and evaluation.
Authors and reviewers who contributed to this paper on the Historical Roots of the System of Expert Panels include: Sue Klein, Linda Jones, and Lois Weinberg, OERI; Christine Dwyer, Susan Klaiber, and Peggy Simon, RMC Research Corporation; Lois-ellin Datta, Datta Analysis; John Evans, Consultant in Educational Research and Evaluation; and Conrad Katzenmeyer, National Science Foundation. All have played active roles in the panels from different perspectives in different years.
This page last updated June 13, 2001 (jcl)