July 25, 1996

EXPERT PANEL IN MATHETMATICS AND SCIENCE EDUCATION MEETING

Introduction

The first meeting of the Expert Panel in Mathematics and Science Education provided attending panel members the opportunity to discuss policies and procedures that will affect panel operations during its first year.

Assistant Secretary Robinson opened the meeting by emphasizing the panel's role in providing feedback between practitioners and the Department of Education (ED). She recognized ED's need for greater connection to education practitioners, stating that the panel would provide a vital link to classrooms. She stressed the necessity of having confidence in the standards that will be used to designate programs as promising or exemplary, because the programs will carry the Federal government's seal of approval. Moreover, the panels will provide ED the opportunity to demonstrate to the Congress and the public that it spends resources efficiently and makes a difference in the quality of the education of students in this country. Finally, she expressed her hope that this panel would help to establish a tradition of service on OERI expert panels.

Sharon Bobbitt presented an overview of the meeting's goals and agenda. She described the meeting as an opportunity to begin a dialogue among panel members, which would facilitate the long-distance communication that would occur throughout the panel's life. She encouraged the panel to make decisions regarding the issues that they would discuss during the meeting, but said that decisions could be saved for the next meeting. She encouraged panelists to submit comments on the standards published in the Federal Register by August 2nd. Although this is one of two inaugural panels, from which OERI hopes to learn valuable lessons for future panels, it is a functioning panel that will be conducting reviews in the upcoming year.

As part of their introductions, panelists described what they considered to be the most serious challenges facing mathematics and science practitioners and the opportunities they saw for the panel to contribute to improving mathematics and science education. They cited a number of challenges:

the isolation of practitioners;

the lack of ongoing, substantive professional interaction among professionals in the field;

the misalignment of goals and incentives of various players (e.g., teachers, principals, the local education agencies, and the state);

financial and time constraints (e.g., to maintain school buildings, implement new programs, work with colleagues, or participate in professional development activities);

external forces that mitigate against teachers (e.g., the transience of leadership);

implementing the "vision" of mathematics and science standards;

understanding performance-based education and assessment;

equity and parity issues and addressing the needs of all students (college-bound, work- bound, specials needs) especially with technology;

misconceptions about how students learn; and

the public's lack of understanding of the relationship between expected teacher performance and professional salary scales.

Panelists suggested the following ways that they could contribute to improving math and science education in this country:

supporting interactive relationships among different sectors of the mathematics and science education community (e.g., researchers, practitioners, mathematicians and scientists, and ED);

illustrating current trends and practices in mathematics and science education;

providing mathematics and science educators across the country, especially those in urban or rural areas, with a range of ideas and materials from which to choose;

illustrating shifts in the preparation of teachers and other professional educators;

shaping public policy and future direction; and

helping to align the mathematics and science standards with assessments (i.e., the incentive system for classroom teachers).

Carol Muscara then presented an overview of her research on Building on Review Efforts in Mathematics and Science Education. She has interviewed 20 organizations that review preK–12 mathematics, science, and technology products and resources, including foundations, non-profit groups, professional societies, states, and regional laboratories. Through telephone surveys with each organization, she has learned about their review processes and the specific criteria that they use. The final paper will be available for panelists upon completion.

The following is a summary of the issues that the panel discussed, organized by the issues that were sent to panel members before the meeting. The issues are divided into three broad categories: (1) panel review process — designation as promising or exemplary; (2) panel review process — the editorial board model; and (3) programs selection and links with the field and the public. The decisions made and remaining questions are listed after the discussion of each issue.

Panel review process — designation as promising or exemplary

Overview of issues. There are a number of different issues to consider regarding the process of designating programs as promising or exemplary. These include: the evaluation criteria, the evaluation form, the nature of comments given to program developers, and how to track the status of programs that have been reviewed.

Panelists discussed at length the criteria for designating programs as promising or exemplary. Most of the discussion focused on the criteria, how criteria would apply to different types of programs, and the difference between promising and exemplary programs.

Issue 1: One of the most critical issues for the panel to discuss is how to adapt evaluation criteria for designating programs as promising or exemplary to mathematics and science education. The current criteria for designating programs as promising or exemplary were published in the Federal Register for public comments that are due by August 2nd. Are there additional criteria that the panel wants to use in designating programs as promising or exemplary? Are the proposed criteria framed well for evaluating mathematics and science programs?

Prior to the meeting, panelists received a copy of the Federal Register, dated June 3, 1996, with Standards for Conduct and Evaluation of Activities Carried Out by the Office of Educational Research and Improvement (OERI) C Designation of Exemplary and Promising Programs, which outlined the proposed criteria for evaluating the programs submitted to the panel (page 27993). Sharon Bobbitt asked panelists to suggest changes to the proposed criteria or to specify additional criteria for evaluating mathematics and science programs.

Student performance

The panel discussed at length whether to add specific language about providing evidence of improved student performance, which one panelist suggested phrasing "well measured, significant, reliable growth in student learning." As written in the Federal Register, the standards do not specifically require evidence of improved student performance. Several panel members considered evidence of improved performance, especially learning, the most important criterion for designating programs as promising or exemplary; however, not all panelists agreed. Several expressed concern that it would be difficult to measure improved student performance, particularly when applied to research findings or policies.

Panelists next discussed what evidence would constitute proof of improved student performance. They seemed to agree that they should not operate as the Program Effectiveness Program (PEP) had, with a very narrow definition of evidence of improved performance. PEP accepted only a few performance measures; developers who used lesser-known tests as evidence had a much larger burden of proving that their programs worked. The panel seemed to favor a broader definition of acceptable measures of performance to ensure that developers who lacked statistical expertise would not be excluded, as they had been with PEP.

The panel suggested that they permit measurements other than increased standardized test scores. Panelists suggested looking at data on student attitude or at student portfolios to gain a sense of the program's impact on student performance. One panel member suggested that the best judge of student achievement is the teacher, and while a teacher's gut feeling may not typically be regarded as evidence, it should not be ignored.

Other criteria

The panel then discussed other criteria for judging programs promising or exemplary, such as whether programs designated as promising or exemplary should align with the NCTM mathematics and science standards. Several panelists pointed out that practitioners would use only programs that matched the standards and requirements in their states; however, since many states were beginning to align their standards with the national standards, using the national standards would be a good start.

Other suggestions for changes or additions to the current criteria included:

requiring evidence of teacher preparation, where appropriate;

ensuring that the standards are not just content-based, rather that they reflect student assessment, how the program fits into the overall system, etc.;

reordering the criteria so that "evidence of success" is not first;

changing the word "problems" in "the program contributes to solving substantial or important problems in teaching or learning", because some programs may not address specific problems, they may simply provide a better approach; and

removing the criterion "program performs as expected" because it would be difficult to provide evidence to support this claim.

Promising vs. exemplary

Following the discussion of evidence of success, the panel debated the difference between "promising" and "exemplary" programs. The current standards define promising programs as those meeting the criteria with respect to one context or population, while exemplary programs meet the criteria with respect to multiple contexts or populations. Several panelists took issue with this distinction — one thought it sounded as if "exemplary" applied to programs that work, while "promising" applied to programs that might work. He felt that both designations should refer to programs that work and suggested eliminating the designation of promising and having instead a continuum of exemplary. Another panelist added that a program can be neither promising nor exemplary if there is no measured change in student learning.

Other suggestions for the distinction between promising and exemplary programs were that promising programs meet the criteria either for multiple contexts or for multiple populations, while exemplary programs meet the criteria for both, or that promising programs should require less rigorous proof than exemplary programs.

Remaining questions

How will the panel address the issue of "evidence of increased student performance" as a criterion for promising and exemplary programs? What measurement will constitute evidence for this criterion?

Should the panel consider adding other criteria in addition to those laid out by OERI?

How will the panel distinguish between promising and exemplary programs?

Issue 2: The second packet of information that panel members received contains a draft of an evaluation form for this panel. The questions reflect the criteria published in the Federal Register. What revisions should be made to the evaluation form?

The panel did not discuss this issue specifically; however, the discussion under Issue 1 is related.

Issue 3: What comments should the panel give to developers (i.e., the extent, the format)? Will the panel provide suggestions for improvement to developers whose programs were not designated as promising or exemplary?

The panel did not discuss this issue directly, but discussed the issue of providing technical assistance to developers whose programs were not designated as promising or exemplary.

Issue 4: Who will prepare the comments for developers? What form will they take?

The panel did not discuss this issue.

Issue 5: The contractor will be responsible for maintaining a database of programs that have been reviewed and the results of panel deliberations. What will be the process for communication between the panel and the contractor so that the data base is accurate and up-to-date?

The panel did not discuss this issue.

Panel review process — editorial board model

Overview of issues. OERI has suggested that the panel operate based on an editorial board model, with three panelists reviewing each submission. Several issues arise with respect to this model. Because panelists are located throughout the country, it will be necessary to establish channels for communication. The contractor will play an active role in facilitating communication throughout the panel process. In addition to the panel members, a set of secondary reviewers, or subpanelists, may be identified to assist in the review process. What will be the composition of review teams? Finally, developers will want to know the time frame for the review process. The Program Effectiveness Panel, which preceded this panel, completed reviews in 6-8 weeks.

The panel discussed the "editorial board model" of reviewing submissions, which would involve distributing submissions among teams of reviewers. Teams likely would include several panelists and several secondary reviewers, who are not panelists. The panel spent time discussing the composition of review teams and whether programs should go through different stages of review (e.g., the panel, then practitioners, then mathematicians or scientists).

Issue 6: The panel should identify a chairperson. What will be the chairperson's responsibilities? Will the panel identify other roles, such as a secretary? Who will be responsible for assigning submissions to the panel members to review?

The panel did not identify a chairperson at the meeting. Sharon Bobbitt asked panelists to think about what they thought should be the duties and responsibilities of the panel chair.

Remaining questions

What will be the duties and responsibilities of the panel chair?

Issue 7: OERI proposes that programs will be reviewed by teams of three panel members who will present the program and their recommendation to the entire panel. How (and when) will the panel identify the teams of three members? Will these teams change throughout the review process? What will be the channels of communication within and among the teams of three? If the three panelists disagree, the program will be discussed by the entire panel. Must the decision for designation as promising or exemplary be unanimous? Are two votes sufficient?

Panelists discussed how many reviewers would be necessary for each team; one suggestion was that the teams include two panel members and three to four others. Another suggested that there be as many reviewers as possible, depending on the volume of submissions. (Although there is no way to predict the volume for this panel, the PEP received between 20 and 30 submissions per year.)

The panel discussed the optimal composition of review teams, suggesting that the teams include a mix of classroom-level practitioners, mathematicians or scientists, learning psychologists, and "others", who may include district- or school-level supervisors, coordinators, or change agents. Further, they felt that the review team should represent various district contexts (e.g., rural and urban, a geographic mix, etc.) to ensure that teams can address the "multiple context" and "multiple population" criteria. Another approach would be for the panelists to pre-screen submissions, then send selected ones to other practicing teachers, mathematicians, or scientists to review teacher "friendliness" and content.

The panel noted that the composition of the review teams would depend upon the content of each submission, making it necessary for someone to read the submissions before assembling an appropriate review team. Sharon Bobbitt indicated that OERI (or the contractor) could be involved in this process, but suggested that she did not want to take any involvement away from the panel itself. It was suggested that the panel chair be responsible for this step.

Decisions made

The panel decided to operate on an editorial board model, with teams of panelists and subpanelists reviewing submissions and making recommendations to the whole panel.

Remaining questions

How many reviewers should be on a team?

What should be the composition of the review teams?

Who will select the review team, OERI, the panel chair, or another panel member?

Issue 8: The contractor will be responsible for maintaining a database of subpanelists that will include each individual's area(s) of expertise. What will be the panel's role in identifying individuals who may be included on this list? Must the subpanelists be approved by the Secretary? What will be the chain of communication for submitting names of potential subpanelists?

PRI will maintain a data base of secondary reviewers. Panel members thought that this data base should be as large as possible, to provide panelists with the greatest selection and expertise for reviewing programs. Also, the data base should contain as much information as possible about the backgrounds of potential secondary reviewers (for example, it would be important to know whether any of the math or science teachers also have expertise in early childhood education). The data base should also contain information about each reviewer's school district (e.g., urban or rural) to ensure a geographic and demographic representation. Finding a representative team of reviewers was considered important for assessing the "multiple context" aspect of programs.

Issue 9: How should external reviewers be used most efficiently in the process?

See discussion under Issue 7.

Issue 10: What should be the time frame for reviewing each submission?

The panel did not discuss this issue.

Program selection and links with the field and public

Overview of issues. The third set of issues relates to the panel's relationship with the field and the public — both in terms of soliciting programs (i.e., the types of programs to solicit and the process of solicitation) and in terms of disseminating information about programs that have been designated promising or exemplary. OERI proposed that the panel focus on K–4 mathematics and science education in the pilot year. Within that broad category, however, the panel may decide to narrow the focus of the submissions that it will review. Panel members will play an active role in identifying programs for consideration. Members may identify programs through various methods, such as using their personal contacts. With respect to pre-screening, ED will be able to screen submissions for completeness before distributing them to the panel, but not along any other parameters. Another issue related to links with the field is the issue of outside pressure. Because of the potential implications for marketing that may arise from designation as promising or exemplary, panel members may find that individuals or organizations are exerting pressure to have programs designated as promising or exemplary. Dissemination is the final aspect of linking to the field. One primary goal of the Expert Panel in Mathematics and Science Education is to promote research-based practice. To do so effectively, information about designated programs must be accessible to practitioners.

The panel spent a significant amount of time discussing whether to consider all four categories of educational programs identified in the legislation — policies, research findings, practices, and products. The panel discussed the definitions of several of these categories, and in particular was confused about how to apply the criteria proposed in the Federal Register to research findings. The last discussion of the day focused on dissemination efforts.

Issue 11: Programs may fall into different categories: classroom materials, teacher development, pedagogical strategies, policies, research findings, curriculum design, and assessments to name a few. What categories of programs should the panel focus on in its first year?

See discussion under Issue 12.

Issue 12: Programs also vary in scope, with some spanning a number of years or grade levels and others having more narrowly defined dimensions (e.g., a two-week unit). Some programs may be geared toward specific parts of the country. What is the scope of programs that the panel will consider in the first year?

The expert panels are mandated to review policies, research findings, practices, and products, collectively referred to in the Federal Register as "educational programs". The panel discussed what precisely was meant by "policies"; one panelist suggested that a policy was "a direction of action that allows us to make future decisions based on that direction and that lets you reach that direction." Panel members thought that it would be difficult to apply the criteria for promising and exemplary to research findings. The panel discussed having one of the commissioned paper authors prepare a paper that describes what the panel would review in determining whether policies and research findings were promising or exemplary.

OERI had suggested limiting the types of educational programs that the panel would review in its initial year to K–4 practices and products. The panel discussed whether any limitations should be placed either on type of program reviewed (e.g., not research findings or policies) or on the grade-level (e.g., only K–4). Sharon Bobbitt suggested that they had three options: (1) limit the type and grade-level; (2) do not set limitations and see what types of submissions the panel receives; or (3) do not set limitations and have panelists actively and personally seek submissions.

One panelist pointed out that part of the panel's task in the initial year was to figure out the process — which includes how to handle the volume of educational programs that are submitted. The panelist suggested that they not limit the number of submissions so they could figure out this part of the process. Other panelists were concerned that limiting the programs considered in the first year to practices and products would set a precedent for future years, thereby limiting the number of policies and research findings submitted.

Panelists discussed whether programs with which panelists had been involved could be considered. The consensus seemed to be that panelists' own research efforts should not be hindered by their participation on the panel, and that they simply would not participate in reviewing programs with which they had any involvement.

Decisions made

The panel will review all types of K–4 educational programs in its first year. Panelists may suggest programs for the panel to review, but will not personally solicit submissions.

Panelists will not review educational programs that they have been involved with; however, these programs can be considered by the rest of the panel.

Issue 13: What process will panel members use to identify programs and invite developers to submit their programs?

Although the panel decided not to personally solicit programs in the first year, panel members discussed ideas for soliciting programs. One suggestion was that when panelists or OERI staff learn about potential educational programs at national meetings or conferences, they would inform the developer of the purposes of the panel and suggest that they submit their program. OERI is considering developing a brochure that describes the goals of the panel and the submission process. These brochures could be made available at national meetings, or would be sent to developers upon request. Other suggestions included sending information to various education associations, putting information on the OERI web site, and printing the information in magazines and journals.

The panel discussed whether developers would feel reluctant to submit their programs and risk rejection. There seemed to be consensus that, since the panel would not make public which programs were not designated as either promising or exemplary, this should not be a problem.

Issue 14: Will the panel consider programs in addition to those formally submitted? For example, should programs that have been identified by other Federal agencies or have received favorable reviews in professional journals be designated as promising or exemplary without further panel action?

The panel did not discuss this issue.

Issue 15: Should any program with sufficient documentation be reviewed by the panel? Should the panel select one member to be in charge of pre-screening?

The panel discussed having developers submit a two-page pre-submission form; the first page would provide brief answers to a number of pre-screening questions and the second page would allow the developer to provide a brief description of the program. Panelists could then decide to seek additional information regarding certain programs. This two-page format would allow panelists to provide technical assistance to program developers in completing their submission forms, which is one of the panel's responsibilities. The panel discussed the submission form that would be used. One panel member wanted to ensure that the submission form state the requirements very clearly, for the benefit of both the developer and the reviewer.

Decisions made

There will be a two-page pre-submission form as described above.

Issue 16: How will the panel handle pressure or lobbying from others for promising or exemplary status?

The panel did not discuss the ethics of handling lobbying or pressure; it did, however, discuss the issue of confidentiality to avoid pressure by publishers or product developers to designate certain programs as promising or exemplary. One reason for not wanting to remain confidential was that serving on the panel was an honor, not to mention a significant time commitment, and panelists wanted to be recognized for their service. Another reason was that there were many highly recognized and respected individuals on the panel whose names would lend credibility to ED's designation as promising or exemplary.

Decisions made

The names of the panel members will be public, but the specific panelists responsible for reviewing each submission will not.

Issue 17: What copyright or other constraints should be considered in judging programs for the promising and exemplary designations?

The panel did not discuss this issue.

Issue 18: Once promising and exemplary programs have been designated, how will practitioners have access to information? Details of designated programs could be posted on a Department of Education electronic bulletin board (or on other bulletin boards), compiled in a newsletter or pamphlet. What are the most likely audiences for panel- designated programs? What would be the most effective formats for reaching these audiences? Once information regarding the programs is widely available, should practitioners be asked to provide feedback about the programs?

The panel discussed methods of disseminating information about programs that have been designated as promising or exemplary. They agreed that the information must be widely available, but more importantly, it must be accessible. One panelist warned that teachers would use information only if it matched their philosophy about how children learn. Some teachers, for example, believe that children themselves construct understanding and knowledge, while others believe that teachers have a more active role in conveying information to their students. Suggestions for the best format for reaching teachers included flexibility and affordability, and something that they already read.

Panel members wondered whether the dissemination process should be evaluated, to determine not only the size of the audience, but whether practitioners actually had incorporated new programs into their classrooms. Panelists were reminded, however, that the main objective of the panel is to make information available to as many practitioners as possible, not necessarily to have as many practitioners as possible using the designated programs in their classrooms.

Panelists agreed that the review process could only go so far to improve education. The panel can identify materials with the potential to enhance learning, but it cannot guarantee results. The panel has no control over individual teachers' use of the designated educational programs.

Remaining questions

Should the dissemination efforts be evaluated?

Summary of decisions

In summary, the panel made the following decisions:

The panel decided to operate on an editorial board model, with teams of panelists and subpanelists reviewing submissions and making recommendations to the whole panel.

The panel will review all types of K–4 educational programs in its first year (i.e., policies, research findings, practices, and products). Panel members may suggest programs for the panel to review, but will not personally solicit submissions.

Panelists will not review educational programs that they personally have been involved with; however, these programs can be considered by the rest of the panel.

Developers will be required to complete a two-page pre-submission form before completing a full submissions. The first page of this form will answer several pre-screening questions and the second page will describe the program.

The names of the panel members will be public, but the specific panelists responsible for reviewing each submission will not.

Activities since last meeting

Several activities have taken place since the first panel meeting:

Next meeting date. The next meeting is scheduled for November 15th and 16th and will again be held in Washington, DC, at PRI. Although no time suited all of the panelists, these dates were chosen to accommodate as many panel members as possible.

Panel chair. OERI has asked Steve Leinwand to serve as temporary panel chair. OERI considered it unfair to ask the panel members to select a chair themselves, since many panel members were unable to attend the first meeting and many of the members have just met. Steve Leinwand will serve for six to nine months, until the panel forms itself into a cohesive group and can select its next chair.

Responses to Federal Register. OERI has consolidated the responses to the criteria outlined in the Federal Register. A document that incorporates the changes will be distributed to the panel by the middle of September.

Draft submission guidelines. The guidelines for submitting programs currently are being drafted. They will be distributed to the panel when available.

Commissioned papers.

Carol Muscara is revising her paper Building on Review Efforts in Mathematics and Science Education based on OERI's comments.

Susan Klaiber's paper on applying lessons learned from the Program Effectiveness Panel to the standing panel will be distributed to panel members shortly.

OERI decided to commission a paper on issues of assessment and evaluation; the panel had discussed wanting to know more about these issues.

Electronic mailing list. PRI has begun to put together an electronic mailing list to facilitate panel communication.

Next steps for the panel

The meeting yielded several steps that should take place in the next several weeks:

Begin to develop the data base of secondary reviewers. Panel members should begin to think about who they would like to include in the data base and the information that should be included about each reviewer.

Suggest programs to review during the next meeting. The panel will begin to review submissions at the next panel meeting. PRI will work with suggestions from the panel and from OERI to obtain several submissions for review in October or November.

DRAFT

Expert Panel

in Mathematics and Science Education

Summary of Second Panel Meeting

November 15 and 16, 1996

Introduction

Overview of the meeting. This summary report is not arranged chronologically; it groups the discussion into the five topics that were covered: (1) history and purpose of the expert panels; (2) guiding principles, identified by the panel; (3) where the panel wants to be in one year; (4) presentations by Maria Lopez-Freeman and Jo Ellen Roseman; and (5) discussion of issues and decisions. The following bullets provide a very brief overview of how the meeting proceeded chronologically.

Friday, November 15th

Pat Ross welcomed the group.

Eve Bither described the history and purpose of the expert panels.

Pat Ross reviewed several decisions made during the July 25th meeting.

Panel members described what "success" for the panel would mean for themselves, personally, and for their organizations.

A set of "guiding principles" for the panel was developed and refined.

Maria Lopez-Freeman described the California textbook adoption process.

Jo Ellen Roseman described Project 2061's process of reviewing science programs.

The panel discussed "where do we want to be in one year?"

The panel agreed on the schedule for Saturday's meeting.

Saturday, November 16th

The panel discussed the role of the panel chair and nominated co-chairs.

Small groups walked through the program review process, from calling for submissions to disseminating findings, and reported to the entire group.

The panel discussed the difference between "promising" and "exemplary".

The panel identified possible next meeting dates and next steps.

The panel discussed criteria for designating programs promising or exemplary.

The details of the meeting follow.

History and Purpose of Expert Panels

Eve Bither described the history and purpose of the expert panels. She explained that OERI realized the need for a process to identify and disseminate promising and exemplary programs after receiving myriad phone calls from practitioners seeking recommendations for how to improve their teaching. Although OERI previously had a system in place to help practitioners identify effective programs and practices (the Program Effectiveness Panel), the process was rather narrow, yielding a limited number of programs.

The legislation that authorized the expert panels called for three sets of standards to be developed: (1) for grants; (2) for promising and exemplary programs; and (3) for how OERI monitors funding. The first set of standards currently is in place; the standards for promising and exemplary have been through the public comment phase and should be adopted by the end of January; and the third set of standards will go for public comment on March 30, 1997.

The authorizing legislation gives the OERI Board the authority to set the standards, but emphasizes the use of peer panels to carry out the review process. Given the broad array of subject matter, the Board decided to conduct two pilot panels to develop the process and ensure that the standards were appropriate. The Board chose the fields of mathematics and science because of the plethora of materials available for review. Bither urged the panel to be flexible and not to feel limited by thinking that certain activities — such as conducting site visits — were necessarily constrained by funding. She wants the panel to think broadly about what process would work best.

Guiding Principles for the Panel

The panel members discussed what benefits they would receive — both personally and for their organizations — if the pilot panel is "successful". The panel's responses to these questions yielded the following "Guiding Principles" of the panel.

Guiding Principles of the Expert Panel in Mathematics and Science Education:

There is a need to increase general awareness (on the part of the public and people in the system) to understand what a standards-based math/science reform effort looks like.

The criteria must encompass all evidence about a program, including its history. This implies that the process may be iterative, with reviewers presenting a preliminary review to developers to make sure that they haven't missed important background information.

Professional development for teachers is central to improving student learning; it should be considered in submissions to the panel.

The panel should link ends (outcomes) and means (inputs), and not confuse the two. The panel will likely evaluate inputs (such as curriculum, professional development, etc.) rather than student outcomes, because so few programs have sufficient outcome data.

The panel should consider a "food labeling" approach, in which they provide a set of descriptors that specify the program's purpose, the range of learning outcomes related to the purpose, and to what extent (e.g., "high", "medium" or "low") the program meets its stated purpose.

The panel should identify models of transferable effectiveness, meaning that, where possible, descriptions should include why programs work (e.g., antecedents and context), for whom they work, and under what context(s) do they work. Although it is difficult to measure why things work, this information is necessary for determining transferability.

Submissions should represent a continuum from promising to exemplary.

The panel process and criteria should be "tight", systemic, and defensible; they shouldn't leave the panel open to criticism or question.

The panel should remember that for change to occur, teachers need support and good ideas that are grounded in how students learn. Teachers do not want "flashy" materials; they want materials that make kids excited about learning.

The panel should consider lessons from the business and non-education communities; these groups understand the need for setting standards.

Criteria should specify types of student outcomes and match program goals to outcomes. It is not enough to ask, "is it a good program?"; the panel must ask "does this program do what it intends to do?" The panel should focus on programs to improve student learning.

Submissions should specify which standards are addressed in the program, and the panel's job is to judge how well the standards are addressed.

Where does the panel want to be in one year?

Pat Ross initiated a discussion of where the panel wants to be in one year (e.g., does it expect to identify programs publicly at that time). The panel seemed to agree that it should set going public with identified programs as an internal goal. At the very least, the panel thought that it should have the criteria and process in place by the end of one year.

The panel wanted to reserve flexibility, and seemed to be more comfortable not setting firm goals for the end of year one. It wanted to reserve the option of deciding that the process does not work, should that be the case. One panel member suggested that the panel review its progress every three or four months to allow for mid-course corrections. The panel also decided that it ought to meet more frequently, perhaps every two months, given the vast amount of work that it faced.

Presentations

Maria Lopez-Freeman and Jo Ellen Roseman presented information regarding their respective review efforts to give the panel a sense of different processes and issues. Lopez- Freeman discussed the California textbook adoption process; Roseman described the Project 2061 curriculum review process. Tab 3 of the meeting binder presents information related to these review efforts. In addition, Roseman made available a paper that describes Project 2061's process.

Specific issues

The panel covered a number of specific issues. The discussion — and any decisions made — are described below.

Panel chair. The panel discussed options for selecting panel chairs. One panel member suggested that there be co-chairs to represent math and science, and that at least one chair should be from Washington, to facilitate communications with OERI. Some members expressed their concern about serving as chair, because their organizations develop products that may be reviewed by the panel, and they were concerned about conflict of interest. The panel agreed that if conflict of interest were an issue, it would not matter whether one served as chair or simply as a member. The panel agreed that whoever is panel chair ought to be able to act independently of his or her own interests.

Steve Leinwand and Jim Rutherford were nominated and agreed to serve as co-chairs, representing math and science, respectively. The panel agreed unanimously that they would serve for six months, at which time the panel would repeat the nomination and selection process.

Scope of first year review activities. The panel agreed not to review research findings or policies in the first year, then discussed whether to consider practices (e.g., techniques for questioning students) as well as programs. One panelist commented that anything that improves student learning, whether a program, practice, or other, should be considered, but the panel seemed to agree that limiting their activities to program review would simplify the process. The panel discussed restricting their first year review to K–4 programs, but decided against this idea. The panel agreed to limit reviews to K–12 instructional programs that are at least one year long.

Pat Ross raised the issue of limiting the panel's first-year activities to reviewing exemplary programs. The panel concluded, however, that it would rather limit its review to promising programs, given the likely shortage of programs that would meet the criteria demanded of exemplary. The panel liked the idea of OERI directing resources toward conducting evaluations of promising programs to determine, based on rigorous research, whether any were exemplary. The panel agreed that this approach would allow leeway in refining the criteria, by saving the designations of exemplary for after the criteria were tested and refined. One panelist favored this approach because he did not like the idea of "giving everyone an A."

Panel review process. The panel broke into three small groups on Saturday morning to walk through the review process to determine what will happen at each stage of the review (i.e., from the call for submissions to the dissemination of results.) The three groups discussed the process for an hour, then reported to the whole group. Below is a brief description of the groups' visions; more detailed descriptions appear in appendices. The models that the three groups presented were not necessarily approved by the entire panel; rather, they were offered to start a discussion of what the process should look like.

Group I decided to set internal deadlines for the next year, then fill in the specific activities that should be accomplished in order to meet the deadlines. The group produced a flow chart of activities and goals, to which they added activities generated by the other groups. A version of this flow chart appears in Appendix 2.

Group II envisioned the process as it would occur after the pilot year is over, with a system of panels (including technical review panels and one or two oversight panels). This "vision" for the system of panels is presented in Appendix 3.

Group III envisioned what the end result of the review process would be, deciding that it would include a brief description of the program, findings of the panel, and a profile of how the program matched the criteria. In envisioning this process, Group 3 raised a number of questions, as outlined in Appendix 4.

Dissemination format. The panel decided that the product that is disseminated should provide sufficient information (e.g., cost of implementing the program, professional development required, etc.) for consumers to make informed decisions about whether to adopt the program. The panel discussed perhaps putting together brochures. Someone suggested that a design firm should be hired to determine the best way of presenting information for the audience.

Criteria for designating programs promising or exemplary. The panel discussed the current criteria for designating programs promising or exemplary, as written in the regulations. Most agreed that the four broad categories (i.e., evidence of success, quality of the program, educational significance, and usefulness to others) were appropriate, but that the specific criteria under each category were not useful. For example, the panel did not want to require that programs meet the criterion "the cost of the program…is reasonable in light of expected benefits and compared to other alternatives…", which implies that each submission would be required to provide a cost-benefit analysis of its own program, as well as alternative programs. The panel agreed that it was unreasonable to expect program developers to evaluate other programs.

The panel decided that it would be a useful exercise for each member to draft criteria for designating promising or exemplary K–12 core instructional programs. The panel agreed to specify core programs, or programs that are designed for all students, not just college-
bound students. One panel member suggested that each member consider the most exemplary program that he or she knows about, and draft the criteria based on that program.

Definitions of "promising" and "exemplary". Sharon Bobbitt asked the panel to discuss the options for defining "promising" and "exemplary" that were distributed via listserv early in November so she could make a recommendation to the OERI Board. Although there was some confusion about the distinction between the four options, the panel seemed to agree that both promising and exemplary programs should meet three of the criteria equally (quality, educational significance, and usefulness to others) and that the distinction should lie in the evidence of effectiveness criterion. "Promising" programs should demonstrate effectiveness on one or two sites, while "exemplary" programs should demonstrate effectiveness in three or more.

Although the panel members preferred the above definition to any other, they generally felt uncomfortable with a "one-word" designation. The panel would prefer a more comprehensive description of the programs, which would allow potential users to select programs that were aligned with their own needs.

The panel discussed the "shelf-life" of designated programs. That is, once a program is designated promising or exemplary, would its designation be reviewed later, say, after five years? The panel did not make a recommendation regarding this issue.

Use the national standards as a filter for review. One panelist expressed concern for using national standards as a filter, because they may be misinterpreted by practitioners. For example, some math teachers may interpret the NCTM standards as suggesting that students use technology at the expense of not learning to prove theorems. This concern was countered by the suggestion that the panel can influence how practitioners apply the standards by articulating, through the criteria, what they really mean. Another panel member responded that millions of dollars and many years were spent developing the national standards, and that this panel would not be able to improve upon them given its available time and resources. Finally, one panelist suggested that national standards are important for providing states a vision of excellence in education, without which some states might set lower expectations.

Focus on student learning outcomes. The panel agreed to emphasize improved learning in the criteria. One panel member wanted to focus on student understanding as a learning outcome, rather than simply memorization or recall, but recognized that little research on measuring student understanding exists. The panel also discussed which inputs indirect (i.e., professional development) may have an impact on student learning.

Site visits. One panel member recommended that they consider conducting site visits before making determinations about designating programs, claiming that this step need not be prohibitively costly. Some panelists were concerned that sites would "clean up their act", presenting a biased image to the reviewers, but others countered that site visits would be conducted only if programs met the criteria, so that people conducting the site visits would already have a grasp of the program's quality.

Professional development programs. The panel raised several issues about professional development. Some thought that programs should include professional development components in order to be designated as promising or exemplary; others thought that professional development could be considered part of the context, but not an essential part of the program.

The panel discussed whether to consider models of professional development as programs, and seemed to agree to remain focused on student learning in the first year, before expanding to include professional development programs. Some thought that it would be difficult to find professional development programs with evidence of improved student learning, since there is no established list of knowledge or skills to test the effectiveness of professional development programs. Others countered that it would be possible to determine whether professional development for teachers reasonably increased a student's "opportunity to learn", in which case the panel could measure whether a professional development program had increased this opportunity to learn.

Directing funds to evaluate promising programs. The panel agreed that it would be extremely difficult for most developers to demonstrate why their programs work. The panel therefore wanted to recommend to the OERI Board that OERI favor field studies for the programs that have been identified as showing promise, to validate the belief that the program works, as well as to find out why it works. The panel discussed the reciprocal of this idea; that is, that OERI provide funding for the implementation of programs that are grounded in research, but have not yet been tested in practice. A discussion about research and practice ensued. The panel agreed that good programs evolve from a cycle of research and practice.

Other uses of "promising". One panelist noted that the word "promising" is used in other review efforts (e.g., those published by the Regional Consortia) for programs that do not necessarily meet the standards set by OERI in its regulations. What will happen to programs identified as "promising" by others? The panel agreed that they would not automatically designate these programs promising, but that they would be a good starting point for soliciting submissions.

Next meetings. The panelists agreed that more frequent meetings would be necessary if they were going to identify criteria. Two potential dates for each of the next three meetings were identified. PRC was to email the dates to panelists and set the dates as soon as possible.

Other ideas or issues. Throughout the course of discussion, the panel raised a number of questions or made statements that either were not resolved or were not fully discussed. Following is a list of these issues that the panel may want to keep in mind for discussion at future meetings.

The submission form should have sufficient information to allow the program to be included in the Eisenhower Clearinghouse.

Rather than using grade levels, the panel may consider using grade bands or grade clusters as specified in the National Standards documents.

Who is the audience? Teachers? Administrators? School board? Does the audience matter for the format and type of information disseminated about programs?

Will cost of the program be a factor? It was suggested that cost should not be one of the criteria, but that perhaps it should be reported in the description of the program so potential users can make informed decisions about whether to adopt the program.

The panel should consider whether it would benefit from additional commissioned papers; if so, they should notify Pat Ross.

Program Submission

Group II began its discussion of the review process by examining what should be required for program submission to OERI. After determining who should be eligible for consideration, OERI should advertise a "call for submissions". The group decided that a program that would like to be designated promising or exemplary should respond to the call for submissions with an abstract of the program. This "pre-proposal" should include:

a statement of the eligibility of the program;

a narrative description of the content of the program;

a review of the effectiveness of the program;

a detail of the program's adherence to national standards and learning goals.

However, the group was concerned with the ambiguity of some of the above terms. Specifically, the group felt that the word, "program," needs to be defined more succinctly. Questions arose whether a program could simply be an intended curriculum. In addition, the group noted that "national standards" and "effectiveness" could be interpreted with considerable latitude.

Structure for Review

The group then developed a potential structure for reviewing submissions. Each submission will be sent to one central point of contact in OERI. This Technical Review Manager will verify that all eligibility criteria are met, and that the submission is complete; if there are questions about either eligibility or completeness, then the submission is returned to the program developer for clarification. Otherwise, the programs will be entered into the Eisenhower database.

The Technical Review Manager will disseminate the eligible, complete submission to two independent Technical Review Panels, whose members will have extensive training and certification. These two panels will discuss the merits of the program, as well the likelihood that the program will achieve all learning goals. Each panel will create a profile of the submission and submit it to the Technical Review Manager, who will forward the two profiles to the Oversight Panel.

The Oversight Panel will review the profiles from the Technical Review Panels to identify any discrepancies in the panels' assessments of the submission. If considerable discrepancies exist, then the Oversight Panel will return the profiles (with comments) to the Technical Review Manager, who will disperse them again to the Technical Review Panels for a second review. After discussion, each Technical Review Panel will then re-submit an updated profile to the Oversight Panel.

The Oversight Panel will then forward their recommendations, based on the profiles from the Technical Review Panels, to the Standing Panel, who will evaluate and refine the recommendations. The Standing Panel will make a final recommendation to the Secretary of Education.

The attached exhibit illustrates this process.

Other Issues

The group raised several concerns about the administration of this review process. First, it was skeptical of the time-effectiveness of sending each submission through multiple panels. If this were to become a burden, then it may be necessary to restrict the process to only one Technical Review Panel. Second, the group was concerned about the relationship of the Technical Review Panel and the Oversight Panel. For example, if the Oversight Panel has the authority to preempt any decisions made by the Technical Review Panels, then members of the Technical Review Panels may feel that they do not have a real "place" in the process. Finally, the group decided that there should be a series of checks and balances built into the structure to ensure that the process is followed.

APPENDIX 4:

Group III's Vision of the Process

Group 3 began its discussion by envisioning the end result of the review process. They concluded that the end result, the designation of promising and exemplary programs, would include a brief description of the program, the panel's findings, and a profile of how the program matched the criteria and its goals (perhaps similar to the profiles that Project 2061 uses). To achieve the end result, the group identified six main stages of the process:

call for submissions;

arrival of submissions;

submissions screened and assigned to subpanels;

subpanels review and make recommendations to full panel;

panel discusses and makes final recommendation to the Secretary; and

feedback or resubmission process.

Within each stage, the group raised a number of issues, as outlined below.

Call for submissions. How should the panel solicit submissions? What would be the format for submissions.

Submission packages arrive. The submission package should include background (descriptive) materials, an application, and "evidence" of effectiveness. Should anything else be included in the package?

Submissions screened/assigned to subpanels. There should be a pool of subpanelists whose characteristics are described in a matrix. Each program would be assigned to a subpanel with appropriate knowledge for that program. The panel may want to seek nominations for the pool of subpanelists from the professional organizations. Where else could the panel solicit nominations for subpanelists? Who should screen the programs before sending them to subpanels? OERI? A contractor? Panel members?

The group discussed the composition of the subpanels, agreeing that they should include, at least teachers, administrators, and one person from the panel. The group proposed that four people serve on each panel (i.e., three secondary reviewers and one panel member) and that there should be majority approval of a program before recommending it to the full panel.

Subpanels review and make recommendations to full panel. The group agreed that training was an essential element of the review process, and raised several questions about training. What would the training entail? When would training take place? Who would conduct the training? The group decided that training should illustrate each criteria with a concrete example.

After the subpanelists have been trained, where would the review take place? The group agreed that the subpanelists would review programs on their own at home, then convene to discuss their recommendations. The issue of communication via electronic mail was raised, but the group concluded that not all teachers on the subpanels would have access to electronic mail.

The group touched on some criteria that they may want to consider. One member suggested that they should consider cost, and the group concurred that cost should not be in the criteria, but may be reported in the project descriptions.

Panel confirms/makes recommendation to Secretary. The group focused on the question of who has the ultimate authority of recommendation. Would the panels overturn the subpanels' recommendations? Would the Secretary veto the panel's recommendations? The group strongly agreed that the subpanels should have ultimate authority.

In considering the dissemination process, the group talked about who the audience was for the information. Is it teachers, administrators, school boards, or others? Who decides which programs to adopt and what influenced those decisions? The group seemed to agree that, since the materials adoption process varies from state to state, they should not focus one specific audience, rather they should produce information that would assist decision makers at all levels.

Feedback and resubmission process. The group wondered whether the "winning" developers should be recognized for their contributions, as are the Blue Ribbon Schools designees. Sue Klein offered that the purpose of the expert panels was not so much to recognize the developers as it was to share good information with the field, and discouraged a recognition ceremony from being part of this process.

Expert Panel

in Mathematics and Science Education

Summary of Third Panel Meeting

January 27 and 28, 1997

Expert Panel in Mathematics and Science Education:

Summary of Third Panel Meeting

Introduction

Objectives and Overview of the Meeting

The panel has agreed to implement a trial program review during the Summer of 1997. In order to do so, the panel must define the process — including the evaluation criteria — by which it will review programs. The meeting provided panel members the opportunity to work together to revise the draft criteria and to discuss details of the review process. Prior to the meeting, panel members received discussion materials that included a compilation of the evaluation criteria submitted by panel members after the November meeting, a synthesis of those evaluation criteria prepared by Jim Rutherford and Jo Ellen Roseman, and an outline of a potential panel review process, prepared by Pelavin staff.

The first day of the meeting focused on the evaluation criteria. Sharon Bobbitt opened the meeting by describing the status of the OERI regulations and criteria. The panel discussed the evaluation criteria as a group before dividing into three groups to refine the criteria. After lunch, the groups presented their criteria for discussion. At the end of the day, Steve Leinwand, Jim Minstrell and Steve Schneider, each representing one of the small groups, met to incorporate the ideas of all three groups into a revised set of criteria. This version of the criteria was distributed on the second day of the meeting for further refinement. Pelavin staff edited and formatted the criteria, which appear in Appendix 1.

On the second day of the meeting, the panel worked through the details of the various stages of the review process, which broadly include: publicizing the process, soliciting program submissions, selecting and training program reviewers, distributing submission packets to the appropriate review panels, reviewing program submissions, preparing reports and recommendations, making recommendations to the Secretary of Education, and disseminating the results.

Evaluation Criteria

The panel divided into three groups to work on the details of the criteria, then reported their ideas back to the panel for discussion. The panel co-chairs focused the discussion on broad conceptual issues, rather than specific wording of the criteria, since the panel agreed that the "wordsmithing" could be done in a later version. Following are the issues that the panel discussed, as well as any decisions that the panel made. The panel believes that the version of the criteria attached as Appendix 1 reflects their discussion.

Teacher Training. The panel discussed whether to include the criterion "teacher training is relatively simple" under the Usefulness to Others category. One panel member pointed out that training is often very complex, particularly with more innovative programs, and expressed concern that requiring "simple" teacher training may rule out innovative programs, leaving more traditional, textbook-centered programs. Panel members also discussed the fact that what seems "simple" to some (i.e., a week-long program) may not be simple for teachers who have little free time.

The panel decided not to mention teacher training in the revised version of the criteria.

Equity for "All Students". The panel discussed whether the criteria should use the phrase "all students" (e.g., "all students succeed at high levels as a result of the program" under the category Evidence of Success), given that some programs are targeted toward a specific population, and are not intended for all students. The panel also discussed whether equity issues should fall under the Evidence of Success category or the Educational Significance category. The panel agreed that the criteria should define success in terms of the specific goals of the program.

The revised version of the criteria addresses equity under the Educational Significance category, and requires evidence that the program targets either "all students" or "underserved students" or "students whose interests and talents go beyond the basics."

Cost Effectiveness. The original proposed OERI regulations would require submittors to provide evidence that their programs are cost-effective relative to similar programs. The panel agreed that requiring developers to perform cost-benefit analyses of their own programs would be significantly burdensome, and that asking them to perform cost-benefit analysis of other programs would be out of the question. The panel agreed that it would be sufficient for developers to provide an estimate of the cost of implementing the program in their submission materials, so that others could decide for themselves whether they considered the program worth implementing.

The revised criteria, therefore, require that the program "specifies the conditions and resources needed for implementation."

Improved Student Attitudes. The panel seemed to favor the idea of looking at changes in student attitudes towards math or science as an outcome measure, to prevent recommending programs that increased student scores but caused the students to hate math or science. However, the panel agreed that student attitudes were difficult to measure. The panel discussed using enrollment patterns and persistence as proxies for student attitudes, since both are measurable, but concluded that documenting changes in enrollment or persistence would require longitudinal study, which most submittors would not be able to provide. One panel member wondered whether it would be possible to do 5-year follow-up studies of program success.

The revised criteria do not mention student attitudes or changes in persistence or enrollment.

Panel Review Process

The panel walked through the steps of the review process, following a diagram sent prior to the meeting. The diagram on the following page illustrates the process to which the panel tentatively agreed during the meeting. Under this model, programs first would be reviewed by two "Quality Review Panels", which would examine the criteria under three of the four broad categories: Educational Significance, Quality of the Program, and Usefulness to Others (not Evidence of Success). Programs meeting the three broad categories of criteria would be considered for designation as either Promising or Exemplary; programs not meeting the criteria under these three categories would not be eligible for any designation, and would be returned to developers with feedback. The panel did not discuss the level of feedback that they would provide submittors at this point in the process.

The Quality Review Panels each would prepare a report and recommendations for every program meeting the criteria under the three broad categories listed above. After the Quality Review Panels review the programs, those programs that do not provide data to demonstrate Evidence of Success would be sent directly to the Expert Panel for consideration as Promising. Programs that provide data to demonstrate evidence of Success would be reviewed by a second panel, the Impact Review Panel, for consideration as Exemplary. The Impact Review Panel would send its reports and recommendations, along with the information from the Quality Review Panel, to the Expert Panel for consideration as Exemplary. After the Expert Panel reviews the reports and recommendations from these panels, it would forward its recommendation on to the Secretary of Education.

Details and discussion of each step follow.

Call for Submissions. The first step of the program review cycle would be to publicize the panel and the process. The panel discussed using the Internet as one vehicle for reaching potential program submittors, but one panel member cautioned that, since many teachers are not connected to the Internet, this should not be its only means of communication. Other methods of spreading the word to potential submittors would include announcements in Education Week or other journals and making presentations and providing written information at professional meetings.

Program Submission. The second step of the process would be program submission. The panel discussed whether potential submittors would be required to complete a preliminary submission form in addition to a complete submission form and, if so, what type of information to request at each stage. Proponents of pre-submission thought that, since developers would contact OERI to request the guidelines for submission anyway, OERI could use that contact to begin collecting information about the programs. Others felt that a pre-submission step was unnecessary.

In discussing what kind of information would be requested on the preliminary submission, the panel discussed how OERI would use the information. The issue of using the pre-submission information as a screening device was raised; however, the panel agreed that it did not feel comfortable with anyone (i.e., OERI or a contractor) screening programs based on a small amount of information. In addition, since the panel would not provide submittors with feedback based on the pre-submission, several panelists argued that it was not worth requesting this information. The panel agreed that there would not be a preliminary submission, but rather the complete submission form would include a one- to two-page summary or cover sheet.

The panel agreed that the evaluation criteria should be very clearly stated in any submission guidelines to ensure that potential program submittors have a solid understanding of the criteria against which their programs will be reviewed. The panel wanted to ensure that program developers do not waste their time developing submission packages for a designation that they have no chance of attaining. The panel hoped that an added benefit of presenting the criteria very clearly would be that teachers who have programs that fall short of the criteria may use the criteria as guidelines for strengthening their programs.

Panel Allocation Process. The panel briefly discussed the process by which programs would be distributed to review panels, but did not come to any conclusions about this step.

Quality Review Panels, Impact Panel and Expert Panel. The panel has agreed that, although its members will be involved in reviewing programs during the design phase, they will not actively review all of the programs that are submitted once the system is fully operational. The panel discussed the best way to incorporate a second level of review panels in the process, as well as the size and composition of the various panels.

The panel discussed how many Quality Review Panels there should be. Although most agreed that two review panels were necessary to increase validity, not all agreed that they both should have the same emphasis, suggesting instead that there be one panel to review the content of the program and another to review the evaluation component of the submission. Another panel member disagreed, saying that this would pull the program apart artificially, and that each Quality Review Panel should include both evaluators and math/science educators. The panel finally agreed that two Quality Review Panels would be necessary during the design phase, to determine the extent to which the recommendations of two panels differ. The Quality Review Panel members would have expertise specific to the program's content. The panel discussed having one Impact Review Panel. Since Impact Review Panel members would evaluate the evidence of success data, they would not necessarily have content-specific expertise, but would be evaluation experts.

The panel discussed the size of the panels. Although no decision was made, the panel seemed to agree that there should be at least three individuals on the Quality Review Panels and the Impact Review Panel. The panel thought that the Expert Panel should be large enough so that one Expert Panel member could serve on each of the Quality Review Panels. It also thought that a larger panel would carry more clout than a smaller panel. One panel member suggested that each of the Expert Panel members should have experience reviewing programs. Regardless of the size or composition, however, the panel agreed that the review panels would require significant training before they could review any programs.

Dissemination to the Public. In addition to disseminating information regarding promising and exemplary programs, the panel discussed publicizing information regarding all of the programs that were submitted for review, not just those that they designated as promising or exemplary. The panel seemed to agree that any program with a solid description of its learning goals and strategies should be included in the Eisenhower Database.

Other Outstanding Issues

In addition to discussing the evaluation criteria and the review process, a number of other issues arose during the meeting.

Difference between Promising and Exemplary Programs. The panel revisited the issue of the difference between "promising" and "exemplary". There was some disagreement as to whether both promising and exemplary programs would be required to meet all of the criteria under the four broad categories, or whether only exemplary programs would have to meet the criteria under the category Evidence of Success.

Some panelists believed that all programs, both promising and exemplary, should meet the Evidence of Success criteria, since they felt uncomfortable having the Department of Education designate programs that do not have any evidence of success. Other panelists did not want Evidence of Success to be the first category of criteria that the panel considered, however, because they felt that simply having data does not mean that a program is good or appropriate, or even that it teaches proper mathematics or science. After discussion, the panel seemed to agree that promising programs would not have to provide Evidence of Success, while exemplary programs would.

The panel agreed that an applicant who was dissatisfied with his or her program's designation (i.e., the developer of a promising program that thought it should be designated exemplary) would have the option of not receiving any designation.

Definition of "Program". The panel is still trying to define clearly what it means by "program". At the last meeting, the panel decided that, for the pilot test, it would consider year-long programs only. Eventually, the definition would broaden to encompass summer programs, modules, semester-long programs, etc. One panelist commented that what the panel considers a "program" must be able to provide some type of student achievement data; for example, a Girl Scout math or science program that collected no assessment data could not be considered. One panelist pointed out that, although the panel may receive more submissions if it looked at shorter programs, there would be a tradeoff in terms of the quantity and quality of outcome data that shorter programs would have. In the end, the panel decided to define the process first, before defining what "program" means.

The Audience. Several panel members discussed who the audience for the product will be. Teachers? School board members? This issue has not been defined.

The "Pilot" Phase. The panel discussed the implications of running a small-scale, pilot review this summer. If the panel conducted a partial competition, could the Secretary then publicly announce that these programs were the only programs considered promising or exemplary? One panel member suggested looking at more "neutral" program during the trial review (e.g., British or Canadian programs). This issue has not been resolved.

Next Steps

Next Meeting. The next meeting is scheduled for Friday and Saturday, March 21 and 22. Panel members have been asked to RSVP to Erica deFur at Pelavin.

Revise Criteria. Pelavin staff edited and formatted the criteria that panel members developed during the meeting. The edited criteria were posted on the listserv and mailed to panel members for comment. Panel members should please comment to Erica deFur or Amy O'Malley at Pelavin as soon as possible. Pelavin staff will compile and synthesize the comments for the March meeting.

Refine the Review Process. Pelavin staff will provide a detailed description of a proposed panel review process before the next meeting. One part of the March meeting will focus on making decisions about the process.

Identify Programs to Review this Summer. Since the panel is scheduled to review several programs this summer, panel members should start thinking about programs that would be good candidates for the trial review. Panelists may want to consider selecting "neutral" programs, as described above, for the trial.

Expert Panel

in Mathematics and Science Education

Summary of the Fourth Panel Meeting

March 21 and 22, 1997

Expert Panel in Mathematics and Science Education:

Summary of the Fourth Panel Meeting

Introduction

Objectives and Overview of the Meeting

Panel members spent much of the March 21 and 22 meeting defining and developing the review process of evaluation, both for future reviews and for this summer's trial run. During the 2- day meeting, panel members clarified the process developed in the last meeting; began conceptualizing potential rating forms, rubrics, and submission packets; revised the criteria; and began to work out the details of this summer's trial run. Prior to the meeting, panel members received a draft review process, sample evaluation rubrics and rating forms, a sample technical report, a sample submission packet, suggested changes to the criteria, an outline of a possible expert panel trial run, and a summary of the third panel meeting. Jo Ellen Roseman and Jim Rutherford of Project 2061 assisted in the compilation of the technical report, the possible trial run, and the submission components.

The first day of the meeting focused on the evaluation process. Steve Leinwand opened the meeting with a description of the goals of the meeting - to work on the process, the criteria, and this summer's trial run. The entire panel discussed major issues within the process before breaking into three groups to work on developing an acceptable method of evaluation and a vision of the submission packet. After lunch, the groups presented the results of their discussions. After the break, the panel discussed its goals for Saturday (the pilot process), and quickly moved to attempt to finalize the criteria.

The panel spent most of the morning of the second day working on the the criteria. The panel then discussed the summer pilot process. The panel decided to run the pilot as a research project rather than a formal competition to avoid facing questions of fairness that might arise if the panel designates programs based on a restricted submission process. There is a tentative plan to send out the submission packet to potential submitters to get feedback on the actual submission. The panel decided that OERI, along with a subpanel, will use its resources to develop the summer training. The appointed subpanel members include: Maria Lopez-Freeman, Jim Minstrell, Steve Schneider, and Jan Tuomi.

The panel concluded with the expectation of significant interaction among panel members, OERI, and Pelavin over the next month to finalize the criteria, the process, and to begin developing details for this summer's pilot. Panel members were sent the latest draft of the process and the criteria.

The Evaluation Process

Summary of the Evaluation Process. After the call for submissions, applicants will send completed submissions to OERI or a contractor for distribution to Quality Review Panels (QRP). Two QRPs will review each submission, and will conduct a pre-screening, using the first criterion under Educational Significance and verifying the existence of some Evidence of Success. When evaluating the programs, the QRP members will use a detailed set of descriptors based on the criteria and across a 5-point scale. An appointed member of the QRP will develop a technical report and a recommendation. Programs that do not meet the criteria will be returned to the applicants with feedback. Programs that do meet the initial criteria of the QRP will be sent to the Expert Panel for Review. Approved programs with data will be sent to the Impact Review Panel (IRP), which is comprised of evaluation as well as content-area experts, to determine if submissions qualify as promising or exemplary. The IRP's recommendation will be sent to the Expert Panel, who will review the recommendations, and send final recommendations to the Secretary for approval.

Steve Leinwand and Jim Rutherford began the meeting by opening the floor to general comments on the process as described in Tab 2 of the pre-meeting materials.

The following issues were raised by panel members:

Evidence of Success. Panel members agreed that all programs must have some amount of data to support the Evidence of success Criteria to be considered as promising or exemplary. Therefore, the process was changed so that all programs must go to the Impact Review Panel to receive designation. In the previous draft, programs with no data could still receive a designation of promising if they fulfilled the requirements of the Quality Review Panel.

Impact Review Panel. In light of the above change, panel members considered having the IRP meet twice a year to reflect the fact that all programs must be evaluated by this panel to receive designation. However, one panel member pointed out that an annual announcement might make the designation more distinctive. A final agreement was not reached.

In addition, panel members agreed that reviewers on the IRP should have evaluation expertise as well as content-area expertise. Panelists had been concerned about creating a dichotomy between content and data.

Expert Panel. Members of the panel decided that the requirements for the Expert Panel, as written in the process, were not stringent enough. The panel agreed that Expert Panel members must also have a record of demonstrated leadership and be appointed by the Secretary of Education.

Innovative v. Successful Programs. One panel member raised the concern that the current process may give precedence to innovative programs rather than programs that actually work. Panel members agreed that there are three areas of potential evaluation mistakes: programs that have evidence of success, but are not recognized; programs that look good but are not effective, and receive designation; or programs that have evidence of success, but no significance, and receive designation. Nonetheless, another panel member argued that it is a mistake to think that there is always a dichotomy between successful and innovative.

Payoffs of Participating. Some panel members expressed a concern that there are not enough payoffs for preparing a submission package and entering one's program in the competition. They maintained that a letter of certification and a listserv posting may not be a strong enough incentive to involve quality programs. Sharon Bobbitt mentioned that ED is considering different ideas on how to deal with this issue (i.e. OERI could provide research funding for continued evaluation of programs designated as promising.) In addition, Bobbitt pointed out that the legislation requires that ED actively solicit programs for review, and OERI plans to conduct a widespread dissemination process for designated programs.

Commercial programs have the most incentive to compete for this type of designation. There is a concern that smaller organizations will not have the ability (time or money) to get involved in such a complex process.

Small Group Discussion of the Process

The panel broke into three groups to address Tabs 3-6 (Sample Evaluation Rubrics, Sample Evaluator Rating Forms, Possible Quality Review Technical Report, and Submission Guidelines) of the pre-meeting materials. Panel members were asked to come back from the small groups with a conception of how to evaluate programs (rubric and rating form) and a conception of the submission packet. Below are the issues that the groups addressed and the decisions made:

Submission Packets. Panel members generally agreed with the conceptualization of the submission packet as presented by Group 2. The submission packet will include: a cover page, an abstract or fact sheet that makes a case for designation as promising or exemplary, a 3 page description of the program, a 10-15 page essay that provides evidence of how the program meets the criteria, and a 10 page summary and presentation of the data that makes a compelling case that the program works. Panelists agreed that the Evidence of Success should be in a separate folder from the rest of the application so that the IRP could review the information at the same time. In addition, submitters should include materials from the program. The entire application (excluding materials) should be less than 30 pages. Marty Apple volunteered to work with Pelavin in developing a new draft of the submission packet. Question: How many copies of these materials do submitters need to provide?

Evaluation Rubric. The panel decided to develop an evaluation rubric along a 5 point scale, with descriptors at the 1, 3, and 5 levels. These descriptors will be similar to the California Rubric - providing detailed descriptions of what constitutes an excellent, acceptable, or unacceptable (or "high," "medium," or "low") program. Each paragraph will be based on one of the numbered criteria under the 4 main categories of criteria. The descriptors will detail the asterisked examples under each criterion. The panel agreed that the descriptors should be developed by individuals with expertise in this area. Staff will generate the first draft of the evaluation rubric.

Pre-Screening. All three groups chose to use the criteria as a basis for the evaluation design. In addition, Group 1 suggested that there are certain criteria that are more important than others. They argued that if a program could not demonstrate its educational significance for some group of students and if a program could not show any evidence of success, then the reviewers should not evaluate the program at all. The panel agreed to use these two criteria for the pre-screening, which the Quality Review Panels would conduct. Specifically, only submissions that the QRPs agree meet the first criterion under Educational Significance AND provide at least some Evidence of Success would receive a complete review. Question: how will QRP's resolve differences at this initial stage of evaluation?

Evaluation Rating Forms. The panel chose to use a rating form that utilized a likert scale as well as a justification for scores given. Panel members decided to include scores of the individual panel members, summed scores, and averaged scores for the summary sheet in addition to a brief summary and a recommendation. The Quality Review Panel will provide six scores (on a 1 to 5 scale) in calculating their total score. The first criterion under Educational Significance will be used in the pre-screening, and therefore will not be applicable; and the Impact Review Panel reviews effectiveness. Please see Appendix A and B for a sample rating form.

Scoring. The panel agreed that a 5-point scale would be an acceptable scoring method. The Quality Review Panel will be able to give a program a total of 30 points - 5 points for Educational Significance (the second criterion), 20 points for Quality of Program (the four numbered criteria), and 5 points for Usefulness (one criterion). Question: Does this scoring system establish the importance of different aspects of the criteria in an appropriate manner?

One member suggested the possibility of reviewers using fractional ratings, such as 3.2, 4.7, or 2.1. There did not appear to be much support for this suggestion.

One group suggested that a program must meet a certain average score to receive designation, AND meet a minimal score for individual criteria. For example, in calculating the quality of the program, if the minimum total score was a 13 and the minimum individual score was a 2, a 5 + 4 + 3 + 2= 14 might be acceptable, but a 5 + 4 + 4 + 1= 14 would not. This issue was not resolved.

Technical Reports. Panel members agreed that the technical reports should reflect minority views within the Quality Review Panel (or the Impact Review Panel). In addition, members wanted to make sure that all individual forms are attached to the group summary. The technical report will provide a more in-depth analysis of the review; the evaluation summary might be like an "executive summary" to the technical report. Question: How does the technical report relate to the evaluation summary?

Weighting. The panel decided not to use additional weighting methods when calculating a program's score. They argued that using the current method already makes Quality of Program the most significant criteria.

Sampling. The panel agreed that a reviewer cannot be expected to read every piece of curriculum for a 12-month program; therefore, reviewers will review samples of the program. In addition, the panel agreed that there should be a technique to sampling curriculum materials so that a submission does not include just the best chapter of a textbook, for example. One panel member argued that the submission form should be based on claims which are supported by a representative sample of materials. However, this issue remained largely unresolved.

Disagreements in Recommendations. The Expert Panel decided that if the Quality Review Panels differ in their recommendations, they will have to work out their differences before sending their recommendations to the Expert Panel. The differing QRPs will convene telephone conferences to discuss the submissions until they come to a consensus. Panel members rejected the idea of the Expert Panel acting as a mediator in these decisions and decided that calling a third panel would not be cost-effective.

The Reviewers. Panel members agreed that all review teams should contain a person who has current classroom experience, a content specialist (i.e. mathematician, biologist), and a development, implementation, or research specialist (e.g. mathematics or science researchers, assessment or evaluation specialists). More specifically, panel members would like to include the following: school teachers, supervisors, researchers, grantees, content experts, curriculum expert, evaluation person, and people who have implemented successful and/or new courses. Panel members agreed that the two QRPs should have different voices to create a broader spectrum of perspectives.

One panel member pointed out that the Impact Review Panel should not contain only evaluators, but should also contain evaluators with some content experience, and the process was thus changed.

Criteria

Panel members spent a significant amount of time Friday afternoon and Saturday morning reviewing the latest draft of the criteria. Panel members were e-mailed a copy of these changes on Thursday, March 27 that reflects the changes made during the meeting. An additional hard copy was sent with a copy of the process on Monday, March 31. This draft of the criteria is hoped to be the last draft until after the pilot process; panel members have been asked for their comments or final approval.

Summer Pilot Test

Much of the discussion of the pilot focused on the scope and ethical implications of the summer's pilot review. Panel members were concerned about the implications of publicizing the results from an experimental process. In addition, one member was concerned about giving an extra edge to programs that submitted this summer as a "practice run." Therefore, the panel decided that this summer's pilot would be labeled as a "research project," and would make no promises of reward or recognition.

Who Submits this Summer? In light of the above discussion, panel members decided to use programs for evaluation without a solicitation process. Over the next few weeks, OERI will find potential programs to use in this trial run. Programs that the panel may use include: EDC, BSCS, NSF, and Department of Education funded programs. The panel wanted to minimize the burden for potential applicants for this summer's trial run. One member suggested that it might take an applicant one-half a day to complete a rough submission. Another member suggested having staff members "write" submissions from information packets on programs. Question: Who will "write" the submissions, and how complete will they be?

The panel agreed that it would review 6 programs this summer - 3 math and 3 science. In addition, the panel agreed that the programs used should have a range of quality to reflect different types of potential applications, including high school, elementary school, and middle school programs; excellent, weak, and borderline programs; and innovative and traditional programs.

Submission Packet Questionnaire. In addition, panel members agreed to conduct a small study with regards to the submission requirements. Panel members plan to send the submission packet to some potential applicants for feedback. (i.e. Would they fill out the form? Is the length reasonable? Are the requirements too high or too low? Is it confusing? Etc.) Details of this study need to be established.

Size of Quality Review Panel. A panel member raised a concern that 3 was not a large enough number to avoid reviewer biases in the Quality Review Panel. After discussion, it was decided that the pilot review will use 2 teams of 3 and two teams of 5 to compare which size works better.

This Summer's Reviewers. The Expert panel decided that it will need 16 reviewers for this summer's evaluation process. There will be 8 science reviewers and 8 mathematics reviewers. Reviewers will receive a $1,000 honoraria for their work, which will involve one week of participation. Panel members suggested using people who already have evaluation experience and can provide feedback on the current evaluation process. The panel discussed holding two separate training sessions - one on the East Coast and one on the West Coast - to reduce travel costs. Steve Schneider volunteered the possibility of using the Consortia. The panel decided that OERI would work with the summer pilot subcommittee to find potential reviewers.

Training. The panel members agreed that the training/evaluation process this summer should occur over 5 days: 2 days training, 2 days rating, and 1 day debriefing. The expert panel plans to participate in the training portion of the process, and some will participate in the entire 5 days. OERI will develop some of the logistics of the training over the next month. Question: Who will develop the training course and conduct the training?

The panel appointed a four person subcommittee to work directly with OERI and Pelavin on the training process. The appointed members include: Maria Lopez-Freeman, Steve Schneider, Jim Minstrell, and Jan Tuomi. If the May meeting is canceled, these panelists may still need to establish a meeting time.

Issues Raised During the Meeting

In summary, the Expert Panel resolved many of the details and questions about the evaluation process, the criteria, and this summer's trial run that had previously not been visualized. However, this has led to new questions, and some aspects about this pilot study remain quite vague. The following are the questions raised throughout the meeting summary.

Evaluation Process

How often will the Impact Review Panel meet?

How will the panel and OERI create incentives for programs to participate?

How many copies of original materials do submitters need to provide?

Who will develop the evaluation descriptors around the different scoring levels?

In the Pre-Screening Stage, how will QRP's resolve differences?

With respect to scoring, does the current system establish the importance of different aspects of the criteria in an appropriate manner?

Will the panel have an average cut-off score as well as a minimum individual cut-off score?

Will reviewers write a summary of their decision or a technical report? How are they related?

How will reviewers choose which materials to sample?

Summer Pilot Test

Who will "write" the "submissions" for the pilot, and how complete will these submissions be?

How should we design the questionnaire that potential reviewers would fill out when looking at the potential Submission Packet?

Who will develop the training course and conduct the training?

Where will we conduct the training and the review?

How will the panel know if the pilot has been a success?

Timeline

Who's Responsible	Anticipated

Date Sent to Panel

Respond by:
1. Criteria	Pelavin	March 27, 1997	April 16
2. Next Draft of the Process	Pelavin	April 10, 1997	April 18
3. Meeting Summary	Pelavin	April 10, 1997	---------
4. Submission Packet	Pelavin	April 18, 1997	April 25
5. Training Program	OERI & Subpanel
6. Develop Descriptors	?	April 28, 1997	May 15, 1997
7. Develop Scoring Forms	Pelavin	April 28, 1997	May 15, 1997
8. Identify Pilot Reviewers	OERI
9. Identify Pilot Programs	OERI

Appendix A

Sample Rating Form from March Meeting of

the Expert Panel in Mathematics and Science Education

Educational Significance

2. The program's learning goals reflect the vision promoted in the national standards.

High	Medium	Low

Detailed Description	Detailed Description	Detailed Description

Using Asterisks	Using Asterisks	Using Asterisks

Rating:	|	|	|	|	|

5	4	3	2	1

Evidence and Justification:

Appendix B

Summary Rating Sheet

Standard	Individual Rater Scores	Sum	Average

#1	#2	#3	Totals	Totals

1. Educational Significance	_____	_____ _____	_______
_______

2. Quality #1	_____	_____ _____	_______
_______

3. Quality #2	_____	_____ _____	_______
_______

4. Quality #3	_____	_____ _____	_______
_______

5. Quality #4	_____	_____ _____	_______
_______

6. Usefulness	_____	_____ _____	_______
_______

Total

Summary of Findings:

Recommendation: __________________________________________