[Federal Register: November 17, 1997 (Volume 62, Number 221)]
[Rules and Regulations]               
[Page 61427-61432]
From the Federal Register Online via GPO Access [wais.access.gpo.gov]
[DOCID:fr17no97-19]


[[Page 61427]]

_______________________________________________________________________

Part II

Department of Education

_______________________________________________________________________

34 CFR Part 701

Office of Educational Research and Improvement (OERI) Conduct and
Activities Evaluation Standards; Designation of Exemplary and Promising 
Programs; Final Rule


[[Page 61428]]



DEPARTMENT OF EDUCATION

34 CFR Part 701

RIN 1850-AA52

 
Standards for Conduct and Evaluation of Activities Carried out by 
the Office of Educational Research and Improvement (OERI)--Designation 
of Exemplary and Promising Programs

AGENCY: Department of Education.

ACTION: Final regulation.

-----------------------------------------------------------------------

SUMMARY: The Assistant Secretary for Educational Research and 
Improvement (OERI) is establishing final regulations pursuant to the 
``Educational Research, Development, Dissemination, and Improvement Act 
of 1994.'' The regulations are intended to provide quality assurance 
that programs designated by the Department of Education as either 
exemplary or promising have met criteria that will allow educators, 
professional organizations, and others to use these programs with 
confidence.

DATES: These regulations take effect December 17, 1997. However, 
affected parties do not have to comply with the information collection 
requirement in Sec. 701.4 until the Department of Education publishes 
in the Federal Register notification of the compliance date and the 
control number assigned by the Office of Management and Budget (OMB) to 
this information collection requirement. Publication of the control 
number notifies the public that OMB has approved this information 
collection requirement under the Paperwork Reduction Act of 1995.

FOR FURTHER INFORMATION CONTACT: Stephen O'Brien, U.S. Department of 
Education, 555 New Jersey Avenue, NW, Room 502B, Washington, D.C. 
Telephone: (202) 219-2141. Internet: (Steve__O'Brien@ed.gov). 
Individuals who use a telecommunication device for the deaf (TDD) may 
call the Federal Information Relay Service (FIRS) at 1-800-877-8339 
between 8 a.m. and 8 p.m., Eastern time, Monday through Friday.
    Individuals with disabilities may obtain this document in an 
alternate format (e.g., Braille, large print, audiotape, or computer 
diskette) on request to the person listed in the preceding paragraph.

SUPPLEMENTARY INFORMATION: On March 31, 1994, President Clinton signed 
Public Law 103-227, which includes Title IX, the Educational Research, 
Development, Dissemination, and Improvement Act of 1994 (the Act). The 
Act restructured OERI and provided it with a broad mandate to conduct 
an array of research, development, dissemination, and improvement 
activities aimed at strengthening the education of all students.
    The Act directed the Assistant Secretary to develop, in 
consultation with the National Educational Research Policy and 
Priorities Board (the Board), the highest standards of professional 
excellence necessary to govern the conduct and evaluation of all 
research, development, and dissemination activities carried out by the 
OERI. The legislation requires that the standards be developed in three 
phases.
    In the first phase, standards were promulgated to establish the 
peer review process and evaluation criteria to be used for reviewing 
applications for grants and cooperative agreements and proposals for 
contracts. The Department published final regulations setting out these 
standards on September 14, 1995 (60 FR 47808). The regulations in this 
announcement address the second phase of development by establishing 
the criteria for panels to use in reviewing potentially exemplary and 
promising educational programs. The Assistant Secretary will later 
publish proposed regulations for phase three of the standards, which 
will govern how OERI evaluates performance of its recipients of grants, 
contracts, and cooperative agreements.
    The OERI legislation requires that expert panels be established to 
review educational programs submitted by individuals or organizations. 
The legislation also provides that the Secretary may identify 
educational programs for the panels to review. The statute requires the 
panels to recommend to the Secretary those programs that should be 
designated as exemplary or promising and disseminated through the 
Department's National Education Dissemination System. The law requires 
that each panel consist of appropriately qualified experts and 
practitioners and requires the Secretary to develop standards that 
describe the procedures the panels will use in reviewing the 
educational programs. Section 941(a)(3) of the law broadly defines 
educational programs to include educational polices, research findings, 
practices, and products. Educational programs may range in size and 
complexity from an individual instructional program--such as an 
elementary school science program--to a comprehensive reform initiative 
involving multiple goals and participants. Programs at all levels of 
education--preschool, elementary, secondary, and postsecondary--are 
eligible for consideration.
    In determining whether an educational program should be recommended 
as exemplary or promising, each panel is required by the Act to 
consider: (a) Whether, based on empirical data, the program is 
effective and should be designated as exemplary or (b) whether there is 
sufficient evidence to demonstrate that the program shows promise for 
improving student achievement and should be designated as promising. 
The Act expressly states that a panel shall not eliminate a program 
from consideration based on the lack of one type of supporting data 
such as test scores.
    The evaluation process set forth in the final regulations will 
ensure that programs disseminated by the Department are high-quality, 
research-based programs that have provided evidence indicating they 
have improved teaching, learning, or both, or has demonstrated other 
worthy educational performance outcomes. The Department's dissemination 
system is designed to make information about these promising and 
exemplary programs available to the public as quickly as possible. The 
system will enable the Department to respond to all forms of requests 
for information and assistance, and to support the applications of 
research and best practice. The system will use electronic networking 
and the capabilities of:

--National Research Institutes;
--Educational Resources Information Center (ERIC);
--Regional Educational Laboratories;
--Department-supported dissemination and technical assistance 
providers;
--National Library of Education;
--Eisenhower Regional Consortia and Clearinghouse, and
--Other public and private nonprofit entities, including education 
associations and networks.

    Until recently, the Department validated exemplary programs through 
its Program Effectiveness Panel (PEP) and disseminated them through the 
National Diffusion Network (NDN). Since this program no longer exists, 
with the adoption of these standards the Department will evaluate and 
disseminate promising educational programs in addition to exemplary 
programs. The Department will also work in partnership with 
constituency groups who have expertise in the specific topic areas 
represented by the expert panels to develop coordinated procedures to 
maximize their involvement in this work.
    On June 3, 1996, the Secretary published a notice of proposed 
rulemaking (NPRM) for this part in the

[[Page 61429]]

Federal Register (61 FR 27990-27993). These final regulations contain 
three major changes from the NPRM. These changes are fully explained in 
the ``Analysis of Comments and Changes'' elsewhere in this preamble. 
The changes pertain to the standing panel; the distinction between 
``promising'' and ``exemplary''; and the factors listed under the 
criteria expert panels will use to evaluate programs.

Analysis of Comments and Changes

    In response to the Secretary's invitation in the NPRM, seven 
parties submitted comments on the proposed regulations. This included 
comments from individual members of two pilot panels (math/science and 
gender equity) that were appointed by the Secretary to field test the 
expert panel process. In addition to the public comment, comments from 
the Board's Subcommittee on Standards are addressed as required by the 
legislation. The full Board approved the final regulations at a meeting 
on September 26, 1997. An analysis of the comments and of the changes 
in the regulations since publication of the NPRM follows.
    Major issues are grouped according to subject with appropriate 
sections of the regulations referenced in parentheses. Technical and 
other minor changes--and suggested changes the Secretary is not legally 
authorized to make under the applicable statutory authority--are not 
addressed.

Eligibility (Sec. 701.3)

    Comments: One commenter asked for clarification on who is eligible 
to submit educational programs for designation as promising or 
exemplary. Specifically, this commenter asked whether federally-funded 
entities, such as the Regional Laboratories, will be required to go 
through this process; whether local agencies that receive Federal 
funding through states, such as under Title I of the Elementary and 
Secondary Education Act (ESEA), may submit programs on their own; 
whether sponsors need to be invited to submit or may submit on their 
own initiative; and whether for-profit entities may submit.
    Discussion: The law provides that ``individuals'' or 
``organizations'' may submit educational programs for review. Since the 
law is silent on the specific nature of the organizations, the 
Secretary believes that for-profit agencies would be eligible to submit 
programs for review. With respect to the OERI-funded Regional 
Educational Laboratories, the law provides that the Secretary may 
identify those programs for panel review. In addition, the Secretary 
believes that the Laboratories could submit one or more of their 
programs on their own initiative. The question of whether local 
agencies that receive Federal funding through a State or Federal 
entity, such as under Title I of the ESEA, can submit on their own or 
must go through their funding agency, will be addressed in 
administrative guidance.
    Changes: None.

Content of Submissions (Sec. 701.4)

    Comments: Three commenters made suggestions about this section. Two 
commenters believed that requiring funding and staffing information was 
burdensome and not germane to the designation of a program as promising 
or exemplary. One commenter believed that this section should require 
program sponsors to submit specific materials related to content and 
methods. Another commenter believed that this section should include 
the requirement that the program include evidence of sustainability of 
improvement with targeted student populations.
    Discussion: The Secretary believes that funding and staffing 
information should be included to help determine whether an educational 
program should be recommended as either exemplary or promising. The 
Secretary agrees that sponsors should be required to submit information 
or materials specific to content and methods, as available and 
appropriate. The Secretary believes that the evidence of sustainability 
of student improvement should be evaluated by peer reviewers in 
accordance with Sec. 701.22.
    Changes: Section 701.4(b)(7) has been renumbered as 
Sec. 701.4(b)(8) and a new Sec. 701.4(b)(7) has been added to include a 
provision for specific materials relevant to content and methods.

Procedures for Submitting Educational Programs (New Sec. 701.5)

    Comments: One commenter believed that the regulations should 
contain more specificity about the procedure for submitting programs to 
the expert panels. This commenter requested specifics on who receives 
the submissions and whether they may be submitted at any time or only 
on specific dates.
    Discussion: The Secretary agrees that the general submission 
procedures should be included in regulations. A sponsor seeking the 
exemplary or promising designation for its educational program may 
submit its program at any time for consideration to the Assistant 
Secretary, who will assign the submitted program to the appropriate 
panel for review. The individual expert panels will set appropriate 
timelines for program submissions. In addition, the Assistant Secretary 
will periodically establish and announce in the Federal Register 
specific topic areas of high priority. Sponsors of educational programs 
in these areas will be invited to submit them for consideration.
    Changes: A new Sec. 701.5 has been added to include general 
procedures for submitting educational programs for review by an expert 
panel.

Establishment of Panels (Sec. 701.10)

    Comments: The Board's Subcommittee on Standards recommended a 
change to the expert panel system. The Subcommittee thought that the 
structure of having members of the expert panels drawn from a separate 
standing panel of educational experts was an unwieldy, overly-
complicated structure. The Board recommended that the expert panels be 
formed separately from a standing panel, which would instead provide an 
administrative oversight and monitoring function for the expert panels.
    Discussion: The Secretary agrees that the expert panel should be 
formed separately from a standing panel. The Secretary will determine 
the feasibility of establishing a separate standing panel for the 
oversight and monitoring functions referred to by the Board--functions 
which are administrative in nature and could also be performed by OERI 
staff. Elimination of a reference to a standing panel in the 
regulations would not alter the composition and function of the expert 
panels as outlined in the NPRM.
    Changes: Section 701.10(a) has been removed, Sec. 701.10(b) has 
been revised, Sec. 701.11 has been removed, Sec. 701.12(a) has been 
revised, and Sec. 701.12 has also been renumbered as Sec. 701.11.

Panel Membership (Secs. 701.11 and 701.12)

    Comments: One commenter observed that Secs. 701.11 and 701.12 in 
the NPRM did not explicitly state that those serving on the panels 
would represent both the community of practice and that of research. 
One commenter believed that each panel should include one or more 
members with evaluation expertise in order to help evaluate evidence of 
effectiveness.
    Discussion: The Secretary agrees with these comments.
    Changes: A sentence has been added at the end of the renumbered 
Sec. 701.11(a) (formerly Sec. 701.12(a)) stating that the membership of 
the expert panels will represent both the community of practice and the 
community of research. Additionally, Secs. 701.11(b)(3) and

[[Page 61430]]

701.11(b)(4) have been renumbered as Secs. 701.11(b)(4) and 
701.11(b)(5), respectively; and a new provision for Sec. 701.11(b)(3) 
has been added to include the selection of at least one individual with 
expertise in evaluating educational programs.

Difference Between Promising and Exemplary Programs (Sec. 701.21)

    Comments: Five comments were received on the distinction between 
promising and exemplary programs. As proposed in Sec. 701.21, the 
distinction was based upon the generalizability of the educational 
programs. Promising programs had to meet each of the criteria of 
educational effectiveness in Sec. 701.22 (success, quality, educational 
significance, and usefulness to others) with respect to only one 
``context or population.'' Exemplary programs had to meet each of the 
criteria ``with respect to multiple contexts or multiple populations.''
    Two commenters believed that the distinction should stay the way it 
was in the NPRM, although one of those suggested some clarifying 
language. However, three commenters questioned the distinction on the 
basis that it was too narrowly and artificially drawn and did not 
reflect the commonly understood meaning of the words ``promising'' and 
``exemplary.'' In this regard, one commenter believed that promising 
programs should not have to meet every criterion in Sec. 701.22 at the 
same level as exemplary programs. Two commenters believed that 
promising programs should have to meet the criteria at the same level 
as exemplary, but that the evidence required of promising programs 
should be less stringent and that exemplary programs should be held to 
a higher standard of evidence.
    Discussion: The Secretary agrees with those commenters who 
questioned the proposed distinction and advised OERI to give a more 
common sense meaning to the terms ``promising'' and ``exemplary.'' The 
Secretary believes that the distinction between promising and exemplary 
programs specified in legislation is sufficient to cover these 
concerns. The Secretary relies upon the expert judgment of the expert 
panel reviewers in determining the nature and weight of evidence 
necessary to designate a program as either promising or exemplary, and 
in applying the criteria listed in Sec. 701.22 in making this 
determination.
    Changes: A revision has been made to the distinction between 
``promising'' and ``exemplary.''

Criteria (Sec. 701.22)

    Comments: Five commenters provided comments on this section and 
suggested revisions to either the wording of the criteria or to the 
content of the factors listed under each criterion or both. These 
comments included comments from one member of the math/science pilot 
panel and three members of the gender equity pilot panel. Although the 
math/science panel member did provide comments specific to the proposed 
criteria and factors, the consensus of this panel was that the expert 
panel process would be better served if each panel developed its own 
factors specific to the content or discipline or both under review by 
the individual panel. One commenter suggested that the word 
``replicability'' would better capture the concept for the criterion 
entitled ``usefulness to others.'' In addition, OERI's Board (The 
Subcommittee on Standards) thought that the regulations should be as 
simple as possible and should give the expert panels as much discretion 
as possible in evaluating programs submitted for review.
    Discussion: In addition to the math/science and gender equity 
panels, the Secretary will establish pilot panels in technology and 
early reading in the next year. The Secretary has determined that until 
the work of all four pilot panels is concluded, the regulation should 
retain only the four criteria outlined in the NPRM in Sec. 701.22 and 
allow each panel the flexibility to establish its own individual 
factors under each criterion that are specific to its content or 
discipline. The fact that the comments from the public suggested 
various changes to the factors underscores the desirability of this 
approach. While the final regulations will therefore no longer require 
the expert panels to apply the factors listed in the NPRM, the 
Secretary encourages each panel to look at these factors as suggested 
examples. The Secretary will review the factors developed by all of the 
panels to see if the criteria set forth in the final regulations need 
to be modified.
    Changes: The factors specified under each of the four criteria have 
been eliminated and the criterion, ``usefulness to others'' has been 
changed to ``replicability.''

Assessment of Educational Impact

    In the NPRM the Secretary requested comments on whether the 
proposed regulations would require transmission of information that is 
being gathered by or is available from any other agency or authority of 
the United States.
    Based on the response to the NPRM and on its own review, the 
Department has determined that the regulations in this document do not 
require transmission of information that is being gathered by or is 
available from any other agency or authority of the United States.

Electronic Access to This Document

    Anyone may view this document, as well as all other Department of 
Education documents published in the Federal Register, in text or 
portable document format (pdf) on the World Wide Web at either of the 
following sites:

http://ocfo.ed.gov/fedreg.htm
http://www.ed.gov/news.html

    To use the pdf you must have the Adobe Acrobat Reader Program with 
Search, which is available free at either of the previous sites. If you 
have questions about using the pdf, call the U.S. Government Printing 
Office toll free at 1-888-293-6498.
    Anyone may also view these documents in text copy only on an 
electronic bulletin board of the Department. Telephone: (202) 219-1511 
or, toll free, 1-800-222-4922. The documents are located under Option 
G--Files/Announcements, Bulletins and Press Releases.

    Note: The official version of this document is the document 
published in the Federal Register.

List of Subjects in 34 CFR Part 701

    Education, Educational research, Reporting and recordkeeping 
requirements.

(Catalog of Federal Domestic Assistance Number does not apply)

    Dated: November 11, 1997.
Ricky T. Takai,
Acting Assistant Secretary for Educational Research and Improvement.

    The Secretary amends chapter VII of title 34 of the Code of Federal 
Regulations by adding a new part 701 to read as follows:

PART 701--STANDARDS FOR CONDUCT AND EVALUATION OF ACTIVITIES 
CARRIED OUT BY THE OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT 
(OERI)--DESIGNATION OF EXEMPLARY AND PROMISING PROGRAMS

Subpart A--General

Sec.
701.1  What is the purpose of these standards?
701.2  What definitions apply?
701.3  Who is eligible to submit an educational program for review?

[[Page 61431]]

701.4  What must a program sponsor submit for review?
701.5  What are the procedures for submitting an educational program 
for review by an expert panel?

Subpart B--Selection of Panel Members

701.10  How are panels established?
701.11  How is the membership of expert panels determined?

Subpart C--The Expert Panel Review Process

701.20  How does an expert panel evaluate programs?
701.21  What is the difference between an exemplary and a promising 
program?
701.22  What criteria are used to evaluate programs for exemplary or 
promising designation?

    Authority: 20 U.S.C. 6011(i), unless otherwise noted.

Subpart A--General


Sec. 701.1  What is the purpose of these standards?

    (a) The standards in this part implement section 941(d) of the 
Educational Research, Development, Dissemination, and Improvement Act 
of 1994.
    (b) These standards are intended to provide quality assurance that 
educational programs designated by the U.S. Department of Education as 
either exemplary or promising have met criteria that will allow 
educators, professional organizations, and others to use these programs 
with confidence.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))


Sec. 701.2  What definitions apply?

    The following definitions apply to this part:
    Assistant Secretary means the Assistant Secretary for the Office of 
Educational Research and Improvement.
    Educational programs mean educational policies, research findings, 
practices, and products.
    Program sponsor means a party submitting an educational program for 
designation by the Secretary as either promising or exemplary.
    Secretary means the Secretary of the Department of Education or an 
official or employee of the Department acting for the Secretary under a 
delegation of authority.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))


Sec. 701.3  Who is eligible to submit an educational program for 
review?

    Any public or private agency, organization or institution, or an 
individual may submit an educational program for review.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))


Sec. 701.4  What must a program sponsor submit for review?

    (a) To have an educational program considered for designation as 
exemplary or promising, a sponsor must submit to the Secretary a 
description of the program, program materials, and a discussion of the 
program that is responsive to the criteria in Sec. 701.22.
    (b) Information submitted must include, to the extent relevant to 
the particular program,--
    (1) A program abstract of 250 words or less;
    (2) A description of the salient features of the program;
    (3) A description of the program's philosophy and history;
    (4) Site information, including demographics;
    (5) A description of evaluation results;
    (6) Funding and staffing information;
    (7) Specific materials relevant to content and methods, as 
appropriate; and
    (8) Organization name, address, telephone and fax numbers, e-mail 
address (if available), and contact person.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))


Sec. 701.5  What are the procedures for submitting an educational 
program for review by an expert panel?

    (a) An applicant seeking the exemplary or promising designation for 
its educational program may submit its program at any time for 
consideration to the Assistant Secretary, who will assign the submitted 
program to the appropriate expert panel for review.
    (b) The Assistant Secretary will periodically establish and 
announce in the Federal Register specific topic areas of high priority. 
Sponsors of educational programs in these areas will be invited to 
submit their programs for consideration.
    (c) The individual expert panels will set appropriate timelines for 
reviewing program submissions.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))

Subpart B--Selection of Panel Members


Sec. 701.10  How are panels established?

    The Assistant Secretary selects individuals, based on their areas 
of expertise, to serve on expert panels in specific topic areas for the 
purpose of reviewing and evaluating educational programs and 
recommending, to the Secretary, those programs that should be 
designated as exemplary or promising.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))


Sec. 701.11  How is the membership of expert panels determined?

    (a) For the review of each program or group of programs, the 
Assistant Secretary establishes an expert panel. The membership of the 
expert panels will represent both the community of practice and the 
community of research.
    (b) In establishing the membership of each expert panel, the 
Assistant Secretary--
    (1) Selects individuals who have in-depth knowledge of the subject 
area or content of the program or group of programs to be evaluated;
    (2) Selects at least one current teacher, principal, or other 
school-based or community-based professional;
    (3) Selects at least one individual with expertise in evaluating 
educational programs;
    (4) Ensures that no more than one-third of the panel members are 
employees of the Federal Government; and
    (5) Ensures that each panel member does not have a conflict of 
interest, as determined in accordance with paragraph (c) of this 
section, with respect to any educational program the panel member is 
asked to review.
    (c) Panel members are considered employees of the U.S. Department 
for the purposes of conflicts of interest analysis and are subject to 
the provisions of 18 U.S.C. 208, 5 CFR 2635.502, and the Department's 
policies used to implement those provisions.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))

Subpart C--The Expert Panel Review Process


Sec. 701.20  How does an expert panel evaluate programs?

    (a) Each panel member shall--
    (1) Independently review each program based on the criteria in 
Sec. 701.22;
    (2) Provide written comments based on an analysis of the strengths 
and weaknesses of the program according to the criteria;
    (3) Participate in site visits or other verification activities, if 
appropriate; and
    (4) Participate in a meeting of the expert panel, if appropriate, 
to discuss the reviews.
    (b) A panel may not eliminate an educational program from 
consideration based solely on the fact that the program does not have 
one specific type of supporting data, such as test scores.
    (c) Each expert panel shall make a recommendation to the Secretary 
as to

[[Page 61432]]

whether the program is exemplary, promising, or neither.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))


Sec. 701.21  What is the difference between an exemplary and a 
promising program?

    (a) In determining whether an educational program should be 
recommended as exemplary or promising, the panel shall consider--
    (1) Whether, based on empirical data, the program is effective and 
should be designated as exemplary; or
    (2) Whether there is sufficient evidence to demonstrate that the 
program shows promise for improving student achievement and should be 
designated as promising.
    (b) The Secretary relies upon the judgment and expertise of peer 
reviewers, as established in Sec. 701.11, to determine the nature and 
extent of evidence required to distinguish between promising and 
exemplary programs and to apply the four criteria established in 
Sec. 701.22, and their own individual factors under each criterion in 
making this determination.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))


Sec. 701.22  What criteria are used to evaluate programs for exemplary 
or promising designation?

    The Secretary establishes the following evaluation criteria for 
expert panels to use in determining whether an educational program 
should be recommended as exemplary, promising, or neither:
    (a) Evidence of success.
    (b) Quality of the program.
    (c) Educational significance.
    (d) Replicability.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and (E), 6041(d))

[FR Doc. 97-30051 Filed 11-14-97; 8:45 am]
BILLING CODE 4000-01-P