|MS Word (175 KB) | PDF (136 KB)|
Submitted PaperThe Logic of Scientific ResearchValerie Reyna
- Why scientific research?
- Research is the only defensible foundation for educational practice.
- If not scientific evidence, then what?
Analogy to Medicine
- Traditions: Bleeding people.
- Good intentions are not enough.
- Clinical trials are recent.
- Why isn't personal experience sufficient?
- Clinical trials: Only way to really be sure of what works. (Logic)
- Same rules apply to education: Brain surgery (NAS).
Strength of Evidence
- In the meantime, hierarchy of evidence.
- Not all-or-none
- Possibly true to probably true versus nothing (NCES example)
- Theory: Evidence-based
- Knowing why and how something works
- Key to generalization
- Pitfalls of theory
- No conflict between science and values.
- Some decisions made on values.
- Evidence is necessary but not sufficient.
- Science with a human face.
- How do we support translation of research into practice?
- Suggestions welcome.
What is EBE?
- Best available empirical evidence in making decisions about how to deliver instruction
- Gaps in scientific evidence: Human judgment (bias and wisdom)
What is empirical evidence?
- Scientifically based research from fields such as psychology, sociology, economics, and neuroscience, and especially from research in educational settings
- Objective measures of performance used to compare, evaluate, and monitor progress
Scientifically Based Research
- Measures and Methods
- Scientific merit (double helix)
- Relevance and Significance
- Number affected and severity
- Two criteria of NSF
Quality: Levels of evidence
- All evidence is NOT created equal
- Randomized trial
- Quasi-experiment, including before & after
- Correlational study with statistical controls
- Correlational study w/o statistical controls (class size, high expectations)
- Case studies
Randomized Trials: The gold standard
- Claim about the effects of an educational intervention on outcomes
- Two or more conditions that differ in levels of exposure to the educational intervention
- Random assignment to conditions
- Tests for differences in outcomes
Why is randomization critical?
- Assures that the participants being compared have the same characteristics across the conditions
- Rules of chance mean that the smart, motivated, experienced, etc. have the same probability of being in condition 1 as in condition 2
- Without randomization, differences between two conditions may result from pre-existing difference in the participants, e.g., more smart ones in condition 1
Why is randomization critical?
Without randomization, simple associations such as between Internet use and science grades have many different interpretations
- Does the study involve a similar intervention and outcome to those of interest?
- Were the participants and settings representative of those of interest?
- Were enough participants involved to justify generalization? (statistical inference)
EBE: How to use existing science
- Search literature (Campbell Collaboration, PsychInfo, etc.)
- Screen literature
- Search for pre-digested evidence
- Narrative reviews (ERIC digests)
- Systematic reviews (meta-analysis)
- Unconditional conclusions
- Conclusions involving hypotheticals
- Conclusions that diverge from evidence
- Strong calls to action
- Mixtures of opinions with evidence
- Low prestige publication outlet
- Publication outlet with ideological agenda
EBE: How to use objective measures
- Find or develop local data
- measure and measure frequently
- Search state and national databases for benchmarks
- Be skeptical
Evidence-Based EducationWhere are we?
Where the Research Dollars Flow
- Of 84 program evaluations and studies planned by the Department of Education for fiscal year 2000, just one involved a randomized field trial
- 51 were a survey of need
- 49 had as their purpose program implementation/monitoring,
- 15 were non-randomized impact evaluations.
- Note: studies could have more than one purpose.
- Source: Robert Boruch, Dorthy de Moya, and Brooke Synder in Robert Boruch and Frederick Mosteller, eds., Evidence Matters (Brookings, 2001).
What ED will do
- The What Works Clearinghouse
- Interventions linked to evidentiary support
- Systematic reviews
- Standards for & providers of evaluations
- Preschool Curriculum Evaluation Research
- Explanatory Research: Why and How
- Funding for evaluations of promising innovations in the field
- Build capacity internally and externally
- ED will provide the tools, information, research, and training to support the development of evidence-based education
- Education across the nation will be continuously improved
- Wide variation in performance across schools and classrooms will be eliminated
The practice of evidence-based education will become routine