|
Home  Our Approach Evaluating the Evaluators
|
Evaluating the Evaluators
As consumers, we know all products and services generally have a broad range in quality from excellent to poor. Consumer guides to products and services are no different. Read Naturally has experienced this range of quality with the consumer guides that have reviewed our strategy and products. To provide a backdrop for analysis of these consumer guide reviews, we would like to share with you our research and experience regarding this range of quality. It is important for educators to understand that an evaluation of how a review was produced is critical to determining if the review has merit. As you know, getting accurate information is necessary in order to make good decisions about products.
The Florida Center for Reading Research, the University of Oregon, and the What Works Clearinghouse (WWC) are three of the consumer guides that have published reviews of the Read Naturally strategy. Based on Read Naturally’s research on and experience with these consumer guides, as well as our knowledge of the studies they analyzed, we assert that the FCRR and the University of Oregon reviews provide valid critiques of our strategy. The WWC reviews do not.
Chart comparing the WWC, FCRR, University of Oregon, & NCRTI
Strengths of the Florida Center for Reading
Research (FCRR) and University of Oregon
As the evaluations and credentials outlined on these organizations’ websites testify, the Florida Center for Reading Research (FCRR) and the University of Oregon do a thorough job of evaluating educational products for struggling readers. At both sites, the leadership and principal investigators are reading experts. For example, on the FCRR’s website, you will note that all 13 of the center’s leadership and curriculum specialists are highly trained experts in reading instruction. Also, the FCRR is led by two of the country’s foremost reading experts, Joe Torgeson and Barbara Foorman.
Experts at the FCRR not only evaluate the research on the various products they review, but they also learn the strategies of the products and evaluate the products based on overall best practices. The University of Oregon evaluated programs' alignment to scientifically based reading research and best practices. The FCRR and the University of Oregon both specialize in reading instruction product evaluations; this expertise is reflected in their analyses and attention to detail.
Shortcomings of the What Works Clearinghouse (WWC)
An analysis of the What Works Clearinghouse (WWC) leadership shows limited expertise in reading instruction among its key staff and principal investigators. Only two of the 14 key staff and principal investigators could be considered experts in reading instruction. Furthermore, WWC does not specialize in reading. This organization reviews a wide range of educational products, from math to high school dropout programs.
WWC looks at selected research that meets its methodological and statistical criteria, but WWC does not evaluate and analyze how studies were implemented in the classroom or the purpose of these studies. This organization is primarily interested in whether the statistical model for the study was implemented properly.
Dr. Grover Whitehurst, Director of the Institute of Education Sciences, confirmed WWC’s limited scope of evaluation when he stated: “WWC emphasizes the methodological and statistical characteristics of the research it reviews. WWC does not concern itself with a conceptual analysis of the strategies on which educational programs are based or with the degree to which programs are aligned with particular principles.”
Educators know that evaluating the purpose of a study and whether or not a reading strategy was implemented properly is essential in helping to discern if a review of the study has value. WWC’s refusal to include an evaluation of a study’s purpose and the implementation of a strategy within that study is a serious omission in its evaluation process.
Read Naturally’s Experience
Read Naturally’s own experience with FCRR, the University of Oregon, and WWC illustrates the differences between these organizations and their respective reviews.
FCRR and the University of Oregon worked diligently to read the research on Read Naturally and to understand the Read Naturally strategy in the context of best practices research.
In contrast, WWC which does NOT consider the purpose or implementation of studies, has posted four evaluations of Read Naturally on its website. Each of the evaluations were based on studies that did not use or assess the Read Naturally strategy properly.
In each case, the study authors clearly state that using the study as an evaluation of Read Naturally would be a misapplication of the data, and that WWC should not do so. Consider the following facts about WWC's reviews:
Hancock Study Author States Study Was Not Intended to Evaluate Read Naturally
Carrie Hancock sent an e-mail to What Works Clearinghouse (WWC) stating that they should not publish a review of her study as a study to evaluate the effectiveness of Read Naturally. In her e-mail, Carrie stated, "While I used Read Naturally materials, I did NOT fully implement the Read Naturally strategy and my study was NOT intended to evaluate the Read Naturally strategy. Rather, the purpose was to determine the impact of ongoing supplemental fluency practice on second grade students rates of learning to read."
The response from Becki Herman, WWC's Project Director, was that WWC does not consider the purpose of the study and does not consider implementation.
Read more about the Hancock study
Denton Study Assessment Measures Incongruous With Read Naturally
Even though Read Naturally is regarded as the best fluency, vocabulary and comprehension building program on the market, the Denton study did not assess fluency. Instead, the Denton study assessed phonemic decoding, which is not part of the Read Naturally strategy. Using phonemic decoding as an assessment measure benefitted Read Well and reflected negatively on Read Naturally. Clearly, the Denton study was designed to measure the effects of Read Well.
One of the authors of the Denton study, Dr. Jan Hasbrouck, acknowledges these flaws. Hasbrouck states: "The measures used to evaluate the effectiveness of the instruction did not include measures of fluency. Since Read Naturally is an intervention targeted at fluency, the measures used did not match the specific purpose and intent of the Read Naturally instruction given to the students in that group."
Read more about the Denton study
WWC’s Review of Kemp Study Deceives Educators
"Another limitation [of the study] is that Read Naturally was not just used with at-risk readers. The Read Naturally program was developed to help students who are at-risk. Typically this would represent students who are scoring below the 25th percentile. Students participating in this study did not fit the description of at-risk."
In addition, the Kemp study had several implementation and design flaws.
Read more about the Kemp study
Chenault Study Authors State WWC’s Review Is “Misapplication of Data”
"The Read Naturally program is a widely recognized and widely used curriculum that has been appropriately validated in other studies. This study was NOT intended as an examination of the Read Naturally program. Using our study to evaluate Read Naturally or its effects is a misapplication of our data."
Belle Chenault's advisor, Dr. Virginia Berninger, states: "You cannot use a contact control (which we knew would be good for the children and not a waste of their time but was not predicted to change the aspect of writing we were training) to draw conclusions about the effectiveness of the contact control (Read Naturally)."
Also, there were significant implementation discrepancies between our recommended process and the way the Chenault study used Read Naturally, including only ten, 25-minute sessions of Read Naturally.
Read more about the Chenault study
Conclusion
Educators need to look critically at evaluators of reading programs. NCRTI, FCRR, and the University of Oregon are discerning and high-quality evaluators who pay attention to the purpose of a study. WWC’s posting of studies, when the authors of the studies did not intend to evaluate Read Naturally, is misinforming educators. It is essential that educators have accurate information when selecting curriculum — the success of our students is at stake.
