Evaluating the Evaluators
As consumers, we know all products and services generally have a broad range in quality from excellent to poor. Consumer guides to products and services are no different. Read Naturally has experienced this range of quality with the consumer guides that have reviewed our strategy and products. To provide a backdrop for analysis of these consumer guide reviews, we would like to share with you our research and experience regarding this range of quality. It is important for educators to understand that an evaluation of how a review was produced is critical to determining if the review has merit. As you know, getting accurate information is necessary in order to make good decisions about products.
The Florida Center for Reading Research, the University of Oregon, and the What Works Clearinghouse (WWC) are three of the consumer guides that have published reviews of the Read Naturally strategy. Based on Read Naturally’s research on and experience with these consumer guides, as well as our knowledge of the studies they analyzed, we assert that the FCRR and the University of Oregon reviews provide valid critiques of our strategy. The WWC reviews do not.
Chart comparing the WWC, FCRR, and University of Oregon
Strengths of the Florida Center for Reading
Research (FCRR) and University of Oregon
As the evaluations and credentials outlined on these organizations’ websites testify, the Florida Center for Reading Research (FCRR) and the University of Oregon do a thorough job of evaluating educational products for struggling readers. At both sites, the leadership and principal investigators are reading experts. For example, on the FCRR’s website, you will note that all 13 of the center’s leadership and curriculum specialists are highly trained experts in reading instruction. Also, the FCRR is led by two of the country’s foremost reading experts, Joe Torgeson and Barbara Foorman.
Experts at the FCRR not only evaluate the research on the various products they review, but they also learn the strategies of the products and evaluate the products based on overall best practices. The University of Oregon evaluated programs' alignment to scientifically based reading research and best practices. The FCRR and the University of Oregon both specialize in reading instruction product evaluations; this expertise is reflected in their analyses and attention to detail.
Shortcomings of the What Works Clearinghouse (WWC)
An analysis of the What Works Clearinghouse (WWC) leadership shows limited expertise in reading instruction among its key staff and principal investigators. Only two of the 14 key staff and principal investigators could be considered experts in reading instruction. Furthermore, WWC does not specialize in reading. This organization reviews a wide range of educational products, from math to high school dropout programs.
WWC looks at selected research that meets its methodological and statistical criteria, but WWC does not evaluate and analyze how studies were implemented in the classroom or the purpose of these studies. This organization is primarily interested in whether the statistical model for the study was implemented properly.
Dr. Grover Whitehurst, Director of the Institute of Education Sciences, confirmed WWC’s limited scope of evaluation when he stated: “WWC emphasizes the methodological and statistical characteristics of the research it reviews. WWC does not concern itself with a conceptual analysis of the strategies on which educational programs are based or with the degree to which programs are aligned with particular principles.”
Educators know that evaluating the purpose of a study and whether or not a reading strategy was implemented properly is essential in helping to discern if a review of the study has value. WWC’s refusal to include an evaluation of a study’s purpose and the implementation of a strategy within that study is a serious omission in its evaluation process.
Read Naturally’s Experience
Read Naturally’s own experience with FCRR, the University of Oregon, and WWC illustrates the differences between these organizations and their respective reviews.
FCRR and the University of Oregon worked diligently to read the research on Read Naturally and to understand the Read Naturally strategy in the context of best practices research.
In contrast, WWC published the findings of two studies that used Read Naturally materials but did not implement the Read Naturally strategy with fidelity. By not using Read Naturally materials as intended, the Hancock and Denton studies reviewed by WWC did not assess the effectiveness of the strategy.
Flaws in the Hancock and Denton Studies
In the Hancock dissertation study reviewed by WWC, the author of the study told WWC in an e-mail that she did NOT intend to evaluate or use the Read Naturally strategy, and the methods used in the study deviated from the Read Naturally study in many significant ways. In spite of this, WWC still chose to publish a review of this study as though the study adhered to the Read Naturally strategy.
More about the Hancock dissertation
Similarly, in the Denton study reviewed by WWC, the Read Naturally strategy was not implemented as intended. In the Denton study, the Read Naturally strategy was significantly modified in a way that compromised its success. Therefore, WWC’s review of this study does not accurately reflect the effectiveness of the Read Naturally strategy.
More about the Denton study
Conclusion
These factors indicate that the FCRR and the University of Oregon reviews of the Read Naturally strategy are far more reliable than the review from the WWC. To further contrast the leadership and procedures of these organizations, visit their websites:
- Florida Center for Reading Research
- University of Oregon's Reading First Website
- What Works Clearinghouse
We are committed to producing the highest quality tools to help struggling readers. We want you and your students to be successful in using Read Naturally products, and we look forward to our continued work with you in this effort.