What Works Clearinghouse (WWC) includes the Arvans (2010) study in its intervention report for Read Naturally programs. In this report, WWC asserts that the Read Naturally group in the Arvans study does not demonstrate significant gains in oral reading fluency. However, an analysis of the Arvans study reveals that the Read Naturally group’s fluency gains were quite significant.
At the end of the eight-week study, the Read Naturally group had a large effect size of .81 for fluency. The control group had a moderate effect size of .57 for fluency. This effect size difference of .24 in eight weeks is significant, especially when extrapolated over a school year.
The Arvans study may also be analyzed using the Hasbrouck-Tindal Oral Reading Fluency Norms. According to these averages, third-grade students at the 50th percentile have an average weekly improvement in fluency of 1.1 words correct per minute (WCPM). An analysis conducted on the Arvans dataset estimates that the performance of the Read Naturally group would be significantly greater than this.
Finally, it should be noted that Read Naturally programs were used extensively at the school before the Arvans study was conducted, and the teachers were well versed in the Read Naturally strategy. Students in both groups had used Read Naturally in the two years prior to the study. Additionally, some students in the control group used Read Naturally during the study. These factors help explain why both groups demonstrated excellent gains in fluency, and they indicate that the control group was not demonstrative of a true control group.
Clearly, WWC underestimates the effect of the Read Naturally intervention on students in the Arvans study. For more details, read the technical analysis of this study below.
Ethan R. Van Norman, M.A., performed an analysis of the Arvans study that is similar to what appeared in the Christ and Davie study (2008). In the Christ and Davie study, the authors first calculated a slope estimate from three time points for each student in the control group and the Read Naturally group. The slope estimate represented the number of words read correct per minute (WCPM) improvement per week. The mean and standard deviation of slopes were then calculated for each group. The percent of improvement of the Read Naturally group in relation to the control group was calculated. After this, the authors used the percent of improvement and applied it to an aggressive rate of growth (1.50 WCPM improvement per week). That value and 1.50 were then multiplied by 36 (the typical number of weeks in a school year). The difference between these two values was interpreted as a hypothetical effect if the Read Naturally intervention was delivered across an entire school year.
Similarly, on the Arvans dataset, slope estimates were calculated for each student from two observations eight weeks apart. The mean slope value for the Read Naturally group was 2.92 WCPM improvement per week (SD = 1.54) compared to the control group, which had a mean slope estimate of 2.24 (SD = 2.36). The .68 difference in mean slope for the Read Naturally group represents a 30% improvement over the control group. Assuming an aggressive rate of growth of 1.50 WCPM for typical students, a 30% increase would translate to a 1.95 rate of growth for Read Naturally students. Extended across 36 weeks, this represents a net increase of 70 WCPM for a Read Naturally student, compared with a 54 WCPM increase for a non-Read Naturally student.
Although not ideal, slope estimates from two time points have been used to summarize growth in previous CBM-R research studies.
See: "Curriculum-Based Measurement of Oral Reading: An Evaluation of Growth Rates and Seasonal Effects Among Students Served in General and Special Education," Christ, Silberglit, Yeo & Cormier, 2010.
Ethan R. Van Norman also did an analysis on the Arvans dataset to extrapolate growth using normative values. For third-grade students, weekly growth estimates for students in the 50th percentile typically approximate 1.10 WCPM per week. Assuming that the Read Naturally group has a 30% improvement over the control group and the intervention is delivered for 36 weeks, a student in the 50th percentile, on average, would improve at a rate of 1.43 WCPM per week. After 36 weeks, this would translate to a 51 WCPM improvement for a Read Naturally student and a 40 WCPM improvement for a non-Read Naturally student. This is a substantial difference.
Please let us know what questions you have so we can assist. For Technical Support, please call us or submit a software support request.