Wong-Ratcliff et al. on Effects of the Reading First Program
Quantitative research studies prioritize the collection and analysis of numerical data in a scientific process of observation and reasoning used for hypothesis testing (Cohen et al, 2011). This was the goal for a research team of three scholars, Wong-Ratcliff, Powell, and Holland, who investigated the effects of the Reading First program on students in grade one in rural school districts in Louisiana in 2010. Reading First (R.F.) is a federal education program described within the Elementary and Secondary School Education Act (ESEA) that provides funding to Title I schools for literacy improvement. In receiving this funding, though, schools are required to comply with scientifically based reading research (SBRR) practices. This requirement has led to comparisons of gains in student literacy between the R.F. and non-RF schools where literacy instructional practices are not mandated. Following the U.S. Department of Education's publication in 2008 of the Reading First Impact Study, which showed no significant difference in literacy gains between R.F. and non-RF schools, smaller-scale studies have provided additional analysis of the problem. The purpose of this post is to explore one such smaller-scale study and examine the quantitative design its authors used to draw conclusions about the efficacy of Reading First in rural Louisiana.
Research Overview: Purpose, Sampling, Instrumentation, Definitions, Variables
Because R.F. schools implement SBRR as part of their program and funding compliance, the authors set the framework for their study from findings of the National Reading Panel or NRP (National Institute of Child Health and Human Development, 2000). Based upon a review of more than 100,000 research studies, the NRP identified five vital areas of reading instruction. These areas include:
1. phonemic awareness;
2. phonics;
3. fluency;
4. vocabulary; and
5. text comprehension.
In addition, the NRP also characterized effective literacy instruction based upon four pillars. Those pillars are:
1. The use of valid, reliable assessments.
2. The alignment of instruction and materials.
3. The alignment of literacy instruction to professional development
programming.
4. The presence of instructional leadership and coaching.
Based on the research-informed framework of the NRP, the study's authors elected to use DIBELS instrumentation to collect and categorize their data. DIBELS, a UNiversity of Oregon literacy assessment that stands for Dynamic Indicators of Basic Early Literacy Skills, includes a series of subtests (letter naming fluency, phoneme segmentation fluency, nonsense word fluency, and oral reading fluency) that align with the vital areas of instruction that the NRP report identified. Sampling for use with the DIBELS instrumentation was based on non-probability convenience. Wong-Ratcliff and her colleagues identified students in grade one at five different schools in rural Louisana, three of which were R.F. schools (N=130), the remaining two being non-RF schools (N=153). A matched sampling method ensured that participants shared the same characteristics of demographics (specifically geography, SES, and diversity of ethnicity). As a design control, the authors introduced only a single difference between the dependent and independent variables; namely, school participation (or not) in the federal Reading First program.
Research Hypothesis, Research Design, and Data Collection
In their study, Wong-Ratcliff and her colleagues used a quasi-experimental design. While their goal was to establish and support a potential cause-and-effect relationship between the study variables – between literacy gains (the independent variable) and participation in R.F. programming (the dependent variable) – the use of non-probability, convenience sampling necessarily defines the study's design as quasi-experimental rather than experimental. The authors examined the hypothesis that there is no statistically significant difference between the mean gains in literacy for students at R.F. schools when compared to students at non-RF schools. They collected data for their investigations at two separate points in the school year using the DIBELS subtests, which they administered during fall and spring benchmark testing.
Validity, Reliability, Bias, and Control
DIBELS, currently in its eighth edition, was in its sixth edition at the time of Wong-Ratcliff, et al.'s study. DIBELS and the particular subtests the authors used in their data collection are broadly administered by schools across the United States. As an assessment instrument, it is known to be both reliable (consistent across administrations) and valid (accurately measuring literacy skills in K-8 student populations). Both validity and reliability have been reported in numerous peer-reviewed research studies, including Hoffman (2009) and Shilling (2007). In Wong-Ratcliff et al.'s study, no conditions of research were adapted or changed during the course of the investigation, which lends independent support to the validity and reliability of their work.
DIBELS is not without controversy, however, as several researchers who originally participated in creating the instrument also served in a consulting capacity to the U.S. Department of Education during the development of the Reading First initiative. This conflict of interest was outlined by Kathleen Manzo in an article for Education Week in 2005. While there is no apparent conflict of interest between the current study's authors (Wong-Ratcliff, Powell, and Holland) and their use of DIBELS (which they contextualize as a function of alignment with the NRP report), I would like to have seen direct discussion of the DIBELS conflict of interest problem as part of the study's introduction and disclosures.
Brief Overview of Data Analysis
The authors used three statistical procedures in their data analysis. First, to examine the significance of the comparison of means for literacy gains between R.F. and non-RF schools in fall benchmarking, they performed an Analysis of Variance or ANOVA test. This showed markedly significant results (p < .001), though the effect size was small (between .096 and .140). As a result, the authors concluded that the difference in baseline literacy scores were meaningful (The three R.F. schools had an overall higher level of literacy than the two non-RF schools at the start of the study.) Second, following spring benchmarking, the authors used a univariate analysis of covariant or ANCOVA test (i.e., an ANOVA with regression) to examine the independent variables. The ANCOVA showed that mean literacy gains as measured by the DIBELS subtests were not significantly different for students at the R.F. schools in comparison with students at the non-RF schools. In other words, while at the outset of the study, the R.F. students' baseline was higher (incidentally, not demonstrably because of SBRR practices), the net gains achieved by both groups were the same. Students at non-RF schools, where SBRR was not mandated, did not see fewer literacy gains than those at the R.F. schools. Finally, the authors used correlated t-tests to analyze the DIBELS subtests for reliability between their use in the current study and results from the prior year's administration.
Study Results, Conclusions, and Limitations
The results of data collection and analysis from the fall and spring benchmarking periods in R.F. and non-RF schools did not show statistically significant differences of means in literacy measures between the fall and spring testing windows. The authors observed that students in the three R.F. schools had a baseline higher literacy proficiency than those at the non-RF schools (though in all cases, the effect sizes were small – between .096 and .140). Following benchmark testing in the spring, however, there was no statistically significant difference between R.F. and non-RF schools in the overall gains in reading skills observed among students. Results from the study, as the authors predicted, were consistent with Reading First Impact Study (U.S. Dept. of Education, 2008) and similar small-scale studies on the effects of Reading First programming. Extended instructional time for reading together with the effective use of para-professionals and reading interventionists were factors at both the R.F. and non-RF schools in this study. Such factors, the authors concluded, are superior predictors of literacy over participation in R.F. programming and the use of mandated SBRR instructional practices.
Conclusion
The quantitative study that Wong-Ratcliff and her colleagues undertook to investigate the research question of whether R.F. programming at schools is correlated to increased mean gains in literacy measures when compared to non-RF schools showed no statistical difference between the two. While their method was scientific, the authors' use of non-probabilistic convenience sampling necessarily means the study was quasi-experimental. Quantitative designs such as this one, when used in educational research, give voice to numbers and test explanations for phenomena. Such studies are valuable not only for creating a greater understanding of social processes (such as literacy improvement), but also for lending support to aspects of important decision-making processes. In this case, schools in Louisiana have access to more information about the role their participation in Reading First may have on student literacy. The choice to participate in Reading Fist comes with restrictions on instructional practice and demands of the already limited time classroom teachers and school administrators have. What quantitative research studies such as this one provide are additional data points, statistically validated descriptions of phenomena and support in a decision-making process. While the question of whether a school in Louisiana should or should not seek a grant award through Reading First is not one that this study answers, the research design and statistical procedures the authors followed mean it can and should serve to inform the broader decision-making process of literacy programming.
References
Cohen, L., Manion, L., and Morrison, K. (2011). Research Methods in Education (7th
ed.). Routledge.
Hoffman, A.R. (2009). Using DIBELS: A survey of purposes and practices. Reading
Psychology, 30, 1-16.
Manzo, K. K. (2005). National clout of DIBELS test draws scrutiny. Education Week,
25 (5), 1-12.
National Institute of Child Health and Human Development, NIH, DHHS. (2000).
Report of the National Reading Panel: Teaching Children to Read: Reports of
the Subgroups (00-4754). U.S. Government Printing Office.
Schilling, S.G. (2007). Are fluency measures accurate predictors of reading
achievement? The Elementary School Journal, 107 (5), 429-447.
Wong-Ratcliff, M., Powell, S., and Holland, G. (2010). Effects of the reading first
program on acquisition of early literacy skills. National Forum of Applied
Educational Research Journal, 23(3).