I’ve defended Hattie from wrong-headed attacks, acknowledged a rather gruesome, but fortunately benign statistical error, and spoken in favour of his general project. I like Hattie’s work. But that doesn’t mean that it’s without some problems and dangers, both technical and practical.
First, some technical concerns. Empirical research into teaching and learning is always dicey at best. You take a group of children who come to the research with widely differing skills, attitudes, experiences, motivations, etc. etc. and you do something with/to them. For the sake of argument, let’s say we give them a reading program for a month. To measure the success of the reading program we’d probably do a pretest and posttest, and if funds allow, a second posttest some time down the road. The obvious problem is that each child has countless other causal factors in her life—parents, friends, trips, health, etc. To try to compensate for all this, we need bigger and bigger samples, hoping that the random differences cancel each other out. Because education research is not deeply funded, these huge sample controlled trials are rarely conducted. Enter the meta-analysis.
To perform a meta-analysis, the researcher takes other people’s research (let’s keep the reading program example alive), selects them based on some criteria (meant to eliminate low-quality trials), and then sorts them according to some other criteria (say, phonics, vs. whole language, vs. “balanced” approaches). But things are already getting hazy. It’s not at all clear that the experimental conditions in the array of studies are comparable. It’s almost never the case that the same measurement tool (reading tests in this case) is used from one study to the next, and it’s not even clear that the same theoretical understanding (i.e. reading) is being applied to each study. In spite of this, the meta-analysis is a respected tool for solidifying conclusions in social science trials.
Now, let’s add another layer to this. Hattie (and he is not alone in this approach) takes meta-analyses and “synthesizes” their results, trying to combine the results of several meta-analyses that look at reasonably similar phenomena. Well, you can guess where this is going. The synthesis does, indeed, increase the sample size, but it also greatly amplifies the differences between the interpretations of each individual test score.
Do I think that meta-analyses are valuable? Absolutely. Do I think that Hattie’s synthesis is valuable? Absolutely. Do I think that you can take any of Hattie’s findings and apply them directly in a classroom, school, or school district? Not many, I’m afraid. And this leads to my practical concern about Hattie.
Visible Learning is deceptively easy to read. To his credit, Hattie explicitly warns against over-interpreting his results and against using the book as a template for educational reform. Yet, it’s happening. Teachers, administrators, school authority leaders and curriculum developers are all drawn to the siren call of easy conclusions. “What should we do?” asks one. “Let’s check Hattie,” suggests the colleague. I don’t anticipate disaster—after all, as Hattie notes, almost everything works in education. But it’s not a responsible way forward.
I like to use Visible Learning as my first stop. If I’m interested in reading programs, I look first to see what Hattie’s general findings are. Next step is the Bibliography. It is crucial that I ensure that Hattie has correctly interpreted the source meta-analyses. After that, it’s to at least some of the individual studies that the meta-analyses reference—no guarantees that they got it right either. Once all this is in place, it’s time to consult with local experts and experienced teachers. Research findings, combined with fiscal reality and practical experience and beliefs all become part of a giant compromise. Action is rarely a matter of choosing the best of everything; it has to be achievable.
When all the groundwork is done, a plan needs to be in place, including implementation and, most importantly, an evaluation. But these are all topics for another day.