On Blogs, Hattie, Effect Size and Statistical Errors
A certain amount of attention has been given to Hattie’s Visible Learning. This is hardly surprising, given its very broad popularity and influence. In amongst all the discussion, you’ll find heaps of praise, tons of claims of irrelevance and a handful of claims that Hattie’s methods and analysis are not wrong-headed but are simply wrong.
I won’t talk about the praise; there’s really not much to say there. The claims of irrelevance or wrong-headedness are interesting, but also not my focus. Some people simply do not believe that statistics have anything interesting or meaningful to contribute to education or educational research. It’s an interesting discussion, but in the end it is hard to avoid the conclusion that statistics are relevant and useful, but are only one small part of what is needed to understand and improve education.
This leaves us the claims of errors. A small blog from a fellow who calls himself “Ollieorange2” has gained notoriety for its strongly worded attacks on Hattie’s statistics. One very provocative title is John Hattie admits that half of the Statistics in Visible Learning are wrong. “Ollieorange2” makes a number of claims, and I’d like to clarify and respond to their substance.
- Hattie uses Effect Size (ES) and Common Language Effect Size (CLE) measurements. These are never used by mathematicians and are therefore suspect. This is a very weak criticism. The reason that mathematicians don’t use them is because they aren’t useful or interesting for doing mathematics. Social scientists use them all the time because they are useful for analyzing social phenomena.
- Hattie miscalculates the CLE statistic. Indeed he does. It looks to me that Hattie probably set up his computer program to do a two stage calculation as two discrete steps, and then simply copied the wrong step output. He or his editors should have caught it. The likely reason that they did not is that it’s irrelevant to the argument of the book, and Hattie doesn’t make any use of the statistic. The CLE is found in several tables in his book, but is only mentioned four times in the text. Two of the mentions point to the idea, and twice he refers to it in the context of conclusions. The criticism is valid and correct. This error did not get caught. The reason it didn’t get caught is precisely the reason it doesn’t matter much: he simply doesn’t use the statistic. He and/or his editors should have had those columns removed because they aren’t part of the book. Hattie got lucky here.
- Half the statistics in Visible Learning are wrong. This is a restatement of #2 and it is simply wrong. If you list ES and CLE as two categories and one of the categories is mistaken, I guess the one-half claim makes a kind of sense. In reality, there are several other statistics in the book, and they do not appear to be incorrect. Further, the great bulk of the discussion focusses on Effect Size, and those seem to be fine.
- Hattie uses Effect Size differently than did its creator. Indeed he does. When Jacob Cohen first published his work on Effect Size in 1969, he intended it to be used in an experimental context. The statistical meta-analysis was not developed until several years later. Hattie’s book focuses on work that was developed after Cohen’s 1969 publication. (Notice also that ollieorange2 dismisses Effect Size as mathematically unimportant, but appeals to Cohen’s work when it suits his purpose.)
The takeaway from all this is that Hattie has his critics. Further, there are mistakes in his book. These should not come as shocks. For the most part, though, Hattie is simply summarizing and synthesizing other people’s work. Hattie contributed very little of the original research that is contained in Visible Learning. It is fair and reasonable to double check the original research in each case; that’s what the Bibliography if for. And it’s fair to have healthy doubt as you read this book, or any other for that matter. But I believe that you can be fairly confident of the general results, and you can use this as a STARTING PLACE as you make educational decisions.
And keep in mind that Hattie uses his real name to stand behind his words, and he answers email questions too.