Evidence, Anna. Where’s the Evidence?

Earlier this month, the C.D. Howe Institute published a polemical paper by University of Winnipeg mathematician Anna Stokke. The paper explores the politically-charged question of What to Do about Canada’s Declining Math Scores. It’s one of those funny papers, written by an academic wandering outside (or to the fringes) of her field. Stokke has some insightful things to say, followed by some absolute howlers. Let us begin with the “problem”: on the mathematics components of the Trends in International Mathematics and Science Study (TIMSS) and Programme for International Student Assessment (PISA) tests, most Canadian provinces have shown a statistically significant decline over the past decade or so. The first two questions that come to mind should be

1. Is it true?
2. Does it matter?

Stokke states that #1 is true and provides some reasonable evidence to back this up. I agree with her basic position that Canadian scores have declined on these tests. I haven’t seen a compelling case for why this is the case, but it definitely is borne out by the data. It is worth noting that Canada has declined from being near the top of the international list in 2003 to being near the top of the list in 2012. The decline is real, but it’s from height to height. Stokke assumes the answer to #2. I won’t go into #2 deeply, but I will note that it is far from obvious that the tests measure the things we value in mathematics education. There are countless possibilities for what we could value in school (computation, reasoning, geometry, algebra, number, problem solving, etc.) No curriculum can do full justice to all of them, so choices have to be made. I have no quarrel with the material emphasized by PISA and TIMMS, but I believe that any political action based on the scores must take such niceties as these into account. Let’s also note that K-12 education is a provincial matter in Canada: each province creates its own curriculum, assessment standards and teaching professional standards. Nation-wide decline becomes a problematic issue to study because each province is teaching a different curriculum. There are, of course, similarities because change doesn’t occur in a vacuum. Provincial ministries of education do talk together and have a pretty good idea of what each other are doing, and American publishers and academic trends tend to influence Anglophone curricula. I am uncertain of the influences on Francophone curricula. So far, we have a few quibbles, but the argument is fine. International scores are declining. Next comes the bogey du jour: “Discovery-based Instruction” whatever that is. The issue that Stokke pushes is the difference between structured “direct instruction”[1] and “discovery learning[2]”. If we accept Stokke’s crude dichotomy here, the empirical evidence is pretty strong. In terms of developing skills, teacher guidance is critical. This is why teachers are necessary. Well-structured instruction, with appropriate questioning, feedback and revision to assist student understanding are crucial to student achievement. So if the goal is to teach students to perform long division, or to multiply polynomials, or to solve quadratic equations, clearly worked examples under the direction of the teacher are clearly and decisively shown to be the best choice for student instruction. Stokke makes reference to some of the empirical literature on page 4 of her report, and I have no quarrel with her summary or with her interpretation of the results. (On a side note, Stokke’s footnote 5 on that page is an absolute joke. The “well-informed journalist” is David Staples, who is well-intentioned but hilariously ill-informed on the issue. The claim that phrases are interchanged “to avoid criticism” is at best a wild guess, and at worst yellow journalism.) So where’s the complaint? It is undeniably true that inquiry forms the basis of what is valued in K-12 education in Canada today. Science classes should and do have laboratory experiments. Social studies classes should and do include source-document analysis and open inquiry into relevant issues. How could one study literature without inquiry? The same goes for mathematics. Students need to acquire the basic tools of mathematics, but they also need to learn how to mobilize these skills to explore questions both practical and theoretical. If there were no room for inquiry in mathematics, Dr. Stokke would be out of a job. Which leads us to the million dollar questions.

1. Which skills must be mastered?
2. Which skills should be developed to the point of acquaintance?
3. How much inquiry is desirable at each stage of the student’s development?
4. What do we ultimately hope that our students will gain from their mathematical education?

Notice that these questions are only slightly empirical. At their hearts are values. What do we think is important and why do we think it’s important? Stokke shows some sensitivity to this problem, but she effectively abandons her project and jumps into speculation and assertion. Starting on page 9 she begins spelling out what she thinks is important for each grade of the curriculum. Stokke provides no justification for this, she makes no appeal to research literature on child development; she somehow just knows what every child should learn, and when they should learn it. Frankly, this is just spitting in the wind. She’s one voice in a multitude. I agree with some of her suggestions, but not all. (But then, I only have 20 years of classroom experience and a doctorate in education; what do I know about the matter?) And then it starts to get weird. Stokke writes

One way to redress the balance between instructional techniques that are effective and those that are less so would be to follow an 80/20 rule whereby at least 80 percent of instructional time is devoted to direct instructional techniques and 20 percent of instructional time (at most) favours discovery-based techniques.

Where does this come from? For which students? At what grades? Where’s the evidence? Stokke is relying on her intuition to make an 80/20 prescription. Is she joking? On the positive side, Stokke recognizes that there’s more to the issue than the simple-minded “discovery vs direct” dichotomy suggests. But to make an ad hoc recommendation of this type is to be irresponsible. It gets worse. Stokke generalizes from an American study of first and third grade teacher knowledge, suggesting that Canadian teachers should all be given regular licensure examinations. Seriously? You’ll need some evidence that

1. The American study applies to Canada.
2. What’s true for first and third grades applies to all grades.
3. Licensure exams will actually improve classroom practice.

Ultimately, Stokke’s report reeks of the well-intentioned dilettante. That Stokke loves mathematics and is a believer in quality mathematics education is evident. Unfortunately, it’s also evident that she jumped on a bandwagon, read a small amount of research in the area and picked up a contract from a “think tank” to join a public debate. As an insider, I’m not thrilled with the current state of mathematics education in my country. But also I am convinced that single-cause arguments that fail to acknowledge differences between students—not even considering that age matters—do nothing to clarify our issues, but serve only to support loud and ill-informed public shouting.

[1] Not to be confused with an American product that uses this name. “Direct instruction” should be read generically as any teaching in which the teacher strongly guides or directs learning throughout the process.

[2] Despite the claims of Stokke and a few other public protesters, you simply won’t find this term or anything like it in current curriculum documents in Canada. Individual teachers are undoubtedly using “discovery” methods to some degree or other, but there is nothing in the curriculum to mandate them.