Dear Tom.. The things we think and do not say.. Thoughts on the populist appeal of deriding data.
I sit here at my laptop starting to write a response to a blog post I have just read in the full awareness that I am about to commit a form of professional twitter/blog based faux pas. On Sunday I woke to find no fewer than 3 separate emails from 3 colleagues (2 of whom were from my own school)passing on a blog post about the use of data in schools and the wider educational system. To get 20+ RTs on twitter about something which is interesting is far from unusual but to get three emails within a few hours about one blog post is fairly rare for me and this clearly added to my sense of interest.
I eagerly pulled up the blog post from a well-liked and respected fellow colleague on twitter who writes brilliantly in an entertaining and lucid style which makes for an engaging read. Tom Bennett’s article ‘The Bones Have Spoken: Is Value-Added Crystal Bollocks?’ which can be found on his excellent blog The Behaviour Guru has clearly resonated well with many in the education community who are sick and fed up with the apparent over and misuse of data within schools and the educational system. The problem I suffer having read the article 5 times now over 4 hours is that it did not resonate with me. I found the article riddled with, in my opinion, a number of pedagogical and statistical flaws which I feel do more harm than good in the debate about how and why we use data within schools. It is in this context that the faux pas I am about to commit is to be critical of a fellow colleagues writing knowing that this is one of those things we think and do not say.
Here are a few of my issues with the article (with quotes from the article clearly shown) which I put to Tom Bennett in the friendly spirit of open discourse.
1. ‘Stand easy citizens- schools will be exhibiting their Contextual Value Added scores from now on, not unlike a baboon, presenting its ghastly floral undercarriage’. This is almost the exact opposite of what is happening. The new government has retired CVA from this current set of results which will be published this week. Whatever my own views of the new Government’s endless and bewildering range of proclamations and ‘Claudio Ranieri style’ tinkerman approach (trust me my own views are caustic of Gove and his buffoons) it surely makes sense to make the main focus of your argument something which you understand. Sifting through twitter feeds MG Harris questions you, asserting that she thought CVA was being abolished and the conversation is shown below.
An expansion of the link above is http://en.m.wikipedia.org/wiki/Contextual_value_addedand you use this to confirm the prominence of CVA. Whilst I do not know how many other sources you have used for your understanding of CVA I would hope it is not just Wikipedia. I am a big fan of Wikipedia and don’t buy into the whole ‘it’s inaccurate’ arguments which people trot out but as an Historian I am also a fan of using a range of sources. An instant Google search such as http://www.tes.co.uk/article.aspx?storycode=6086535 would have confirmed that CVA had been abolished as MG Harris asserts.
2. I know this sound awfully rude but the whole article lacks any coherent understanding of the term CVA. The title of the article includes the words ‘value added’ which is an entirely different measure than CVA and is one of the measures the government will be keeping but with some adjustment. Furthermore, in the next paragraph you state that, ‘ my CVA is, thankfully, bulletproof, fireproof, and susceptible only to Kryptonite, and I only say that so you don’t think I’m a bitter victim of its diabolic engines.’ The key issue here is individual teachers, students, classes and even school leaders and Headteachers don’t have a CVA. A school was given a CVA as a measure of performance but individual teachers and groups don’t have a CVA. The whole article seems to confuse wide ranging measures of performance and reflection which for me is the heart of the issue.
3. ‘In many ways picking a fight with predictions is an easy task, because I’m attacking the belief that we can tell what is going to happen about things that have not happened yet.’ One would think that ‘attacking predictions’ is indeed an easy task but again the incoherence of the argument is apparent. CVA is not a prediction it is one attempt to retrospectively measure performance. You do not know what your CVA needs to be and whether you will hit it with predictions as the measure is always adjusted. You can simply use it as a tool to see how well a cohort and subset within that cohort performed relative to other educational institutions using a Government measure. When you speak of predictions you are perhaps talking of FFT estimates or some other system which you reference within the article. The problem is you jump from measure to measure so frequently acting as if all the terms are interchangeable when this is in no way the case.
4. ‘What am FFT? They’re an organisation that sell data’ I am assuming that the use of am as the second word is actually are but have not doctored your own words. My issue here is the insinuation that the whole purpose of using data is a means of institutions making money as if it is all a big capitalist hoax. I may of course have misinterpreted this as you do not directly say it but I believe many people will form the opinion that you are making this populist assertion. A link on the Fischer Family Trust site http://www.fischertrust.org/about.aspx shows that they are actually, ‘an independent, non-profit organisation which is mainly involved in undertaking and supporting projects addressing the development of education in the UK.’ Whilst I am not aware of their funding mechanism given this context they do not have shareholders or a need for profit. The Government may fund them as they would any organisation (again I do not know how they are maintained) but I think it is unfair to suggest they ‘sell data’.
5. ‘It’s de-professionalises the whole role of the teacher. Excuse me? You want to say what one of my students is probably going to get this year….and you haven’t even met them? The statement above is exactly how I or any professional would feel if it were true. My issue here is that I think at the heart of the problems with our use of data is the insane way many schools have used it, throwing away any pedagogical underpinning in order to create quick and easy targets. Nobody who produced this data ever suggested it be used to create de-facto targets or suggested the role of the teacher is not the most crucial factor in understanding their potential within any given subject and examination. Having met Mike Treadaway who is Director of Research for the Fischer Family Trust I confronted him 3 years ago about the way in which their data was being boiled down into instant target setting systems by the vast majority of schools. It is fair to say he was horrified and the terminology you later refer to on the use of FFT data as estimates and not predictions/targets largely stems from that conversation. The simple fact is that we only have our self to blame as educators and school leaders if we create a mechanistic, ill-conceived and downright stupid approach to understanding data in the way that most schools have. Whilst it is easy to attack Governments and Quangos the answer to this problem lies much closer to home.
6. ‘You know what my expectation of my children is? An A. For everyone. That’s the target I set myself, and if I don’t get it, well, I try again next year.’ I think you speak for all of us in that as teachers we passionately aspire for children to achieve the best they can in every aspect of school life. At the same time the idea that telling a student with very weak literacy skills and a delayed reading age that a D is discouraging is another piece of educational madness that we have brought upon ourselves. How the hell did we get to a place where an examination that grades A*-G is only really valuable if everyone attains at grades A*-C? I undertake lots of formal and informal examinations of my capability in my professional and personal life. In some I excel, in others my performance is perhaps middling and in others I recognise I have major weaknesses. That this is the case is called real life and applies as much to students as any of us.
The idea that telling every student they are all capable of getting an A is genuinely laudable and in some I ways do this myself when I tell students that on a given question all students are capable of scoring an A*. This however is not without its own challenges and is not, in my view, the panacea to estimates. When students are highly motivated by an inspirational teacher who tells them they can achieve anything there is genuine value but surely this is often tempered by the despondency which greets the same student who scores a D in their mock exam and does not understand what went wrong. If this student started Secondary School with a reading age of 7 Years and was below the level for KS2 English Tests it may be an incredible outcome that they made progress to get to a D in a final examination. For me the value in estimates comes from giving a sense of the possible and in the hands of educators can be used skilfully to motivate and inspire and when used in a crass manner it has the power to demotivates and makes the child feel they are just part of a big machine. Again the choice and opportunity lies with us as educators and schools leaders and our lack of understanding lies at the heart of misuse when it comes to the notion of target setting.
7. ‘Here’s a thing: what does it even mean to ‘aim for a C, or a B’? Have you ever seen a kid revise, and try to get a B?’ Maybe we work in different ways or understand children differently but I have seen a student improve over the course of two years during a GCSE. You assert in your analogy that a student just runs and does not have a target in mind when they do this. What happens if they want to start competing for their school and need to make a certain time to enter a specific competition. Would they not run, find out their current time and then ask their teacher, ‘how can I go faster’. If the teacher explains that they need to use their arms better when running and shows them how to do this to try and cut half a second of their time is this not a valuable exercise. It is a target after all and one which the child may appreciate when they achieve a goal they really wanted to. After all I am sure we both recognise that they are on their Xbox of PlayStation constantly setting themselves targets (perhaps both realistic and not) every day of the week. For me aiming for something is not evil or impossible. It can be motivating or demotivating. This in turn depends on context and that is the craft of the teacher. Some students I meet just want to ‘try their best’ and that is fine with me whereas others look at chances graphs, smile and say, ‘I want to try for a really difficult target of an A sir’. At the risk of sound repetitious, the information is not the problem here, it is the methodology many use.
As I said earlier, I hope that this response will in some way be seen as a useful discussion between us both and the various educators who may chip in. After all I am sure we both have the same desire for students to love learning and achieve everything they want. I like you believe there is a chronic issue with the use of data. At the same time I believe the use of data itself can be inherently valuable in the right context. When used poorly and in a damaging way I genuinely believe we only have ourselves to blame.