Why does data use punch above its weight when it comes to workload? (and what can we do about it?)

The following is (slightly expanded) transcript of a ‘7 minute research briefing’ I’ll be giving at SSAT this evening. It’s informed by a research project I’ve been involved with (in conjunction with NCTL and the DFE before they were same thing). Thanks to all those who agreed to be interviewed and whose responses are included (directly or indirectly) below.  I’ve also benefited from the expertise of a number of experts- in particular Ruth Dann (at University College London) @Sam_Sims_ and (via the wonders of the interweb)@drbeckyallen. Any erroneous interpretations of their work or guidance are entirely my own!

What can teachers actually do?

  • Explain things
  • Ask questions
  • Direct behaviour (in lesson) and to some extent outside of it.

Or ‘just teach’ (as @Teacherhead writes) The rest is arguably just noise. More of this later.

Data management processes and systems take significantly less teacher time than planning and marking.  However they can punch well above their weight in relation to workload. In fact, workload and the hours which teachers report working are less closely related than you might imagine.

Recently published work by Sam Sims for the DfE showed little correlation,  between reported hours working and perception of how reasonable workload was.  Arguably, we need to stop focusing solely on reported hours and start thinking about how teachers perceive the nature of the demands placed on them in order to get a better grip on the issue.

Crawford et Al (2010) provide a useful model for understanding this. It looks at the relationship between two types of demands and levels of engagement + burnout. (the original research is paywalled but worth reading if you have access: there’s a longer explanation from available here.

  • Challenge demands: May be difficult or time consuming but are linked to my core purpose.  They are associated with increased engagement (and v. slight increase in burnout)
  • Hindrance demands: Tasks which seems to hinder my ability to achieve my core goals. They are associated with increased burnout and decreased engagement.

The problem is it’s easy for those of us in leadership roles to introduce demands which (on paper/digitally) at least, link to our core purpose – driving up standards/progress etc. But which actually seem a hindrance to teachers.

This is because we can fail to see the complexity behind the summaries and reports which are now a few mere key strokes away. They give the impression that we can know things which are probably not knowable. It’s easy to see why we might want to know.:

If you’re going to face some fairly stark summative judgments about how you’re performing as a school, you know, it’s no good being in the dark for two years and then finding out how well, or otherwise, your examination groups have done. You need to be on it, you know, right the way through the two years. So I think it’s just a chain of accountability. (Headteacher, secondary school)

A key finding for our data use interview process was that dichotomy between what is useful for managers and useful for teachers wasn’t just an SLT vs classroom teacher divide. It was a role divide. People with both roles, expressed both views. In the same interview. In the same answer even. As seen in the example below.

Q. ‘How useful do you find your data management system’?

As a leader it’s really good ’cause you can click on whatever you want and it will give you whatever you want, so it gives you really good reports, and if you want it summarised – the summaries are really good. As a teacher it’s quite laborious and it has steps and statements, so the actual steps are difficult to… I don’t think they show progression particularly well, would be the best way to say it, I think. Useful for management roles if not the teacher. (Senior leader, primary school)

The gap between these two roles probably exists for a number of reasons. Clearly not all responsibility lies with those leading individual schools. There are systemic issues in place here too. However, regardless of blame, school leaders can challenge practices and process which are not helping staff or students.

How?

Firstly by ensuring that leadership teams have a grasp of the reliability and validity of the data which they are acting upon. Basic statistical literacy is key.  We could start by engaging fully with these three claims. I’m happy to argue the case regarding any of them afterwards.

  • Individual grade predictions are not particularly accurate. The more precise we try to make them, the less accurate they will be. Fine-grade predictions will be even less accurate.
  • Flight paths are not accurate for individuals. The more precise they are, the more wrong they are likely to be. There is no should about it. Pupils with lower KS2 grades, for example will gain lower GCSES – on average. But numerous individuals with low KS2 grades will outperform others with high KS2 grades. This graph helps illustrate the reality behind the ‘average progress’ line (full report here):

national example education data lab report

  • Annual exam results are naturally quite variable AND (in some cases) remarkably unreliable. This is report, for example shows just how unreliable exam grading can be. (whole report available here)

box and whiskers

Failing to understand the above, makes it likely that we’ll pick ‘the wrong kids’ in order, potentially to conduct an ill-considered intervention premised upon the idea that they should be progressing more closely in line with an aggregated average with pupils who share a couple of prior assessment scores with them. Or we could make knee jerk reactions to grade variations which might not mean what we think they mean.

It’s actually very hard to know what sort of teaching and learning is happening. It’s even harder to predict the future based on the little we can know.

As Dr Rebecca Allen put it in this year’s Caroloine Benn Memorial Lecture , whilst technology and tracking systems may give a different impression.

‘Auditing teaching and learning isn’t really possible… a headteacher cannot know what is going on in a classroom, unless they are there.’

Systems which ignore the tenuous and limited reliability of the data on which they are based risk turning off staff. As one of our interviewees pointed out:

If you make it overly complicated, or you try and drill too deep with it, you actually draw faulty conclusions…you lose half the staff who just switch off after a while. I think there is a risk with data. (Headteacher, secondary school)

Teachers daily live with the uncertainty and unpredictability of actual teaching. It’s part of what makes the job so rewarding (and frustrating). Subjecting staff to a system which overlooks this uncertainty is doing their professionalism a disservice and probably has a negative impact upon their experience of being a teacher.

If you make it overly complicated, or you try and drill too deep with it, you actually draw faulty conclusions…you lose half the staff who just switch off after a while. I think there is a risk with data. (Head teacher, secondary school)

Teachers can

  • explain
  • ask questions
  • direct pupil behaviour

Terms like intervention, target, focus, boundary pupils etc,  can easily become mere lip-service to accountability measures,  or (more likely) synonyms for worry,  unless they link directly to the above actions. Explaining something differently (or again). Asking different questions. Directing helpful behaviours (encouraging pupils to act in ways which will help them learn).

It probably takes a subject specialist to carry out the first two actions (explaining, asking questions).

At which point the line between ‘intervention’ and teaching is blurry at best. (Though nothing is stopping us from providing specific staff with time/resources to teach specific topics to specific students who may have not understood something the first time.)

Otherwise, at whole-school level we’re probably best focusing on behaviours.

For example: Who amongst your year 11s has not adopted a revision plan for their mock exams? Could form tutors identify who hasn’t?  They would make a valid ‘intervention’ group. It would be tough work, but potentially helpful (and possibly even rewarding). If you could reduce the numbers from 70 say, to a handful of pupils then you’ve probably helped improve chances of academic success.

I will tweet link to a version of this – with additional support/links shortly.

Post-script: 

A few weeks ago Becky Allen did an incomparably more informed and eloquent job than I have managed here.  If you haven’t read her speech (or @miss_mcinerney ‘s tweeted version) I can’t recommend them enough.

I shared an extract of Becky’s speech with a colleague (psedonym Jane) who has left teaching. Her response was as follows:

I’ve just read the whole speech.  I love her!  It articulates precisely why I left primary teaching.  After having had a professional career elsewhere before teaching, I could not believe the level of intrusion and lack of trust in what I was doing, and the amount of work beyond the classroom – if you have never done anything else, then I suppose you might not know better. It’s a real shame because I loved it and it’s truly rewarding, but when you don’t have time to see your own kids you have to draw a line.   (Jane, ex-primary school teacher)

What was Jane’s ‘professional career elsewhere’ before teaching? She was a senior executive in the civil service. Reporting directly to the leader of the House of Commons. I’ll leave you to draw out any ironies there. And I’ll let Jane finish this post….

…it is very difficult to for a school to put its head above the parapet and say we are not going down this path.  Anyway, I’m going the buy the book…but will anyone else take any notice?

 

 

Advertisements

2 thoughts on “Why does data use punch above its weight when it comes to workload? (and what can we do about it?)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s