In a recent article for Alliance Magazine Caroline Fiennes and Ken Berger say a mea culpa for the impact revolution in social programmes and suggest two ways in which we ought to work in the future in order to do a better job on evidence.

Their first recommendation is that social programmes should be consumers rather than producers of evidence.

Their second is that impact should be assessed by independent experts.

Fiennes and Berger come to these conclusions after analysing the ‘system’ in which social programmes operate and noting a set of perverse incentives and an absence of appropriate skills. Their response is to take the bulk of evidence gathering out of programmes altogether.

There are a couple of assumptions at work in that argument that deserve to be analysed in detail.

One assumption is that independence will get us away from the mess and muddle that comes with self-evaluation. This implies that independent evaluators are at one remove from the ‘system’ of social programmes and therefore immune to its incentives.

That ought to give us pause.

We need to ask blunt questions about who is funding these independent experts to secure their immunity from influence. An academic funded by her university to produce a systematic review of, say, evaluations of interventions to reduce reoffending is perhaps unlikely to be influenced by those running the interventions on the frontline. It can be reasonably argued that she is ‘looking in’ on those social programmes in a disinterested way.

An evaluation consultancy retained by a provider (or a group of providers) of those same interventions is in a very different position. It is hard to see how that consultancy is outside the ‘system’ in any meaningful way at all.

More fundamentally, the idea that research of this kind is ever free from broader influences and judgements is almost certainly a false one. There is a thriving school of research in science studies that argues that all science, from quarks to Maxwell’s equations, is contingent, and might have gone in entirely different, but equally productive, directions under a different set of circumstances. This school of thought outlines a ‘robust fit’ between theories, models and apparatus that settles into an accepted narrative about a series of phenomena. This all takes place amongst “practices, bodies, places, groups, instruments, objects, nodes, networks”^[1]. Call it a ‘system’, if you like.

There’s no getting out of these kinds of ‘systems’, or even of standing outside them temporarily. What we can and should do is remain alert to them and how they operate. This requires reflectiveness and transparency. On its own, independence is likely to be a spurious guarantee of the quality of evidence of social programme effectiveness.

Fiennes and Berger also diagnose an absence of the required skills and resources to produce valid evidence within social programmes themselves. This is where they make another assumption, this time overtly:

“… (our definition of good impact research) requires knowing about sample size calculations and sampling techniques that avoid ‘confounding factors’ – factors that look like causes but aren’t – and statistical knowledge regarding reliability and validity.”

Whether or not social programmes have this kind of knowledge is an empirical question; I’m prepared to accept the authors’ statement that they don’t. I don’t so readily accept that this is the only kind of evidence worth collecting.

This assumption, much like the assumption about independence, appears to be aiming at the maximum amount of ‘purity’ in the data produced by social programmes.

The ‘purity test’ is one way of looking at measurement, but it isn’t the only one.

Measures don’t have to be perfectionist to produce worthwhile information. We may have become used to thinking of measures in perfectionist terms, as absolute and defined according to fixed convention, but this is a relatively recent development.

Measures can be, and for a very long time were, representational. They were a means, not of holding ourselves up to a conventional and ‘validated’ standard, but of helping us to achieve our ends. Instead of saying a field consisted of X number of hectares, we measured in terms of days of labour or probable yield. Agricultural measures of this kind were predictive of a desirable outcome for us as labouring human beings. There is a case for saying that representational measures of this sort would be infinitely more useful to social programmes, and for suggesting that those best placed to implement them, and collect data using them, are within those programmes, not outside them.

Statistical research methods of the kind advocated by Fiennes and Berger have status and respectability but that doesn’t necessarily make them the best, certainly it doesn’t make them the only, ones we can use. There is nothing to stop us using representational measures to assess the effectiveness of our social programmes. Indeed, the authors hint at this in passing by approving the collection of feedback.

Using measures that are responsive to the messiness of our ‘systems’ and the reality of human aspiration to account for our impact, now that would be truly revolutionary.

Genevieve Maitland Hudson is a researcher and consultant. She works with the consultancy Osca. Email gmhudson@osca.co