Against impact

1. Introduction

An assessment committee composed of fellow academics recently applauded my faculty’s research policy for emphasizing the social impact of research. The committee called this shift in policy ‘progress’. ¹ Since then, my faculty has convened an ‘impact board’, hired an ‘impact developer’, and asked all colleagues to regularly reflect on the impact of their research. For some time, the collective of Dutch universities has been organizing Impact Festivals during which Impact Teams hand out Impact Awards. ² Funding agencies across jurisdictions have added considerations of ‘societal impact’ to their criteria for assessment, and some funding schemes, like the EU Missions, are fully geared towards bringing ‘concrete solutions to some of our greatest challenges’, and to creating ‘real and lasting impact’. ³ My faculty’s revised Quality Indicators for Legal Research now state that the importance of impact is ‘self-evident’. But the seeming ease of this proclaimed self-evidence is false and forced. Too many are the problems that quickly reveal themselves.

Granted, what is now called impact had other names before, and on first blush, little may have changed. While crude and managerial, the lean notion of ‘impact’ even strikes me as preferable to the odd ‘valorisation’ or the clunky speak about researchers’ ‘active contribution to knowledge utilization’, which the Dutch Research Council (NWO) used to ask for. As different terms have come and gone, there is a continuity in the underlying theme: a concern about putting academic knowledge to good use within society. Who could possibly disagree? Not me. But what now comes under the notion of impact ends up equating societal relevance with plain practical usefulness, and that is a problematic shift in more than one way. If the push towards societal relevance once had a progressive potential, it has today become reactionary. More than anything else, it has become a mechanism of control. To see that, we need to turn from documents of research policy to a brief history of the theory of science.

2. Science and society

Bear with me just a moment: In the wake of the scientific revolutions of the sixteenth and seventeenth centuries, commitments to knowledge and truth, on the one hand, and societal relevance, on the other, have stood in a harmonious, symbiotic relationship. Just think of the now-clichéd slogan attributed to Francis Bacon that knowledge is power (scientia potentia est) and then of Cartesian rationalism, which would measure, explain, subjugate, and exploit nature to man’s benefit. More recent critiques have problematized the radical opposition of the thinking subject (res cognitas) and the inanimate object (res extensa), and the unflinching anthropocentrism it heralded, as a root cause of the climate crisis. ⁴ But I am getting ahead of myself. For now, it is important to note that the societal relevance of scientific research was, for a long time, a non-issue. It was just presumed, no questions asked. Stronger still, the benefits of science were deemed to be stronger the less society interfered with its operation. Fortified by this unquestioning confidence, science broke ever-new boundaries in an increasingly ‘disenchanted world’ (to also name Max Weber’s short-hand phrase that is difficult to pass over here).

But as science professionalized and specialized in autonomous and self-referential ways, academics themselves became weary about their own trajectory as science arguably lost its connection with societal, real-life problems. It was for that reason that, by the 1930s, Edmund Husserl spoke of the ‘crisis of European sciences’. ⁵ My own go-to place for questions of the relationship of science and society remains Max Horkheimer’s endlessly re-readable essay ‘Traditional and Critical Theory’ (1937). For Horkheimer, once a student of Husserl, René Descartes’s ‘Discourse on Method’ stands in for traditional theory – as a scapegoat, really. Critical theory, by contrast, is exemplified by what Karl Marx was doing. Horkheimer rejected traditional theory for its unencumbered positivism – its mistaken belief that scientific discovery proceeds through reason and logic based on sensory experience, whereas scientific work is really enmeshed in social structures that condition, shape, and, yes, construct what science purports to find: ‘Bringing hypotheses to bear on facts is an activity that goes on, ultimately, not in the savant’s head but in industry.’ ⁶ Conversely, critical theory has ‘human society itself for its object’. ⁷ It aims at answering questions that perhaps define the whole fields of a sociology of science and of knowledge: How do we come to hold beliefs about what is true? What are the standards for truth claims, and why do we stick to those? Here I am reminded of the observation that ‘questions of epistemology are also questions of the social order’. ⁸ A point that has been drawn out well, but that tends to be glossed over. ⁹ I will come back to that.

There is another troubling issue for Horkheimer when it comes to traditional theory next to its scientific positivism: its overwhelming instrumental rationality that only looks at the means, but not at the ends for which the means are employed. In other writings, Horkheimer notes how ‘reason’ and ‘reasonable’ have morphed into meaning nothing more than subjective usefulness – It is rational or reasonable if it suits me. ¹⁰ The capacity for ethical judgement, for a choice between different ends, got lost. At this passage I approach the most troubling and darkest prong of Horkheimer’s thought, which is shaped by his close collaboration with Theodor W. Adorno. Together they argue that the focus on instrumental reason is bound to collapse into irrationality. In their words, written in the early 1940s: ‘Enlightenment, understood in the widest sense as the advance of thought, has always aimed at liberating human beings from fear and installing them as masters. Yet the wholly enlightened earth is radiant with triumphant calamity.’ ¹¹ Their inquiry into why that is the case – why enlightenment turned into its opposite – leads to the relationship between the alienating domination of humans over nature and non-human animals, on the one hand, and their similar domination over other humans, on the other. A line of thought that is very tempting and most topical to pursue further, but not here. Horkheimer and Adorno notably sharpen the indictment of the sciences and charge their dominant mode of traditional theorizing not just with societal irrelevance, but with responsibility for some of the greatest calamities!

It has been a vexed question for Horkheimer and his critics whether and where there would still be potential for progress – progress in the form of emancipation from modes of domination. ¹² The best contenders in response have been the critique of society and the emancipation from false beliefs – critiques of ideology. The target of critique in that vein is the conservativeness of traditional theory in which people respect a ‘given reality, which they themselves constantly create’, a reality that then ‘becomes itself a positive fact’. ¹³ Horkheimer’s concern thus focused on false beliefs and the apparent fact that all ‘senseless wretchedness seems to be an unchangeable force of nature, a fate beyond man’s control’. ¹⁴

In the second half of the twentieth century, the demand for societal relevance carried forward this critical, progressive impetus. It aimed at reinstituting a debate about ends, about meaning and being in the world beyond the world’s appropriation for supposedly given, unquestioned, ends through instrumental reason. My faculty’s mission still breathes that lofty and laudable spirit – that is, at least, how I try to read it. On its own acclaim, my faculty aims at ‘progressive and trail-blazing education and research—for stronger legal institutions, solutions to social problems, and a better and just society’. But today, this acclamation of progressive intent no longer works and has even become regressive. What has changed?

3. The non-theory of change

If the demand for societal relevance once challenged and disturbed abstract formalism and self-referential sciences, today it does the opposite. It forgets to critique society and instead accepts society’s desiderata about what researchers should do. Only a few decades ago it was clear that the societal relevance of research must not be narrowed down to its practical usefulness for pursuing somehow given ends, especially not if that practical usefulness was to be assessed on the basis of ‘the system’ that was to be changed. But today, the demand for ‘societal relevance in terms of impact’ does precisely that when it asks for solutions. ¹⁵ The capacity of offering solutions, of solving problems, is what defines impactful research. As such, the once progressive demand for societal impact has been domesticated. It has become dull as a critique of power and potentially threatening only to researchers themselves: ‘Be operational … or disappear.’ ¹⁶ This is how the ‘societal relevance of impact’ is bound to be understood.

In spite of its efforts, my faculty has not yet adopted impact indicators. ¹⁷ Its claim to being progressive may be best served by its unruly character and slowness in catching up with the latest fashion. My argument against impact is supported by other experiences, historical and present. Here, look: Over at the Erasmus University in Rotterdam, they rolled out impact indicators in a colour-coded spread sheet that is organized around types of change to which output and outcome indicators are attached. It makes little sense and is full of mistakes, such as the constant conflation between knowledge, attitudes, and changes in behaviour. It is all based on a ‘Theory of Change’.

When I first heard about such a theory in a different context a couple of years ago, I was innocent in my excitement. Explaining change – what a magnificent task on which great minds across disciplines have trained themselves for centuries! Never had they achieved, to my knowledge, any such thing, a theory of change, tout court. Where would that theory locate the sources and dynamics of potential progress, I wondered. What kind of dialectics, which mix of mind and matter would the theory advocate? Or would it pick up a new materialism? Which subject and which dynamics would be sources of emancipation? Etc. How deflating to learn that the term (let’s not call it a theory) comes from the practice of management which, naturally, equates questions of change with, well, impact. It is about getting what you want, about return for money – for donors, philanthropists, funding agencies, and now for universities. The title of a recent book that comes up under the name says it all: ‘Theory of Change: A Practical Guide to Social Impact’. It is the stunning perfection of instrumental rationality.

An unshattered seventeenth-century rationalism defines the core of the impact indictors in Rotterdam: change knowledge to change attitude to change behaviour. Ach, if only! The indicators defy long-standing research insights concerning the dynamics of change with an ease of ignorance that only management economics can muster. Gone are decades of critical theory across disciplines that aimed at understanding and analysing society with the aspiration of emancipation through, well, critique, which could work in support of awareness about true causes and in favour of political mobilization. When and how such progress in terms of emancipation could happen has been the subject of extensive, invaluable debate. Whatever one may think of it, the rationalism of impact indicators is by comparison ludicrous in its simplicity, as though actors wanted to do the right thing, if only they knew how!

Just recall that close to 50 per cent of all CO₂ that was emitted since 1850 was emitted after 1990, in a timeframe during which knowledge about catastrophic effects was solid and widely available. Behaviour in spite of knowledge strikes me as the much more troubling problem than the lack of knowledge per se. This is where the poverty of traditional theory and the promise of critique is revealed once more: Unsustainable behaviour persists not least due to false beliefs about what is possible – beliefs that are so strong that they turn reality into ‘a positive fact’, ‘a fortress’, as Horkheimer had put it, ‘before which even the revolutionary imagination feels shamed as utopianism’. Much has been written since then about how ‘the system’ defines what is realistic, and what it would take to get out of those confines. ¹⁸ The demand for impact has to be understood in this context. Impact is impact according to the terms of ‘the system’. It has the regressive effect of favouring traditional over critical theory, of privileging prescription over analysis, solutions over problematizations. The ‘Theory of Change’ is neither a theory, nor is it about change. But sure enough, it is about impact.

4. Whose problems?

It goes without saying that lacking knowledge remains a problem, and new knowledge offers solutions. Science continues to deliver, in health crises (e.g., vaccines) or in the climate crisis (e.g., electrification). And academics should continue to engage in knowledge transfer, perhaps more so than they already do. Yes, please! That takes time, and the most charitable interpretation of valuing impact that I can think of is to recognize this time commitment. ¹⁹ But it would be naïve to stop there, not to recognize the politics and deeper problems that come with a shift towards impact – a shift that is not open-textured at all but closes in on solutions.

We are past the era of knowledge transfer. Today’s preferred term is that of the ‘knowledge cycle’. The idea is that knowledge held outside academia should also feed into academic research to improve it with practical insight. Once more: Who could disagree? But different things are getting mixed up. It is one thing to suggest that research should be relevant for society and quite another to suggest that society can be relevant for research. ²⁰ Those things are distinct even though they are of course linked: Research for which society is relevant is likely to be socially more relevant. This sense of relevance is arguably why ‘international lawyers revel in a good crisis’. ²¹ With the consequence that questions about the law are narrowed down, simplified, and sanitized at the expense of considering broader context, structural problems, and complex histories. ²²

Still deeper trouble arises when one zooms in on the degree of society’s relevance for research: How far should it go? Would it be to merely add insights for researchers to pursue their research questions, or would society’s relevance extend to co-determining the research questions to begin with? Who should decide what researchers do, researchers or society? Emphasis on synergies and mutual gain cannot hide what is at play here: What is being negotiated anew underneath the surface of the turn to ‘impact’ is of course the time-worn question about the autonomy of science vis-à-vis society, with a clear pivot towards reducing this autonomy in favour of societal control and with all the problems that this entails.

My argument has hammered the point that, just like science continues to be relevant by filling gaps in knowledge, it is relevant for the emancipatory critique of society – questioning how problems are understood and represented, unmasking ideology, thinking without the constraints of having to be operational. This freedom characterizes universities as opposed to most other institutions in society, what defines their distinct societal responsibility, and even their relevance. Remarkably so, everyone seems to agree: impact may be unexpected and may take a lot of time to see, a decade and more. ²³ The dilemmas that this should cause for research policy’s ambition to steer researchers in the present remains awkwardly unresolved. Impact cannot now be known for some time, but meanwhile we should do what has impact.

Here, then, is an additional concern about asking for impact without being able to tell what has impact. There are enough societal actors, mostly those with money and influence, who do know what will have impact – they know what would be useful to them. That can also be useful more generally (back to the examples of vaccines and electrification). But what about the research that, sticking with the examples, critiques the related distribution of access and profits? Useless? Even if such research were to offer solutions in the form of alternative arrangements – ‘shamed as utopianism’? Irrelevant?

No discipline is alone with these headaches. The arts and aesthetic judgment have arguably been the ultimate stronghold of autonomy vis-à-vis societal and practical usefulness. Demands for impact have not shied away from this field, but they have been disarmed by the kitsch they produce if taken seriously. For us international lawyers, what will it be: kitsch or critique? ²⁴