17 - Evaluation of RDS

Summary

Aims

The purpose of this chapter is to consider how to evaluate RDS and RDM.

Introduction

Evaluation is a controversial topic. On the one hand, it feels obvious that without clearly defined objectives it is impossible to say whether a set of activities are worthwhile. And if we do not collect some data about how we are performing against those objectives we do not know how well we are performing. Tools like SMART objectives reflect this thinking and encourage us to measure whether objectives have been met. At the same time, it may be difficult to define purposes, especially where complex, intangible values are being pursued. In such a context defining our purposes and defining valid measures, i.e. ones that actually measure what we want to achieve, are hard. Capturing data about our achievements, especially quantitative data, is likely to be problematic. At some point more complex measures break down because they are hard to collect and understand.

In the library world, for example, there is pressure to demonstrate the value of the library to student learning. But learning is such a complex construct that it is hard to see how it could ever easily be measured, in a totally valid way. Tools such as the LibQual survey, while widely used in libraries to measure service performance, only compare satisfaction against expectation, not learning itself. Other types of measure, such as the number of resources or even downloads, do not directly link to learning, only the levels of activity.

In addition, the question of evaluation has resonances with the debate around the new public management and neo-liberalisation (see Chapter 5). For many, the way that academia is increasingly run like a private organisation with crude quantitative measures of performance – ‘key performance indicators’ – erodes the university's true purpose to promote learning and research in their widest sense.

Nevertheless, it would be odd not to write about the evaluation of RDS/RDM here, even though the literature on the topic is actually surprisingly sparse. From a management perspective it makes sense to collect data about performance, even if it is purely for internal consumption within the RDS team.