2 - Compilation: setting the right foundations

Summary

• • Different types of evaluation

• • Choosing the evaluation focus – measurement vs understanding

• • Tackling objective-setting and evaluation rationales

• • Meeting needs and avoiding unrealistic expectations

• • Addressing the moral dimension – evaluation ethics

• • Resourcing, capability, independence and engagement in Evaluation

Introduction

Compilation is the first of the three ‘Cs’ of evaluations. It is about the time and effort put into setting the foundations for an evaluation that is fit for purpose, and doing this before starting to consider methods and approach. Digging foundations may seem obvious, but the pressures on getting an evaluation started can mean that some of the compilation issues looked at in this chapter are too easily neglected. Paying full and urgent attention to compilation pays dividends downstream and avoids the risks of later delays or damage to credibility from starting off on the wrong foot.

Types of evaluation

The starting point in compilation is deciding just what sort of evaluation is to be conducted. The labels used to describe different types of evaluation do not always make it easy to make this choice, and the terminology can vary between disciplinary traditions, and even within them. In essence, the options can be reduced to:

• • process evaluations

• • economic evaluations

• • impact evaluations.

Process evaluations: These focus on evaluating the mechanisms through which an intervention takes place, usually with a view to seeing how they could be improved. Process evaluations have attracted more variety in descriptions, sometimes being called developmental evaluations, or management evaluations. Whatever the label used, they share an emphasis on giving decision makers evidence of how (well) an intervention has been implemented or managed against expectations, how it operates, and how it produces what it does, rather than an understanding of what it has produced or its wider effects. With growing pressure on publicly funded or grant-funded services and initiatives, process evaluations are often a key evidence contribution to choices to be made on future priorities, streamlining or restructuring of activities.

Economic evaluations: A tightening public purse and pressure on margins and the ‘bottom line’ in many organisations mean that evaluations sometimes are focused wholly, or mainly, on measuring the costs of resource inputs or an intervention's value. Often this will need some comparative element to provide information about costeffectiveness, usually in financial terms.