6 - Conducting impact evaluation

Summary

• • Outcomes, intermediate impacts and impacts – getting to grips with the jargon

• • Understanding ‘gross’ and ‘net’ impacts, attribution and the Counterfactual

• • Use of randomised controlled trials, scope and limitations of fully experimental impact evaluation methods

• • Partial or quasi-experimental methods as a practical impact measurement alternative

• • Using non-experimental evaluation approaches to estimate Impacts

• • The use of qualitative approaches to assessing impact and Contributions

• • Avoiding pitfalls in impact evaluation

Introduction

Impact assessment has deep roots, but its increasing use in the social world, and relevance to public policy and decision-making, has seen an acceleration of new ideas and practice, especially in the last 20 years or so. The pace of these developments has not been easy to keep up with, and more than any other area of evaluation practice this has left a legacy of what may seem to be confusing and even contradictory approaches. Impact evaluation has consequently developed a special mystique, which can lead inexperienced evaluators to be cautious about what can, and cannot, be reliably measured or estimated. This chapter aims to uncloak some of the confusion that has arisen, explain the main methodological options and practical approaches now available, and set out their strengths and limitations. A good starting point is demystifying some of the jargon.

Unpicking the jargon

As Chapter 2 has shown, impact evaluation focuses on the consequential changes arising from an intervention. These are quite different from intervention outputs, which might be something delivered within the intervention, perhaps counselling or mentoring sessions, work trials, training, or developing new qualifications. These outputs are agents of change but not the change (impact) itself. A training course, for example, or even a qualification obtained at the end of it, does not of itself change someone's job situation, but securing a first or better job using that qualification – the impact – does. Other ‘consequential changes’ might be a school or college leaver making better-informed career or university choices, an unemployed participant securing a job, or a homeless person moving into secure accommodation.

These, and other impacts, are the real end product of whatever is being evaluated. They come in all shapes and sizes, depending on what is being evaluated, and present different challenges of measurement.