12 - Improving the Testing Process

Summary

Those who forget the lessons of history are doomed to repeat its mistakes.

George Santayana

Introduction

The idea that we should try to learn by our experiences is both natural and intuitive, but one we frequently seem to ignore in a software development and test context.

One particular approach to “learning by experience” in a project environment is to collect and use metrics observed during the course of the project, such as how much effort was expended in completing the project, how many defects were found during testing, or how many defects were found by the client following delivery of the tested system. This information can subsequently be used to determine whether the testing process is improving over time, by, for example, comparing the current effectiveness in finding defects against previously observed results.

Metrics can also be used to investigate the benefits of adopting some specific approach to testing, such as the introduction of automated testing, by comparing the efficiency of the testing process before introduction with that observed following rollout of the tool. The Crown Management Systems case study (Chapter 16) provides just such an example of the role and use of metrics in a testing project environment.

If no formal, quantitative measurements are made, it is possible only to make qualitative statements about the effectiveness of the testing process, which may in the short term assure senior management, but which in the long term will not help to improve the testing process.