9 - Conclusion

Summary

We conclude the discussion on various aspects of performance evaluation of learning algorithms by unifying these seemingly disparate parts and putting them in perspective. The raison d'être of the following discussion is to appreciate the breadth and depth of the overall evaluation process, emphasizing the fact that such evaluation experiments should not be put together in an ad hoc manner, as they are currently done in many cases, by merely selecting a random subset of some or all of the components discussed in various chapters so far. Indeed, a careful consideration is required of both the underlying evaluation requirements and, in this context, of the correlation between the different choices for each component of the evaluation framework.

This chapter attempts to give a brief snapshot of the various components of the evaluation framework and highlights some of their major dependencies. Moreover, for each component we also give a template of the various steps necessary to make appropriate choices along with some of the main concerns and interrelations to take into account with respect to both, other steps in a given component and other evaluation components themselves. Unfortunately, because of the intricate dependencies between various steps as well as components, it might seem necessary to make simultaneous choices and check their compatibility.

The general model evaluation framework should serve as a representative template and not as a definitive guide.