7 - Datasets and Experimental Framework

Summary

We have discussed different aspects and components pertaining to the evaluation of learning algorithms. Given one or more fixed domains, we have discussed various performance measures, a number of sampling and resampling methods designed to estimate the outcome of these performance measures in a reliable manner and tests allowing us to estimate the statistical significance of the observed results. Many other aspects linked to each of these steps were also surveyed along the line, such as the notion of bias and variance and the debate on the need to practice statistical significance testing. The only aspect of evaluation that was not questioned, so far, is the issue of determining an appropriate testbed for our experiments. All the components that we discussed so far in this book made the implicit assumption expressed in the second sentence of this chapter: “Given one or more fixed domains.” It is now time to expand on this issue because the application, results, and subsequent interpretation of the different components of the evaluation process depend critically on the domains on which these are assessed and quantified. Furthermore, selecting datasets for evaluating the algorithms is certainly a nontrivial issue.

One important result connected to the choice of datasets on which to evaluate learning algorithms is summarized in Wolpert's “No Free Lunch” theorems. These theorems show the importance of evaluating algorithms on a large number of problems, because, if only a small sample of problems is considered, the results could be biased.