Introduction

Face recognition is one of the most widely researched topics in computer vision because of a wide variety of applications that require identity management. Most existing face recognition studies are focused on two-dimensional (2D) images with nearly frontal-view faces and constrained illumination. However, 2D facial images are strongly affected by varying illumination conditions and changes in pose. Thus, although existing methods are able to provide satisfactory performance under constrained conditions, they are challenged by unconstrained pose and illumination conditions.

FRVT 2006 explored the feasibility of using three-dimensional (3D) data for both enrollment and authentication (Phillips et al. 2007). The algorithms using 3D data have demonstrated their ability to provide good recognition rates. For practical purposes, however, it is unlikely that large scale deployments of 3D systems will take place in the near future because of the high cost of the hardware. Nevertheless, it is not unreasonable to assume that an institution may want to invest in a limited number of 3D scanners, if having 3D data for enrollment can yield higher accuracy for 2D face authentication/identification.

In this respect we have developed a face recognition method that makes use of 3D face data for enrollment while requiring only 2D data for authentication. During enrollment, different from the existing methods (e.g., Blanz and Vetter 2003) that use a 2D image to infer a 3D model in the gallery, we use 2D+3D data (2D texture plus 3D shape) to build subject-specific annotated 3D models.

Introduction

Multibiometrics

With the ever increasing demand for security and identification systems, the adoption of biometric systems is becoming widespread. There are many reasons for developing multibiometric systems; for example, a subject may conceal or lack the biometric a system is based on. This can be a significant problem with noncontact biometrics in some applications (e.g., surveillance). Many noncontact biometric modalities exist. Of these face recognition has been the most widely studied, resulting in both its benefits and drawbacks being well understood. Others include gait, ear, and soft biometrics. Automatic gait recognition is attractive because it enables the identification of a subject from a distance, meaning that it will find applications in a variety of different environments (Nixon et al. 2005). The advantage of the ear biometric is that the problems associated with age appear to be slight, though enrolment can be impeded by hair (Hurley et al. 2008). There are also new approaches to using semantic descriptions to enhance biometric capability, sometimes known as soft biometrics (Samangooei et al. 2008). The semantic data can be used alone, or in tandem with other biometrics, and are suited particularly to analysis of surveillance data.

The deployment of mutilbiometric systems is largely still at a research phase (Ross et al. 2006). Of the biometrics discussed here, some approaches fuse face with gait (Shakhnarovich and Darrell 2002; Liu and Sarkar 2007; Zhou and Bhanu 2007; Yamauchi et al. 2009) and some that fuse ear with face (Chang et al. 2003). No approach has fused gait and ear data.

Introduction

Personal identification and verification by using biometric traits, such as finger-prints and faces, cover a large variety of applications. However, performance of current systems is still far from humans' (Sinha et al. 2006a, b).

The core of a biometric recognition system is the so-called enrollment stage. For each client, one or more biometric traits (e.g., a fingerprint and face images) are acquired and processed to represent it with a feature set (e.g., minutiae points). This feature set, labeled with the user's identity, is called a template and stored as a prototype of user's biometric trait in the system's database. The template is used in the recognition stage by comparing it with the input biometric(s), thus obtaining the so-called matching score, a real value into [0,1], which is the degree of similarity between the input sample and the given template.

As pointed out clearly in Uludag et al. (2004), in real operational scenarios, we have to handle substantial variations of each person's appearance. This large intraclass variability is due to changes of the environment conditions (e.g., illumination changes), aging of the biometric traits, variations of the interaction between the sensor and the individual (e.g., variations of the person pose), etc. Therefore, the enrolled templates could be poorly representative of the biometric data to be recognized, resulting in poor recognition performance.

The evaluation of learning algorithms both in absolute terms and in relation to other algorithms involves addressing four main components:

• performance measures,

• error estimation,

• statistical significance testing, and

• test benchmark selection.

The first component concerns the property of the algorithm's performance that one wishes to measure. The answers are sought for questions such as these: Do we measure how accurate the algorithm is? If so, how do we define accuracy? Do we value one aspect of the algorithm's performance more than other? These and related issues are the focus of this and the next chapter. Once a performance measure is chosen, the next big concern is to estimate it in as unbiased a manner as possible, making the best possible use of the available data. This is the focus of Chapter 5, on performance estimation. Chapter 6 then focuses on investigating whether the differences in the performances obtained by the algorithm alone or in relation to others are statistically significant. Finally, we try to complete the puzzle with a discussion on what domains can be deemed suitable as benchmarks to evaluate learning approaches. This is the focus of Chapter 7.

Performance measures have arguably received the greatest amount of attention in the field. As a consequence of the inherent multidisciplinary nature of the machine learning tasks, different variants of these performance measures have been influenced by approaches from a variety of disciplines, including statistics, medicine, and information retrieval.

The advances in performance measure characterization discussed in Chapters 3 and 4 have armed researchers with more precise estimates of classifier performance. However, these are not by themselves sufficient to fully evaluate the difference in performances between classifiers on one or more test domains. More precisely, even though the performance of different classifiers may be shown to be different on specified sets of data, it needs to be confirmed whether the observed differences are statistically significant and not merely coincidental. Chapter 5 started to look at this issue, but focused primarily on the objectivity and stability of the results. This can be construed as the first step to assessing the significance of a difference. Only in the case of the comparison of two classifiers on a single domain did the discussion actually move on to significance issues. Statistical significance testing, which is the subject of this chapter, enables researchers to move on to more precise assessments of significance of the results obtained (within certain constraints). The importance of statistical significance testing hence cannot be overstated. Nonetheless, the use of available statistical tools for such testing in the fields of machine learning and data mining has been limited at best. Researchers have concentrated on using the paired t test, many times inappropriately, to confirm the difference in classifiers' performance. Moreover, this has sometimes been done at the cost of excluding other, more appropriate, tests.

We conclude the discussion on various aspects of performance evaluation of learning algorithms by unifying these seemingly disparate parts and putting them in perspective. The raison d'être of the following discussion is to appreciate the breadth and depth of the overall evaluation process, emphasizing the fact that such evaluation experiments should not be put together in an ad hoc manner, as they are currently done in many cases, by merely selecting a random subset of some or all of the components discussed in various chapters so far. Indeed, a careful consideration is required of both the underlying evaluation requirements and, in this context, of the correlation between the different choices for each component of the evaluation framework.

This chapter attempts to give a brief snapshot of the various components of the evaluation framework and highlights some of their major dependencies. Moreover, for each component we also give a template of the various steps necessary to make appropriate choices along with some of the main concerns and interrelations to take into account with respect to both, other steps in a given component and other evaluation components themselves. Unfortunately, because of the intricate dependencies between various steps as well as components, it might seem necessary to make simultaneous choices and check their compatibility.

The general model evaluation framework should serve as a representative template and not as a definitive guide.

We reviewed the major components of the evaluation framework in the last chapters and described in detail the various techniques pertaining to each, together with their applications. The focus of this chapter is to complement this review by outlining various advancements that have been made relatively recently, but that have not yet become mainstream. We also look into some approaches aimed at addressing problems arising from the developments on the machine learning front in various application settings, such as ensemble classifiers.

Just as with the traditional developments in performance evaluation, recent attempts at improving as well as designing new performance metrics have led the way. These have resulted in both improvements to existing performance metrics, thereby claiming to ameliorate the issues with the current versions, and proposals for novel metrics aimed at addressing the areas of algorithm evaluation not satisfactorily addressed by current metrics. We discuss in brief some of these advancements in Section 8.1. In Section 8.2, we focus on the attempts at unifying these performance metrics as well as studying their interrelation in the form of both theoretical and experimental frameworks. A natural extension to such studies is the design of more general or broader measures of performance, either as a result of insights obtained from the theoretical framework or by combining existing metrics based on observations from the experimental frameworks. Such metric combinations for obtaining general measures are the focus of Section 8.3.

We have discussed different aspects and components pertaining to the evaluation of learning algorithms. Given one or more fixed domains, we have discussed various performance measures, a number of sampling and resampling methods designed to estimate the outcome of these performance measures in a reliable manner and tests allowing us to estimate the statistical significance of the observed results. Many other aspects linked to each of these steps were also surveyed along the line, such as the notion of bias and variance and the debate on the need to practice statistical significance testing. The only aspect of evaluation that was not questioned, so far, is the issue of determining an appropriate testbed for our experiments. All the components that we discussed so far in this book made the implicit assumption expressed in the second sentence of this chapter: “Given one or more fixed domains.” It is now time to expand on this issue because the application, results, and subsequent interpretation of the different components of the evaluation process depend critically on the domains on which these are assessed and quantified. Furthermore, selecting datasets for evaluating the algorithms is certainly a nontrivial issue.

One important result connected to the choice of datasets on which to evaluate learning algorithms is summarized in Wolpert's “No Free Lunch” theorems. These theorems show the importance of evaluating algorithms on a large number of problems, because, if only a small sample of problems is considered, the results could be biased.

This chapter is aimed at establishing the conceptual foundation of the relevant aspects of machine learning and statistics on which the book rests. This very brief overview is in no way exhaustive. Rather, our main aim is to elucidate the relationship of these concepts to the performance evaluation of learning algorithms. The chapter is composed of two parts. The first part discusses concepts most specific to machine learning; the second part focuses on the statistical elements. Even though these may seem like two disparate parts, they are not entirely independent. We try to highlight the relationship between the concepts discussed in one field to the problems at hand in the other. Let us start with a brief discussion of the important concepts of machine learning.

Machine Learning Overview

Learning is the human process that allows us to acquire the skills necessary to adapt to the multitude of situations that we encounter throughout our lives. As human beings, we rely on many different kinds of learning processes at different stages to acquire different functionalities. We learn a variety of different skills, e.g., motor, verbal, mathematical, and so on. Moreover, the learning process differs with the situations and time, e.g., learning how to speak as a toddler is different from learning similar skill sets in a given profession. Variations in learning are also visible in terms of contexts and the related tools, e.g., classroom learning differs from social contexts, rote learning may be more suitable for memorizing but differs from learning how to reason.

This book was started at Monash University (Melbourne, Australia) and Laval University (Quebec City, Canada) with the subsequent writing taking place at the University of Ottawa (Ottawa, Canada) and McGill University (Montreal, Canada). The main idea stemmed from the observation that while machine learning as a field is maturing, the importance of evaluation has not received due appreciation from the developers of learning systems. Although almost all studies make a case for the evaluation of the algorithms they present, we find that many (in fact a majority) demonstrate a limited understanding of the issues involved in proper evaluation, despite the best intention of their authors. We concede that optimal choices cannot always be made due to limiting circumstances, and trade-offs are inevitable. However, the methods adopted in many cases do not reflect attention to the details warranted by a proper evaluation approach (of course there are exceptions and we do not mean to generalize this observation).

Our aim here is not to present the readers with yet another recipe for evaluation that can replace the current default approach. Rather, we try to develop an understanding of and appreciation for the different concerns of importance in the practical application and deployment of learning systems. Once these concerns are well understood, the other pieces of the puzzle fall quickly in place since the researcher is not left shooting in the dark.

Technological advances in recent decades have made it possible to automate many tasks that previously required significant amounts of manual time, performing regular or repetitive activities. Certainly, computing machines have proven to be a great asset in improving human speed and efficiency as well as in reducing errors in these essentially mechanical tasks. More impressive, however, is the fact that the emergence of computing technologies has also enabled the automation of tasks that require significant understanding of intrinsically human domains that can in no way be qualified as merely mechanical. Although we humans have maintained an edge in performing some of these tasks, e.g., recognizing pictures or delineating boundaries in a given picture, we have been less successful at others, e.g., fraud or computer network attack detection, owing to the sheer volume of data involved and to the presence of nonlinear patterns to be discerned and analyzed simultaneously within these data. Machine learning and datamining, on the other hand, have heralded significant advances, both theoretical and applied, in this direction, thus getting us one step closer to realizing such goals.

Machine learning is embodied by different learning approaches, which are themselves implemented within various frameworks. Examples of some of the most prominent of these learning paradigms include supervised learning, in which the data labels are available and generally discrete; unsupervised learning, in which the data labels are unavailable; semisupervised learning, in which some, generally discrete, data labels are available, but not all; regression, in which the data labels are continuous; and reinforcement learning, in which learning is based on an agent policy optimization in a reward setting.

Tables B.1 and B.2 show the results obtained using 10-fold cross validation by c45 and NB on each instance of the labor data respectively as output by WEKA. The first column lists the instance number; the second column lists the instance label, where class 1 corresponds to class “bad” and class 2 corresponds to class “good”; the third column lists the predicted class, using the same naming convention; column 4 uses the “+” symbol to indicate whether the predicted label differs from the actual one and a blank if they are in agreement; finally, the last two values, which are complementary and add up to 1, indicate the confidence of their prediction. The first value indicates how much the classifier believes the instance to be of class 1 (bad), and the second indicates how much the classifier believes the instance to be of classs 2 (good). The dominant value is preceded by a “*” symbol and corresponds to the value of the predicted label.

Please note that the numbers denoting the instances in the first column are not sequential. After number 6 or 7 is reached, a 1–6 or 1–7 sequence is repeated. This is because every 1–6 or 1–7 sequence represents a different fold. Indeed, it can be seen that 10 different sequences are present in each classifier run, corresponding to the 10 folds of 10-fold cross-validation.