3 - PCA and kernel PCA

Summary

Introduction

Two primary techniques for dimension-reducing feature extraction are subspace projection and feature selection. This chapter will explore the key subspace projection approaches, i.e. PCA and KPCA.

(i) Section 3.2 provides motivations for dimension reduction by pointing out (1) the potential adverse effect of large feature dimensions and (2) the potential advantage of focusing on a good set of highly selective representations.

(ii) Section 3.3 introduces subspace projection approaches to feature-dimension reduction. It shows that the well-known PCA offers the optimal solution under two information-preserving criteria: least-squares error and maximum entropy.

(iii) Section 3.4 discusses several numerical methods commonly adopted for computation of PCA, including singular value decomposition (on the data matrix), spectral decomposition (on the scatter matrix), and spectral decomposition (on the kernel matrix).

(iv) Section 3.5 shows that spectral factorization of the kernel matrix leads to both kernel-based spectral space and kernel PCA (KPCA) [238]. In fact, KPCA is synonymous with the kernel-induced spectral feature vector. We shall show that nonlinear KPCA offers an enhanced capability in handling complex data analysis. By use of examples, it will be demonstrated that nonlinear kernels offer greater visualization flexibility in unsupervised learning and higher discriminating power in supervised learning.

Why dimension reduction?

In many real-world applications, the feature dimension (i.e. the number of features or attributes in an input vector) could easily be as high as tens of thousands. Such an extreme dimensionality could be very detrimental to data analysis and processing.