10 - Nonlinear principal component analysis

Summary

In Chapter 9, we have seen machine learning methods nonlinearly generalizing the linear regression method. In this chapter, we will examine ways to nonlinearly generalize principal component analysis (PCA) and related methods. Figure 10.1 illustrates the difference between linear regression, PCA, nonlinear regression, and nonlinear PCA.

Principal component analysis can be performed using neural network (NN) methods (Oja, 1982; Sanger, 1989). However, far more interesting is the nonlinear generalization of PCA, where the straight line in PCA is replaced by a curve which minimizes the mean squared error (MSE) (Fig. 10.1). Nonlinear PCA can be performed by a variety of methods, e.g. the auto-associative NN model using multi-layer perceptrons (MLP) (Kramer, 1991; Hsieh, 2001b), and the kernel PCA model (Schölkopf et al., 1998). Nonlinear PCA belongs to the class of nonlinear dimensionality reduction techniques, which also includes principal curves (Hastie and Stuetzle, 1989), locally linear embedding (LLE) (Roweis and Saul, 2000) and isomap (Tenenbaum et al., 2000). Self-organizing map (SOM) (Kohonen, 1982) can also be regarded as a discrete version of NLPCA. Dong and McAvoy (1996) combined the principal curve and MLP approaches, while Newbigging et al. (2003) used the principal curve projection concept to improve on the MLP approach. Another way to generalize PCA is via independent component analysis (ICA) (Comon, 1994; Hyvärinen et al., 2001), which was developed from information theory, and has been applied to study the tropical Pacific sea surface temperature (SST) variability by Aires et al. (2000).