So far, we have used NN and kernel methods for nonlinear regression. However, when the output variables are discrete rather than continuous, the regression problem turns into a classification problem. For instance, instead of a prediction of the wind speed, the public may be more interested in a forecast of either ‘storm’ or ‘no storm’. There can be more than two classes for the outcome –for seasonal temperature, forecasts are often issued as one of three classes, ‘warm’, ‘normal’ and ‘cool’ conditions. Issuing just a forecast for one of the three conditions (e.g. ‘cool conditions for next season’) is often not as informative to the public as issuing posterior probability forecasts. For instance, a ‘25% chance of warm, 30% normal and 45% cool’ condition forecast, is quite different from a ‘5% warm, 10% normal and 85% cool’ condition forecast, even though both are forecasting cool conditions.We will examine methods which choose one out of k classes, and methods which issue posterior probability over k classes. In cases where a class is not precisely defined, clustering methods are used to group the data points into clusters.

In Section 1.6, we introduced the two main approaches to classification – the first by discriminant functions, the second by posterior probability. A discriminant function is simply a function which tells us which class we should assign to a given predictor data point x (also called a feature vector).

The human brain is an organ of marvel – with a massive network of about 1011 interconnecting neural cells called neurons, it performs highly parallel computing. The brain is exceedingly robust and fault tolerant. After all, we still recognize a friend whom we have not seen in a decade, though many of our brain cells have since died. A neuron is only a very simple processor, and its ‘clockspeed’ is actually quite slow, of the order of a millisecond, about a million times slower than that of a computer, yet the human brain beats the computer on many tasks involving vision, motor control, common sense, etc. Hence, the power of the brain lies not in its clockspeed, nor in the computing power of a neuron, but in its massive network structure. What computational capability is offered by a massive network of interconnected neurons is a question which has greatly intrigued scientists, leading to the development of the field of neural networks (NN).

Of course there are medical researchers who are interested in how real neural networks function. However, there are far more scientists from all disciplines who are interested in artificial neural networks, i.e. how to borrow ideas from neural network structures to develop better techniques in computing, artificial intelligence, data analysis, modelling and prediction. In fact, neural network methods have spread beyond medicine and science into engineering, economics and commerce.

In Chapter 4, we have learned that NN models are capable of approximating any nonlinear relation y = f(x) to arbitrary accuracy by using enough model parameters. However, data generally contain both signal and noise. In the process of fitting the highly flexible NN models to the data to find the underlying relation, one can easily fit to the noise in the data. Like Ulysses who had to balance the twin evils of Scylla and Charybdis, the NN modeller must also steer a careful course between using a model with too little flexibility to model the underlying nonlinear relation adequately (underfitting), and using a model with too much flexibility, which readily fits to the noise (overfitting). Finding the closest fit to the data – an objective adopted when using linear models – often leads to overfitting when using nonlinear NN models. It needs to be replaced by a wiser objective, that of learning the underlying relation accurately with the NN model. When the NN has found an overfitted solution, it will not fit new data well (Fig. 4.5), but if the NN has learned the underlying relationship well, it will be able to generalize from the original dataset, so that the extracted relationship even fits new data not used in training the NN model. This chapter surveys the various approaches which lead to proper learning and generalization. A comparison of different approaches to estimating the predictive uncertainty of a model is discussed in Section 6.9.

As one often encounters datasets with more than a few variables, multivariate statistical techniques are needed to extract the information contained in these datasets effectively. In the environmental sciences, examples of multivariate datasets are ubiquitous – the air temperatures recorded by all the weather stations around the globe, the satellite infrared images composed of numerous small pixels, the gridded output from a general circulation model, etc. The number of variables or time series from these datasets ranges from thousands to millions.Without a mastery of multivariate techniques, one is overwhelmed by these gigantic datasets. In this chapter, we review the principal component analysis method and its many variants, and the canonical correlation analysis method. These methods, using standard matrix techniques such as singular value decomposition, are relatively easy to use, but suffer from being linear, a limitation which will be lifted with neural network and kernel methods in later chapters.

Principal component analysis (PCA)

Geometric approach to PCA

We have a dataset with variables y1, …, ym. These variables have been sampled n times. In many situations, the m variables are m time series each containing n observations in time. For instance, one may have a dataset containing the monthly air temperature measured at m stations over n months.

Broadly speaking, the goal of (mainstream) learning theory is to approximate a function (or some function features) from data samples, perhaps perturbed by noise. To attain this goal, learning theory draws on a variety of diverse subjects. It relies on statistics whose purpose is precisely to infer information from random samples. It also relies on approximation theory, since our estimate of the function must belong to a prespecified class, and therefore the ability of this class to approximate the function accurately is of the essence. And algorithmic considerations are critical because our estimate of the function is the outcome of algorithmic procedures, and the efficiency of these procedures is crucial in practice. Ideas from all these areas have blended together to form a subject whose many successful applications have triggered its rapid growth during the past two decades.

This book aims to give a general overview of the theoretical foundations of learning theory. It is not the first to do so.Yet we wish to emphasize a viewpoint that has drawn little attention in other expositions, namely, that of approximation theory. This emphasis fulfills two purposes. First, we believe it provides a balanced view of the subject. Second, we expect to attract mathematicians working on related fields who find the problems raised in learning theory close to their interests.

While writing this book, we faced a dilemma common to the writing of any book in mathematics: to strike a balance between clarity and conciseness. In particular, we faced the problem of finding a suitable degree of self-containment for a book relying on a variety of subjects.