In this book, we study a system which perceives the real world. Such a system has to estimate an information source by observation. If the information source is a probability distribution, then the estimation process is called statistical learning, and the system is said to be a statistical model or a learning machine.

A lot of statistical models have hierarchical layers, hidden variables, a collection of modules, or grammatical structures. Such models are nonidentifiable and contain singularities in their parameter spaces. In fact, the map from a parameter to a statistical model is not one-to-one, and the Fisher information matrix is not positive definite. Such statistical models are called singular. It has been difficult to examine the learning process of singular models, because there has been no mathematical theory for such models.

In this book, we establish a mathematical foundation which enables us to understand the learning process of singular models. This chapter gives an overview of the book before a rigorous mathematical foundation is developed.

Basic concepts in statistical learning

To describe what statistical learning is, we need some basic concepts in probability theory. For the reader who is unfamiliar with probability theory, Section 1.6 summarizes the key results.

Random samples

Let N be a natural number and ℝN be the N-dimensional real Euclidean space. We study a case when information data are represented by vectors in ℝN.

In this book, we introduce a fundamental relation between algebraic geometry and statistical learning theory.

A lot of statistical models and learning machines used in information science, for example, mixtures of probability distributions, neural networks, hidden Markov models, Bayesian networks, stochastic context-free grammars, and topological data analysis, are not regular but singular, because they are nonidentifiable and their Fisher information matrices are singular. In such models, knowledge to be discovered from examples corresponds to a singularity, hence it has been difficult to develop a mathematical method that enables us to understand statistical estimation and learning processes.

Recently, we established singular learning theory, in which four general formulas are proved for singular statistical models. Firstly, the log likelihood ratio function of any singular model can be represented by the common standard form even if it contains singularities. Secondly, the asymptotic behavior of the evidence or stochastic complexity is clarified, giving the result that the learning coefficient is equal to the maximum pole of the zeta function of a statistical model. Thirdly, there exist equations of states that express the universal relation of the Bayes quartet. We can predict Bayes and Gibbs generalization errors using Bayes and Gibbs training errors without any knowledge of the true distribution. And lastly, the symmetry of the generalization and training errors holds in the maximum likelihood and a posteriori estimators. If one-point estimation is applied to statistical learning, the generalization error is equal to the maximum value of a Gaussian process on a real analytic set.

In this chapter, we study statistical model evaluation and statistical hypothesis tests in singular learning machines. Firstly, we show that there is no universally optimal learning in general and that model evaluation and hypothesis tests are necessary in statistics. Secondly, we analyze two information criteria: stochastic complexity and generalization error in singular learning machines. Thirdly, we show a method to produce a statistical hypothesis test if the null hypothesis is a singularity of the alternative hypothesis. Then the methods by which the Bayes a posteriori distribution is generated are introduced. We discuss the Markov chain Monte Carlo and variational approximation. In the last part of this chapter, we compare regular and singular learning theories. Regular learning theory is based on the quadratic approximation of the log likelihood ratio function and the central limit theorem on the parameter space, whereas singular learning theory is based on the resolution of singularities and the central limit theorem on the functional space. Mathematically speaking, this book generalizes regular learning theory to singular statistical models.

Universally optimal learning

There are a lot of statistical estimation methods. One might expect that there is a universally optimal method, which always gives a smaller generalization error than any other method. However, in general, such a method does not exist.

Assumption. Assume that Φ(w) is the probability density function on ℝd, and that a parameter ω is chosen with respect to Φ(ω).

Machine learning is a major sub-field in computational intelligence (also called artificial intelligence). Its main objective is to use computational methods to extract information from data. Machine learning has a wide spectrum of applications including handwriting and speech recognition, object recognition in computer vision, robotics and computer games, natural language processing, brain–machine interfaces, medical diagnosis, DNA classification, search engines, spam and fraud detection, and stock market analysis. Neural network methods, generally regarded as forming the first wave of breakthrough in machine learning, became popular in the late 1980s, while kernel methods arrived in a second wave in the second half of the 1990s.

In the 1990s, machine learning methods began to infiltrate the environmental sciences. Today, they are no longer an exotic fringe species, since their presence is ubiquitous in the environmental sciences, as illustrated by the lengthy References section of this book. They are heavily used in satellite data processing, in general circulation models (GCM) for emulating physics, in post-processing of GCM model outputs, in weather and climate prediction, air quality forecasting, analysis and modelling of environmental data, oceanographic and hydrological forecasting, ecological modelling, and in monitoring of snow, ice and forests, etc.

This book presents machine learning methods (mainly neural network and kernel methods) and their applications in the environmental sciences, written at a level suitable for beginning graduate students and advanced undergraduates.

In this final chapter, we survey the applications of machine learning methods in the various areas of environmental sciences – e.g. remote sensing, oceanography, atmospheric science, hydrology and ecology. More examples of applications are given in Haupt et al. (2009).

In the early applications of methods like NN to environmental problems, researchers often did not fully understand the problem of overfitting, and the need to validate using as much independent data as possible, by e.g. using crossvalidation. For historical reasons, some of these early papers are still cited here, although the control of overfitting and/or the validation process may seem primitive by latter day standards.

In some of the papers, the authors used more than one type of machine learning method and compared the performance of the various methods. A word of caution is needed here. In our present peer-reviewed publication system, a new method can be published only if it is shown to be better in some way than established methods. Authors therefore tend to present their new methods in the best possible light to enhance their chance of passing the peer review. For instance, an author might test his new method A against the traditional method B on two different datasets and find method A and B each outperformed the other on one dataset.

In Chapter 9, we have seen machine learning methods nonlinearly generalizing the linear regression method. In this chapter, we will examine ways to nonlinearly generalize principal component analysis (PCA) and related methods. Figure 10.1 illustrates the difference between linear regression, PCA, nonlinear regression, and nonlinear PCA.

Principal component analysis can be performed using neural network (NN) methods (Oja, 1982; Sanger, 1989). However, far more interesting is the nonlinear generalization of PCA, where the straight line in PCA is replaced by a curve which minimizes the mean squared error (MSE) (Fig. 10.1). Nonlinear PCA can be performed by a variety of methods, e.g. the auto-associative NN model using multi-layer perceptrons (MLP) (Kramer, 1991; Hsieh, 2001b), and the kernel PCA model (Schölkopf et al., 1998). Nonlinear PCA belongs to the class of nonlinear dimensionality reduction techniques, which also includes principal curves (Hastie and Stuetzle, 1989), locally linear embedding (LLE) (Roweis and Saul, 2000) and isomap (Tenenbaum et al., 2000). Self-organizing map (SOM) (Kohonen, 1982) can also be regarded as a discrete version of NLPCA. Dong and McAvoy (1996) combined the principal curve and MLP approaches, while Newbigging et al. (2003) used the principal curve projection concept to improve on the MLP approach. Another way to generalize PCA is via independent component analysis (ICA) (Comon, 1994; Hyvärinen et al., 2001), which was developed from information theory, and has been applied to study the tropical Pacific sea surface temperature (SST) variability by Aires et al. (2000).