6 - Learning and generalization

Summary

In Chapter 4, we have learned that NN models are capable of approximating any nonlinear relation y = f(x) to arbitrary accuracy by using enough model parameters. However, data generally contain both signal and noise. In the process of fitting the highly flexible NN models to the data to find the underlying relation, one can easily fit to the noise in the data. Like Ulysses who had to balance the twin evils of Scylla and Charybdis, the NN modeller must also steer a careful course between using a model with too little flexibility to model the underlying nonlinear relation adequately (underfitting), and using a model with too much flexibility, which readily fits to the noise (overfitting). Finding the closest fit to the data – an objective adopted when using linear models – often leads to overfitting when using nonlinear NN models. It needs to be replaced by a wiser objective, that of learning the underlying relation accurately with the NN model. When the NN has found an overfitted solution, it will not fit new data well (Fig. 4.5), but if the NN has learned the underlying relationship well, it will be able to generalize from the original dataset, so that the extracted relationship even fits new data not used in training the NN model. This chapter surveys the various approaches which lead to proper learning and generalization. A comparison of different approaches to estimating the predictive uncertainty of a model is discussed in Section 6.9.