In this chapter, we consider inference and learning problems that can be formulated as the optimization of an information-theoretic measure (ITM).

As introduced in Chapter 4, setting up a learning problem requires the selection of an inductive bias, which consists of a model class $\mathscr{H}$ and a training algorithm. By the no-free-lunch theorem, this first step is essential in order to make generalization possible. A trained model generalizes if it performs well outside the training set, on average with respect to the unknown population distribution.

As discussed in Chapter 2, learning is needed when a “physics”-based mathematical model for the data generation mechanism is not available or is too complex to use for design purposes. As an essential benchmark setting, this chapter discusses the ideal case in which an accurate mathematical model is known, and hence learning is not necessary. As in large part of machine learning, we specifically focus on the problem of prediction. The goal is to predict a target variable given the observation of an input variable based on a mathematical model that describes the joint generation of both variables. Model-based prediction is also known as inference.

Previous chapters have formulated learning problems within a frequentist framework. Frequentist learning aims to determine a value of the model parameter $\theta$ that approximately minimizes the population loss. Since the population loss is not known, this is in practice done by minimizing an estimate of the population loss $L_p(\theta )$ based on training data – the training loss $L_{𝒟}(\theta )$.

As seen in the preceding chapter, when a reliable model $p(x,t)$ is available to describe the probabilistic relationship between input variable x and target variable t, one is faced with a model-based prediction problem, also known as inference. Inference can in principle be optimally addressed by evaluating functions of the posterior distribution $p(t|x)=p(x,t)/p(x)$ of the output t given the input x.

This chapter provides a refresher on probability and linear algebra with the aim of reviewing the necessary background for the rest of the book. Readers not familiar with probability and linear algebra are invited to first consult one of the standard textbooks mentioned in Recommended Resources, Sec. 2.14. Readers well versed on these topics may briefly skim through this chapter to get a sense of the notation used in the book.

In the examples studied in Chapter 4, the exact optimization of the (regularized) training loss was feasible through simple numerical procedures or via closed-form analytical solutions. In practice, exact optimization is often computationally intractable, and scalable implementations must rely on approximate optimization methods that perform local, iterative updates in search of an optimized solution. This chapter provides an introduction to local optimization methods for machine learning.

The previous chapter, as well as Chapter 4, have focused on supervised learning problems, which assume the availability of a labeled training set $𝒟$. A labeled data set consists of examples in the form of pairs (𝑥, 𝑡) of input 𝑥 and desired output 𝑡.

This chapter aims to motivate the study of machine learning, having in mind as the intended audience students and researchers with an engineering background.