We described several data-based methods for inference and learning in the previous chapters. These methods operate directly on the data to arrive at classification or inference decisions. One key challenge these methods face is that the available training data need not provide sufficient representation for the sample space.

We indicated in the concluding remarks of the previous chapter that feedforward neural networks have powerful modeling capabilities, as reflected by the universal approximation theorem. In one of its versions, the theorem asserts that networks with a single hidden layer are rich enough to model almost any arbitrary function.

We encountered one instance of Bayesian inference in Chapter 50, based on the quadratic loss in the context of mean-square-error (MSE) estimation. We explained there that the optimal solution for inferring a hidden zero-mean random variable $x$ from observations of another zero-mean random variable $y$ is given by the conditional estimator, $\mathbb{E}\left(x|y\right)$, whose computation requires knowledge of the conditional distribution, $f_{x|y}\left(x|y\right)$.

In supervised methods, learning is attained by training on a sufficient amount of labeled data in order to deliver reliable levels of classification. However, there are important situations in practice where data is scarce because it is either difficult or expensive to collect. This scenario leads to few-shot learning, where it is desired to train a classifier by using only a few training samples for each class.

We illustrated in Example 63.2 one limitation of linear separation surfaces by considering the XOR mapping (63.11). The example showed that certain feature spaces are not linearly separable and cannot be resolved by the perceptron algorithm. The result in the example was used to motivate one powerful approach to nonlinear separation surfaces by means of kernel methods.

In the immediate past chapters we developed several techniques for the design of linear classifiers, such as logistic regression, perceptron, and support vector machines (SVM). These algorithms are suitable for data that are linearly separable; otherwise, their performance degrades significantly. In this chapter we explain how the methods can be adjusted to determine nonlinear separation surfaces.

In most multistage decision problems, we are interested in determining the optimal strategy, ${\pi }^{\star }\left(a\right|s)$ (i.e., the optimal actions to follow in the state–action space). Most of the algorithms described in the previous chapters focused on evaluating the state and state–action value functions, ${\upsilon }^{\pi }\left(s\right)$ and $q^{\pi }(s,a)$, for a given policy $\pi \left(a\right|s)$. More is needed to learn the optimal policy.

We derived in the previous two chapters procedures for assessing the performance of strategies used by agents interacting with a Markov decision process (MDP), including obtaining optimal policies. Among other methods, we discussed the policy evaluation algorithm (44.116) and the value and policy iterations (45.23) and (45.43), respectively.

Principal component analysis (PCA) is a formidable tool for dimensionality reduction. Given feature vectors $\left\{h_n\right\}$ in $M$‐dimensional space, PCA replaces them by lower‐dimensional vectors $\left\{h_n^{\prime }\right\}$ of size $M^{\prime }\ll M$ each.

Markov decision processes (MDPs) are at the core of reinforcement learning theory. Similar to Markov chains, MDPs involve an underlying Markovian process that evolves from one state to another, with the probability of visiting a new state being dependent on the most recent state. Different from Markov chains, MDPs involve both agents and actions taken by these agents. As a result, the next state is dependent on which action was chosen at the state preceding it. MDPs therefore provide a powerful framework to explore state spaces and to learn from actions and rewards.

In the feedforward networks and convolutional neural networks (CNNs) studied in the previous chapters, the training data was assumed to be static, with no sequential relation among the samples. Using the data, we were able to train the networks to perform reliable classification tasks. There are many applications, however, where the input data will be sequential in nature, with one sample following another in some ordered manner, as happens with words in a sentence.