In this chapter we discuss some classical distributions and their manipulations. In previous chapters we've assumed that we know the distributions and have concentrated on the inference problem. In machine learning we will typically not fully know the distributions and need to learn them from available data. This means we need familiarity with standard distributions, for which the data will later be used to set the parameters.

Representing data

The numeric encoding of data can have a significant effect on performance and an understanding of the options for representing data is therefore of considerable importance. We briefly outline three central encodings below.

Categorical

For categorical (or nominal) data, the observed value belongs to one of a number of classes, with no intrinsic ordering, and can be represented simply by an integer. An example of a categorical variable would be the description of the type of job that someone does, e.g. healthcare, education, financial services, transport, homeworker, unemployed, engineering etc. which could be represented by the values 1, 2, …, 7. Another way to transform such data into numerical values would be to use 1-of-m encoding. For example, if there are four kinds of jobs: soldier, sailor, tinker, spy, we could represent a soldier as (1,0,0,0), a sailor as (0,1,0,0), a tinker as (0,0,1,0) and a spy as (0,0,0,1). In this encoding the distance between the vectors representing two different professions is constant.

Machine learning is the body of research related to automated large-scale data analysis. Historically, the field was centred around biologically inspired models and the long-term goals of much of the community are oriented to producing models and algorithms that can process information as well as biological systems. The field also encompasses many of the traditional areas of statistics with, however, a strong focus on mathematical models and also prediction. Machine learning is now central to many areas of interest in computer science and related large-scale information processing domains.

Styles of learning

Broadly speaking the main two subfields of machine learning are supervised learning and unsupervised learning. In supervised learning the focus is on accurate prediction, whereas in unsupervised learning the aim is to find compact descriptions of the data. In both cases, one is interested in methods that generalise well to previously unseen data. In this sense, one distinguishes between data that is used to train a model and data that is used to test the performance of the trained model, see Fig. 13.1. We discuss first the basic characteristics of some learning frameworks before discussing supervised learning in more detail.

Supervised learning

Consider a database of face images, each represented by a vector x. Along with each image x is an output class y ∈ {male, female} that states if the image is of a male or female.

Machine learning is fundamentally about extracting value from large datasets. Often themotivation is ultimately to produce an algorithm that can either mimic or enhance human/biological performance.

In Part III we begin by discussing some of the basic concepts in machine learning, namely supervised and unsupervised learning. We then move on to discuss some standard models in machine learning and related areas.

High-dimensional data is prevalent in machine learning and related areas. Indeed, there often arises the situation in which there are more data dimensions than there are data examples. In such cases we seek a lower-dimensional representation of the data. In this chapter we discuss some standard methods which can also improve the prediction performance by removing ‘noise’ from the representation.

High-dimensional spaces – low-dimensional manifolds

In machine learning problems data is often high dimensional – images, bag-of-word descriptions, gene-expressions etc. In such cases we cannot expect the training data to densely populate the space, meaning that there will be large parts in which little is known about the data. For the hand written digits from Chapter 14, the data is 784 dimensional and for binary valued pixels the number of possible images is 2784 ≈ 10236. Nevertheless, we would expect that only a handful of examples of a digit should be sufficient (for a human) to understand how to recognise a 7. Digit-like images must therefore occupy a highly constrained volume in the 784 dimensions and we expect only a small number of degrees of freedom to be required to describe the data to a reasonable accuracy. Whilst the data vectors may be very high dimensional, they will therefore typically lie close to a much lower-dimensional ‘manifold’ (informally, a two-dimensional manifold corresponds to a warped sheet of paper embedded in a high-dimensional space), meaning that the distribution of the data is heavily constrained.

Timeseries require specialised models since the number of variables can be very large and typically increases as new datapoints arrive. In this chapter we discuss models in which the process generating the observed data is fundamentally discrete. These models give rise to classical models with interesting applications in many fields from finance to speech processing and website ranking.

Markov models

Timeseries are datasets for which the constituent datapoints can be naturally ordered. This order often corresponds to an underlying single physical dimension, typically time, though any other single dimension may be used. The timeseries models we consider are probability models over a collection of random variables v 1, …, vT with individual variables vt indexed by discrete time t. A probabilistic timeseries model requires a specification of the joint distribution p(v 1, …, vT ). For the case in which the observed data vt are discrete, the joint probability table for p(v 1, …, vT ) has exponentially many entries.We therefore cannot expect to independently specify all the exponentially many entries and need to make simplified models under which these entries can be parameterised in a lower-dimensional manner. Such simplifications are at the heart of timeseries modelling and we will discuss some classical models in the following sections.

In Chapter 3 we saw how belief networks are used to represent statements about independence of variables in a probabilistic model. Belief networks are simply one way to unite probability and graphical representation. Many others exist, all under the general heading of ‘graphical models’. Each has specific strengths and weaknesses. Broadly, graphical models fall into two classes: those useful for modelling, such as belief networks, and those useful for inference. This chapter will survey the most popular models from each class.

Graphical models

Graphical Models (GMs) are depictions of independence/dependence relationships for distributions. Each class of GM is a particular union of graph and probability constructs and details the form of independence assumptions represented. Graphical models are useful since they provide a framework for studying a wide class of probabilistic models and associated algorithms. In particular they help to clarify modelling assumptions and provide a unified framework under which inference algorithms in different communities can be related.

It needs to be emphasised that all forms of GM have a limited ability to graphically express conditional (in)dependence statements [281]. As we've seen, belief networks are useful formodelling ancestral conditional independence. In this chapter we'll introduce other types of GM that are more suited to representing different assumptions. Here we'll focus on Markov networks, chain graphs (which marry belief and Markov networks) and factor graphs. There are many more inhabitants of the zoo of graphical models, see [73, 314].