7 - Likelihood-Based Procedures

Summary

Consider observations X = X⁽ⁿ⁾ from law P_f indexed by a parameter space F. From a very basic perspective, statistical inference is about finding the value of f that is ‘most likely’ to have generated the observed values X = x. This perspective can be transformed into a rigorous, principled approach to any statistical problem and has resulted in the development of two paradigms of statistical inference that rely on the concept of the likelihood function.

The first approach follows the well-known maximum likelihood principle, which takes the preceding perspective literally and attempts to maximise a likelihood function which represents the joint distribution of the data as a function of f over the parameter space F. The second approach, to be introduced in more detail later, starts with a probability distribution Π on the parameter space F, often called the prior distribution, makes the assumption that X ∼ P_f conditional on f having been drawn from Π and then computes the conditional posterior distribution of f given the observations X, which is a reweighted version of the likelihood function. As the last ‘updating’ step is often based on an application of Bayes’ rule for conditional probabilities, this approach is called the Bayesian approach to statistical inference.

In this chapter we develop some basic aspects of the theory of likelihood-based inference for infinite-dimensional models F. A central role will be played by the Hellinger distance – a metric that is naturally compatible with likelihood techniques in the i.i.d. sampling model – and by the corresponding L²-distance in the Gaussian white noise model. We start with nonparametric testing problems and show that certain likelihood ratio–based procedures allow for general results in the sampling model, replacing the analytic assumptions on the functions employed in Chapter 6 by general Hellinger-distance compactness conditions. We then study the maximum likelihood principle and give a general rate of convergence result using a bracketing version of these Hellinger compactness conditions. We illustrate the theory for two concrete nonparametric maximum likelihood estimators in some detail: the cases where F equals a ball in a Sobolev space and where F equals the set of monotone decreasing densities. We will derive convergence rate results in Hellinger and related metrics, and we shall prove an infinite-dimensional version of the classical asymptotic normality result for maximum likelihood estimators.