A main motivation for the study of nonparametric models is that they do not impose potentially unrealistic finite-dimensional, or parametric, a priori restrictions. The minimax paradigm has revealed that the statistical performance of optimal nonparametric procedures depends heavily on structural properties of the parameter to be estimated and does not typically scale at the universal rate encountered in classical parametric models. This dependence arises typically through the choice of tuning parameters which require choices of usually unknown aspects of the function fto be estimated, for instance, its smoothness r and the corresponding bound on the Besov norm. The question arises as to how fully automatic procedures that do not require the specification of such parameters can perform from a minimax point of view and whether procedures exist that ‘adapt’ to the unknown values of r,B. We shall show in this chapter that full adaptation is possible in many testing and estimation problems and that mild losses occur for some adaptive testing problems. In contrast, the theory of adaptive confidence sets – and, more generally, the problem of adaptive uncertainty quantification – is more intricate, and the price for adaptation can be severe unless some additional structural assumptions on the parameter space are imposed. We shall explicitly characterise the parameter regions in nonparametric models where this discrepancy between estimation and uncertainty quantification arises and reveal the underlying relationship to certain nonparametric hypothesis-testing problems.

The theory of adaptive inference in infinite-dimensional models reveals fundamental, and in this form previously unseen, information-theoretic differences between the three main pillars of statistics, that is, between estimation, testing and the construction of confidence sets. The insights drawn from the results in this chapter belong to the most intriguing statistical findings of the nonparametric theory, showcasing the genuine challenges of statistical inference in infinite dimensions. To meet this challenge, a class of ‘self-similar’ functions will be introduced, for which a unified theory of estimation, testing and confidence sets can be demonstrated to exist.

In this chapter we introduce and motivate the statistical models that will be considered in this book. Some of the materials depend on basic facts developed in subsequent chapters – mostly the basic Gaussian process and Hilbert space theory. This will be hinted at when necessary.

Very generally speaking, a statistical model for a random observation Y is a family

{Pf : f ∈ F}

of probability distributions Pf, each of which is a candidate for having generated the observation Y. The parameter f belongs to the parameter space F. The problem of statistical inference on f, broadly speaking, can be divided into three intimately connected problems of using the observation Y to

1. (a) Estimate the parameter f by an estimator T(Y),

2. (b) Test hypotheses on f based on test functions Ψ(Y) and/or

3. (c) Construct confidence sets C(Y) that contain f with high probability.

To interpret inferential results of these kinds, we will typically need to specify a distance, or loss function on F, and for a given model, different loss functions may or may not lead to very different conclusions.

The statistical models we will introduce in this chapter are, on the one hand, conceptually closely related to each other in that the parameter space F is infinite or high dimensional and the loss functions relevant to the analysis of the performance of statistical procedures are similar. On the other hand, these models are naturally divided by the different probabilistic frameworks in which they occur – which will be either a Gaussian noise model or an independent sampling model. These frameworks are asymptotically related in a fundamental way (see the discussion after Theorem 1.2.1). However, the most effective probabilistic techniques available are based on a direct, nonasymptotic analysis of the Gaussian or product probability measures that arise in the relevant sampling context and hence require a separate treatment.

Thus, while many of the statistical intuitions are common to both the sampling and the Gaussian noise models and in fact inform each other, the probabilistic foundations of these models will be laid out independently.

Consider observations X = X(n) from law Pf indexed by a parameter space F. From a very basic perspective, statistical inference is about finding the value of f that is ‘most likely’ to have generated the observed values X = x. This perspective can be transformed into a rigorous, principled approach to any statistical problem and has resulted in the development of two paradigms of statistical inference that rely on the concept of the likelihood function.

The first approach follows the well-known maximum likelihood principle, which takes the preceding perspective literally and attempts to maximise a likelihood function which represents the joint distribution of the data as a function of f over the parameter space F. The second approach, to be introduced in more detail later, starts with a probability distribution Π on the parameter space F, often called the prior distribution, makes the assumption that X ∼ Pf conditional on f having been drawn from Π and then computes the conditional posterior distribution of f given the observations X, which is a reweighted version of the likelihood function. As the last ‘updating’ step is often based on an application of Bayes’ rule for conditional probabilities, this approach is called the Bayesian approach to statistical inference.

In this chapter we develop some basic aspects of the theory of likelihood-based inference for infinite-dimensional models F. A central role will be played by the Hellinger distance – a metric that is naturally compatible with likelihood techniques in the i.i.d. sampling model – and by the corresponding L2-distance in the Gaussian white noise model. We start with nonparametric testing problems and show that certain likelihood ratio–based procedures allow for general results in the sampling model, replacing the analytic assumptions on the functions employed in Chapter 6 by general Hellinger-distance compactness conditions. We then study the maximum likelihood principle and give a general rate of convergence result using a bracketing version of these Hellinger compactness conditions. We illustrate the theory for two concrete nonparametric maximum likelihood estimators in some detail: the cases where F equals a ball in a Sobolev space and where F equals the set of monotone decreasing densities. We will derive convergence rate results in Hellinger and related metrics, and we shall prove an infinite-dimensional version of the classical asymptotic normality result for maximum likelihood estimators.

This chapter develops some classical theory and fundamental tools for Gaussian random processes. We start with the basic definitions of Gaussian processes indexed by abstract parameter spaces and, by way of introduction to the subject, derive some elementary yet powerful properties. We present the isoperimetric and log-Sobolev inequalities for Gaussian measures in ℝn and apply them to establish concentration properties for the supremum of a Gaussian process about its median and mean, which are some of the deepest and most useful results on Gaussian processes. Then we introduce Dudley's metric entropy bounds for moments of suprema of (sub-) Gaussian processes as well as for their a.s. modulus of continuity. The chapter also contains a thorough discussion of convexity and comparison properties of Gaussian measures and of reproducing kernel Hilbert spaces and ends with an exposition of the limit theory for suprema of stationary Gaussian processes.

Definitions, Separability, 0-1 Law, Concentration

We start with some preliminaries about stochastic processes, mainly to fix notation and terminology. Then these concepts are specialised to Gaussian processes, and some first properties of Gaussian processes are developed. The fundamental observation is that a Gaussian process X indexed by a a set T induces an intrinsic distance dX on T (dX(s,t) is the L2-distance between X(s) and X(t)), and all the probabilistic information about X is contained in the metric or pseudo-metric space (T,d). This is tested on some of the first properties, such as the 0-1 law and the existence of separable versions of X. One of the main properties of Gaussian processes, namely, their concentration about the mean, is introduced; this subject will be treated in the next section, but a first result on it, which is not sharp but that has been chosen for its simplicity, is given in this section.

Large-Dimensional Data and New Asymptotic Statistics

In a multivariate analysis problem, we are given a sample x1, x2, …, xn of random observations of dimension p. Statistical methods, such as principal component analysis, have been developed since the beginning of the 20th century. When the observations are Gaussian, some nonasymptotic methods exist, such as Student's test, Fisher's test, or the analysis of variance. However, in most applications, observations are non-Gaussian, at least in part, so that nonasymptotic results become hard to obtain and statistical methods are built using limiting theorems on model statistics.

Most of these asymptotic results are derived under the assumption that the data dimension p is fixed while the sample size n tends to infinity (large sample theory). This theory had been adopted by most practitioners until very recently, when they were faced with a new challenge: the analysis of large dimensional data.

Large-dimensional data appear in various fields for different reasons. In finance, as a consequence of the generalisation of Internet and electronic commerce supported by the exponentially increasing power of computing, online data from markets around the world are accumulated on a giga-octet basis every day. In genetic experiments, such as micro-arrays, it becomes possible to record the expression of several thousand of genes from a single tissue. Table 1.1 displays some typical data dimensions and sample sizes. We can see from this table that the data dimension p is far from the “usual” situations where p is commonly less than 10. We refer to this new type of data as large-dimensional data.

It has been observed for a long time that several well-known methods in multivariate analysis become inefficient or even misleading when the data dimension p is not as small as, say, several tens. A seminal example was provided by Dempster in 1958, when he established the inefficiency of Hotelling's T2 in such cases and provided a remedy (named a non-exact test). However, by that time, no statistician was able to discover the fundamental reasons for such a breakdown in the well-established methods.