Something important changed in the world of statistics in the new millennium. Twentieth-century statistics, even after the heated expansion of its late period, could still be contained within the classic Bayesian–frequentist– Fisherian inferential triangle (Figure 14.1). This is not so in the twenty-first century. Some of the topics discussed in Part III—false-discovery rates, post-selection inference, empirical Bayes modeling, the lasso—fit within the triangle but others seem to have escaped, heading south from the frequentist corner, perhaps in the direction of computer science.

The escapees were the large-scale prediction algorithms of Chapters 17– 19: neural nets, deep learning, boosting, random forests, and support-vector machines. Notably missing from their development were parametric probability models, the building blocks of classical inference. Prediction algorithms are the media stars of the big-data era. It is worth asking why they have taken center stage and what it means for the future of the statistics discipline.

The why is easy enough: prediction is commercially valuable. Modern equipment has enabled the collection of mountainous data troves, which the “data miners” can then burrow into, extracting valuable information. Moreover, prediction is the simplest use of regression theory (Section 8.4). It can be carried out successfully without probability models, perhaps with the assistance of nonparametric analysis tools such as cross-validation, permutations, and the bootstrap.

A great amount of ingenuity and experimentation has gone into the development of modern prediction algorithms, with statisticians playing an important but not dominant role.1 There is no shortage of impressive success stories. In the absence of optimality criteria, either frequentist or Bayesian, the prediction community grades algorithmic excellence on per-formance within a catalog of often-visited examples such as the spam and digits data sets of Chapters 17 and 18.2 Meanwhile, “traditional statistics” —probability models, optimality criteria, Bayes priors, asymptotics—has continued successfully along on a parallel track. Pessimistically or optimistically, one can consider this as a bipolar disorder of the field or as a healthy duality that is bound to improve both branches. There are historical and intellectual arguments favoring the optimists’ side of the story.

By the final decade of the twentieth century, electronic computation fully dominated statistical practice. Almost all applications, classical or otherwise, were now performed on a suite of computer platforms: SAS, SPSS, Minitab, Matlab, S (later R), and others.

The trend accelerates when we enter the twenty-first century, as statistical methodology struggles, most often successfully, to keep up with the vastly expanding pace of scientific data production. This has been a twoway game of pursuit, with statistical algorithms chasing ever larger data sets, while inferential analysis labors to rationalize the algorithms. Part III of our book concerns topics in twenty-first-century1 statistics.

The word “topics” is intended to signal selections made from a wide catalog of possibilities. Part II was able to review a large portion (though certainly not all) of the important developments during the postwar period. Now, deprived of the advantage of hindsight, our survey will be more illustrative than definitive.

For many statisticians, microarrays provided an introduction to largescale data analysis. These were revolutionary biomedical devices that enabled the assessment of individual activity for thousands of genes at once— and, in doing so, raised the need to carry out thousands of simultaneous hypothesis tests, done with the prospect of finding only a few interesting genes among a haystack of null cases. This chapter concerns large-scale hypothesis testing and the false-discovery rate, the breakthrough in statistical inference it elicited.

Large-Scale Testing

The prostate cancer data, Figure 3.4, came from a microarray study of n = 102 men, 52 prostate cancer patients and 50 normal controls. Each man's gene expression levels were measured on a panel of N = 6033 genes, yielding a 6033 102 matrix of measurements xij,

For each gene, a two-sample t statistic (2.17) ti was computed comparing gene i 's expression levels for the 52 patients with those for the 50 controls. Under the null hypothesis H0i that the patients’ and the controls’ responses come from the same normal distribution of gene i expression levels, ti will follow a standard Student t distribution with 100 degrees of freedom, t100.

The fundamentals of statistical inference—frequentist, Bayesian, Fisherian —were set in place by the end of the first half of the twentieth century, as discussed in Part I of this book. The postwar era witnessed a massive expansion of statistical methodology, responding to the data-driven demands of modern scientific technology. We are now at the end of Part II, “Early Computer-Age Methods,” having surveyed the march of new statistical algorithms and their inferential justification from the 1950s through the 1990s.

This was a time of opportunity for the discipline of statistics, when the speed of computation increased by a factor of a thousand, and then another thousand. As we said before, a land bridge had opened to a new continent, but not everyone was eager to cross. We saw a mixed picture: the computer played a minor or negligible role in the development of some influential topics such as empirical Bayes, but was fundamental to others such as the bootstrap.

Fifteen major topics were examined in Chapters 6 through 13. What follows is a short scorecard of their inferential affinities, Bayesian, frequentist, or Fisherian, as well as an assessment of the computer's role in their development. None of this is very precise, but the overall picture, illustrated in Figure 14.1, is evocative.

Empirical Bayes

Robbins’ original development of formula (6.5) was frequentistic, but most statistical researchers were frequentists in the postwar era so that could be expected. The obvious Bayesian component of empirical Bayes arguments is balanced by their frequentist emphasis on (nearly) unbiased estimation of Bayesian estimators, as well as the restriction to using only current data for inference. Electronic computation played hardly any role in the theory's development (as indicated by blue coloring in the figure). Of course mod-ern empirical Bayes applications are heavily computational, but that is the case for most methods now.

James–Stein and Ridge Regression

The frequentist roots of James–Stein estimation are more definitive, especially given the force of the James–Stein theorem (7.16). Nevertheless, the empirical Bayes interpretation (7.13) lends James–Stein some Bayesian credibility. Electronic computation played no role in its development.

The classical theory of statistics was developed for parametric models with finite-dimensional parameter spaces, building on fundamental ideas of C. F. Gauss, R. A. Fisher and L. Le Cam, among others. It has been successful in providing modern science with a paradigm for making statistical inferences, in particular, in the ‘frequentist large sample size’ scenario. A comprehensive account of the mathematical foundations of this classical theory is given in the monograph by A. van der Vaart, Asymptotic Statistics (Cambridge University Press, 1998).

The last three decades have seen the development of statistical models that are infinite (or ‘high’) dimensional. The principal target of statistical inference in these models is a function or an infinite vector f that itself is not modelled further parametrically. Hence, these models are often called, in some abuse of terminology, nonparametric models, although f itself clearly also is a parameter. In view of modern computational techniques, such models are tractable and in fact attractive in statistical practice. Moreover, a mathematical theory of such nonparametric models has emerged, originally driven by the Russian school in the early 1980s and since then followed by a phase of very high international activity.

This book is an attempt to describe some elements of the mathematical theory of statistical inference in such nonparametric, or infinite-dimensional, models. We will first establish the main probabilistic foundations: the theory of Gaussian and empirical processes, with an emphasis on the ‘nonasymptotic concentration of measure’ perspective on these areas, including the pathbreaking work by M. Talagrand and M. Ledoux on concentration inequalities for product measures. Moreover, since a thorough understanding of infinite-dimensional models requires a solid background in functional analysis and approximation theory, some of the most relevant results from these areas, particularly the theory of wavelets and of Besov spaces, will be developed from first principles in this book.

After these foundations have been laid, we turn to the statistical core of the book. Comparing nonparametric models in a very informal way with classical parametric models, one may think of them as models in which the number of parameters that one estimates from the observations is growing proportionally to sample size n and has to be carefully selected by the statistician, ideally in a data-driven way.