3 - Bayesian methods

Summary

This chapter develops the key ideas in the Bayesian approach to inference. Fundamental ideas are described in Section 3.1. The key conceptual point is the way that the prior distribution on the unknown parameter θ is updated, on observing the realised value of the data x, to the posterior distribution, via Bayes’ law. Inference about θ is then extracted from this posterior. In Section 3.2 we revisit decision theory, to provide a characterisation of the Bayes decision rule in terms of the posterior distribution. The remainder of the chapter discusses various issues of importance in the implementation of Bayesian ideas. Key issues that emerge, in particular in realistic data analytic examples, include the question of choice of prior distribution and computational difficulties in summarising the posterior distribution. Of particular importance, therefore, in practice are ideas of empirical Bayes inference (Section 3.5), Monte Carlo techniques for application of Bayesian inference (Section 3.7) and hierarchical modelling (Section 3.8). Elsewhere in the chapter we provide discussion of Stein's paradox and the notion of shrinkage (Section 3.4). Though not primarily a Bayesian problem, we shall see that the James–Stein estimator may be justified (Section 3.5.1) as an empirical Bayes procedure, and the concept of shrinkage is central to practical application of Bayesian thinking. We also provide here a discussion of predictive inference (Section 3.9) from a Bayesian perspective, as well as a historical description of the development of the Bayesian paradigm (Section 3.6).