7 - Data modelling and parameter estimation: advanced topics

Summary

Frustra fit per plura quod potest fieri per pauciora – it is futile to do with more things that which can be done with fewer.

(William of Ockham, c.1285–1349)

Nature laughs at the difficulties of integration.

(Pierre-Simon de Laplace, 1749–1827; Gordon & Sorkin, 1959)

One of the attractive features of the Bayesian method is that it offers a principled way of making choices between models. In classical statistics, we may fit to a model, say by least squares, and then use the resulting χ² statistic to decide if we should reject the model. We would do this if the deviations from the model are unlikely to have occurred by chance. However, it is not clear what to do if the deviations are likely to have occurred, and it is even less clear what to do if several models are available. For example, if a model is in fact correct, the significance level derived from a χ² test (or, indeed, any significance test) will be uniformly distributed between zero and one (Exercise 7.1).

The problem with model choice by χ² (or any similar classical method) is that these methods do not answer the question we wish to ask. For a model H and data D, a significance level derived from a minimum χ² tells us about the conditional probability, prob(D | H).