5 - Hypothesis testing

Summary

How do our data look?

I've carried out a Kolmogorov–Smirnov test …

Ah. Thatbad.

(interchange between Peter Scheuer and his then student, CRJ)

(The) premise that statistical significance is the only reliable indication of causation is flawed.

(US Supreme Court, Matrixx Initiatives, Inc. vs. Siracusano, 22 March 2011)

It is often the case that we need to do sample comparison: we have someone else's data to compare with ours; or someone else's model to compare with our data; or even our data to compare with our model. We need to make the comparison and to decide something. We are doing hypothesis testing – are our data consistent with a model, with somebody else's data? In searching for correlations as we were in Chapter 4, we were hypothesis testing; in the model-fitting of Chapter 6 we are involved in data modelling and parameter estimation.

A frequentist point of view might be to consider the entire science of statistical inference as hypothesis testing followed by parameter estimation. However, if experiments were properly designed, the Bayesian approach would be right: it answers the sample-comparison questions we wished to pose in the first place, namely what is the probability, given the data, that a particular model is right? Or: what is the probability, given two sets of data, that they agree? The two-stage process should be unecessary at best.