Formulating a hypothesis

Up until now we have discussed how to define a measurement in terms of a central value, uncertainties, and units, as well as how to extend these concepts to encompass confidence levels (both one- and two-sided). A related aspect of performing a measurement is to test a theory or, as it is usually phrased, a hypothesis. For example, consider the case where a theorist writes a paper proposing the existence of some effect that can be tested via some physical process. It is then down to an experimentalist to develop a method that can be used to test the validity of that theory. In this example the default hypothesis (usually referred to as the null hypothesis and often denoted by H0) would be that the theory is valid, and the experimenter would then embark on a measurement that could be used to test the null hypothesis.

Having defined a null hypothesis, by default the complement of that hypothesis exists as an alternative hypothesis often denoted by H1. Given data we can test the null hypothesis to see if the data are in agreement or in disagreement with it. In order to qualify what is meant by data being in agreement with the null hypothesis, we need to quantify what we consider to be agreement (at some C L).

This section summarises many of the terms used in this book. A brief description of the meaning of each term is given, and where appropriate a cross reference to the corresponding section in the text is suggested for further reading.

∧: Logical and.

∨: Logical or.

X2: See chi-square.

μ, μ(i)or μi: See mean.

ρ, ρ(x, y)or ρxy: See correlation.

σ, σ(i)or σi: See standard deviation.

σxy or σ(x, y)or σij: See covariance.

Alternate hypothesis (H1): The complement of a null hypothesis. If a statistical test is performed for a null hypothesis, and that test fails, then the alternate hypothesis (as the complement of the null hypothesis) is the result supported by the data.

Arithmetic mean: See mean.

Artificial neural network: Also called a neural network. This is a collection of perceptrons fed by some number of input variables, with one or more output nodes. The function of the neural network is to try and optimally distinguish between two or more types of data. See Section 10.4.

Bagging: This is the name given to a technique of oversampling data used for training decision trees. See Section 10.5.2.

Bayes' theorem: See Eq. (3.4).

Bayesian classifier: A classification algorithm based on Bayes' theorem, see Section 10.2.

Bayesian statistics: The branch of statistics derived from the work of Rev. Bayes. This is a subjective approach to the computation of probability that can be used in a variety of situations, including for those that are not repeatable. See Section 3.2.

This chapter introduces four important distributions that can be used to describe a variety of situations. The first distribution encountered is that of the binomial distribution (Section 5.2). This is used to understand problems where the possible outcomes are binary, and usually categorised in terms of success and failure. For example, one can consider the situation of either detecting of failing to detect a particle passing through some apparatus as a binary event. The detection efficiency in this particular problem is the parameter p of the binomial distribution. Typically one finds that p ~ 1 when working with efficient detectors. The Poisson distribution (Section 5.3) can be used to understand rare events where the total number of trials is not necessarily known, and the distribution depends on only the number of observed events and a single parameter λ that is both the mean and variance of the distribution. For example, the Poisson distribution can be used to describe the uncertainties on the content of each bin in Figure 1.2, which is a topic discussed in more detail in Chapter 7. The third distribution discussed here is the Gaussian distribution (Section 5.4). This plays a significant role in describing the uncertainties on measurements where the number of data are large. Finally the X2 distribution is introduced in Section 5.5.

The use of statistical data analysis in physics means different things to different people. The reason for this is that most problems are different, and so someone concentrating on areas where the experimental data collected are relatively straightforward to analyse will naturally tend to use techniques that are simpler than those required for a more complicated problem or for a sparse data sample. Ultimately we all need to use statistical methods in order to translate data into some measure of a physical observable. This book will discuss a number of different concepts and techniques in order of increasing complexity. Before embarking on a discussion of statistics the remainder of this chapter introduces three common experimental problems encountered by students studying physics: (i) using a pendulum to measure acceleration due to gravity (Section 1.1), (ii) testing the validity of Ohm's law for a conductor (Section 1.2), and (iii) measuring the half-life of a radioactive isotope (Section 1.3). These examples rely on material covered in Chapters 4 through 7 and Chapter 9. Readers who appreciate the context of material in the remainder of this book may wish to skip forward to the next chapter.

Consider a data sample Ω described by the set of variables x that is composed of two (or more) populations. Often we are faced with the task of trying to identify or separate one sub-sample from the other (as these are different classes or types of events). In practice it is often not possible to completely separate samples of one class A from another class B as was seen in the case of likelihood fits to data. There are a number of techniques that can be used in order to try and optimally identify or separate a sub-sample of data from the whole, and some of these are described below. Each of the techniques described has its own benefits and disadvantages, and the final choice of the ‘optimal’ solution of how to separate A and B can require subjective input from the analyst. In general this type of situation requires the use of multivariate analysis (MVA).

The simplest approach is that of cutting on the data to improve the purity of a class of events, as described in Section 10.1. More advanced classifiers such as Bayesian classifiers, Fisher discriminants, neural networks, and decision trees are subsequently discussed. The Fisher discriminant described in Section 10.3 has the advantage that the coefficients required to optimally separate two populations of events are determined analytically up to an arbitrary scale factor. The neural network (Section 10.4) and decision tree (Section 10.5) algorithms described here require a numerical optimisation to be performed.

This chapter develops the notion introduced in Chapter 6 on how one defines a statistical error and extends this to look at one- and two-sided intervals (see Sections 7.1 and 7.2). One-sided intervals fall into the categories of upper or lower limits, which we can place on a hypothesised effect or process that has not been observed (see Section 7.2). Each of these concepts can be interpreted in terms of a frequentist or Bayesian approach. Section 9.7.3 discusses the concept of Bayesian upper limits in the context of a fit to data, and Appendix E contains tables of integrals of several common PDFs that can be used to determine confidence intervals and limits.

Two-sided intervals

The relevance of two-sided confidence intervals is discussed in the context of useful distributions used to represent PDFs. In particular the following sections highlight the use of Gaussian, Poisson, and binomial distributions as particular use cases of such intervals.

In general, for a distribution f(x) given by some variable x, we can define some region of interest in x called a two-sided interval such that x1 < x < x2.

Before embarking upon a detailed discussion of statistics, it is useful to introduce some notation to help describe data. This section introduces elementary set theory notation and Venn diagrams.

The notion of a set is a collection of objects or elements. This collection can also be referred to as data, and the individual elements in the data can themselves be referred to as data (in the singular sense), or as an event or element. We usually denote a set with a capital letter, for example Ω. The element of a set is denoted by a lower case letter, for example either ω or ωi, where the latter explicitly references the ith element of the set. The elements of a set are written within curly braces ‘{‘ and ‘}’. For example we can write a set ΩBinary that contains the elements 1 and 0 as

This is called the binary set, as it contains the elements required to represent a binary system. The order of elements in a set is irrelevant, so we could write ΩBinary in an equivalent form as

If we want to express the information that a given element is or is not a part of a set, we use the symbols ∈ and ∉, respectively. For example we may write

to express that both 0 and 1 are elements of ΩBinary, but 2 is not an element of this set.

2. Sets

1. 2.1 (i) A ∪ B = {1, 2, 3, 4, 5, 9}; (ii) A ∩ B = {1, 3}; (iii) A \ B = {2, 4};(iv) B \ A = {5, 9};(v) Ā = {5, 9}, and B = {2, 4}

2. 2.2 (i) A ∪ B = {0, 1, 2, 3, 5, 6, 9};(ii) A ∩ B = {5}; (iii) A \ B = {0, 2, 6};(iv) B \ A = {1, 3, 9}

3. 2.3 (i) For example see the top part of Figure 2.3; (ii) see Figure 2.2, where A and B are interchanged.

4. 2.4 A ∪ B = R

5. 2.5 A ∪ B = R

6. 2.6 A ∩ B = 1, 2, 3, 4, 5

7. 2.7 A ∩ B = ø

8. 2.8 A ∪ B ∩ C = Z+

9. 2.9 A ∪ B ∩ C = {0, Z-}

10. 2.10 ΩDecimal = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}

11. 2.11 {red, green, blue}

12. 2.12 C

13. 2.13 Z

14. 2.14 {x|5 < x < 10}

15. 2.15 {x |x ∈ R}

3. Probability

1. 3.1 4/52 = 0.077 to 3 s.f.

2. 3.2 0.45%

3. 3.3 20/52 = 0.385 to 3 s.f.

4. 3.4 The probability of the getting an ace followed by a 10 point card, where the other player also gets a 10 point card is 0.72%. The probability of getting an ace followed by a 10 point card, where the other player gets a card with a different value is 1.69%. The total probability of being dealt an ace followed by a 10 point card is the sum: 2.41%.

5. […]

The foundations of science are built upon centuries of careful observation. These constitute measurements that are interpreted in terms of hypotheses, models, and ultimately well-tested theories that may stand the test of time for only a few years or for centuries. In order to understand what a single measurement means we need to appreciate a diverse range of statistical methods. Without such an appreciation it would be impossible for scientific method to turn observations of nature into theories that describe the behaviour of the Universe from sub-atomic to cosmic scales. In other words science would be impracticable without statistical data analysis. The data analysis principles underpinning scientific method pervade our everyday lives, from the use of statistics we are subjected to through advertising to the smooth operation of SPAM filters that we take for granted as we read our e-mail. These methods also impact upon the wider economy, as some areas of the financial industry use data mining and other statistical techniques to predict trading performance or to perform risk analysis for insurance purposes.

This book evolved from a one-semester advanced undergraduate course on statistical data analysis for physics students at Queen Mary, University of London with the aim of covering the rudimentary techniques required for many disciplines, as well as some of the more advanced topics that can be employed when dealing with limited data samples. This has been written by a physicist with a non-specialist audience in mind. This is not a statistics book for statisticians, and references have been provided for the interested reader to refer to for more rigorous treatment of the techniques discussed here.

The probability of something occurring is the quantification of the chance of observing a particular outcome given a single event. The event itself may be the result of a single experiment, or one single data point collected by an un-repeatable experiment. We refer to a single event or an ensemble of events as data, and the way we refer to data implies if data is singular or plural. If we quantify the probability of a repeatable experiment, then this understanding can be used to make predictions of the outcomes of future experiments. We cannot predict the outcome of a given experiment with certainty; however, we can assign a level of confidence to our predictions that incorporates the uncertainty from our previous knowledge and any information of the limitations of the experiment to be performed.

Consider the following. A scientist builds an experiment with two distinct outputs A and B. Having prepared the experiment, the apparatus is configured to always return the result A, and never return the result B. If the experiment is performed over and over again one will always obtain the result A with certainty. The probability of obtaining this result is 1.0 (100%). The result B will never be observed, and so the probability of obtaining that result is 0.0 (0%).