9 - Sequential data – 1D statistics

Summary

The stock market is an excellent economic forecaster. It has predicted six of the last three recessions.

(Paul Samuelson)

The only function of economic forecasting is to make astrology look respectable.

(John Kenneth Galbraith)

In contrast to previous chapters, we now consider data transformation, how to transform data in order to produce improved outcomes in either extracting or enhancing signal.

There are many observations consisting of sequential data, such as intensity as a function of position as a radio telescope is scanned across the sky or as signal varies across a row on a CCD detector, single-slit spectra, time-measurements of intensity (or any other property). What sort of issues might concern us?

1. (i) trend-finding; can we predict the future behaviour of data?

2. (ii) baseline detection and/or assessment, so that signal on this baseline can be analysed;

3. (iii) signal detection, identification, for example, of a spectral line or source in sequential data for which the noise may be comparable in magnitude to the signal;

4. (iv) filtering to improve signal-to-noise ratio;

5. (v) quantifying the noise;

6. (vi) period-finding; searching the data for periodicities;

7. (vii) correlation of time series to find correlated signal between antenna pairs or to find spectral lines;

8. (viii) modelling; many astronomical systems give us our data convolved with some more or less known instrumental function, and we need to take this into account to get back to the true data.