8 - Sequential data – 1D statistics

Summary

The stock market is an excellent economic forecaster. It has predicted six of the last three recessions.

(Paul Samuelson)

In contrast to previous chapters, we now consider data transformation, how to transform data in order to produce better statistics, either to extract signal or to enhance signal.

There are many observations consisting of sequential data, such as intensity as a function of position as a radio telescope is scanned across the sky or as signal varies across a row on a CCD detector, single-slit spectra, time-measurements of intensity (or any other property). What sort of issues might concern us?

1. baseline detection and/or assessment, so that signal on this baseline can be analysed;

2. signal detection, identification for example of a spectral line or source in sequential data for which the noise may be comparable in magnitude to the signal;

3. filtering to improve signal-to-noise ratio;

4. quantifying the noise;

5. period-finding; searching the data for periodicities;

6. trend-finding; can we predict the future behaviour of subsequent data?

7. correlation of time series to find correlated signal between antenna pairs or to find spectral lines;

8. modelling; many astronomical systems give us our data convolved with some more-or-less known instrumental function, and we need to take this into account to get back to the true data.

The distinctive aspect of these types of analysis is that the feature of interest only emerges after a transformation.