10 - Nondetections: censored and truncated data

Summary

The astronomical context

Observational astronomers always struggle, and often fail, to characterize celestial populations in an unbiased fashion. Many surveys are flux-limited (or, as expressed in traditional optical astronomy, magnitude-limited) so that only the brighter objects are detected. As flux is a convolution of the object's intrinsic luminosity and the (often uninteresting) distance to the observer according to Flux = L/4πd², this produces a sample with a complicated bias in luminosity: high-luminosity objects at large distances are over-represented and lowluminosity objects are under-represented in a flux-limited survey. This and related issues with nondetections have confronted astronomers for nearly 200 years.

A blind astronomical survey of a portion of the sky is thus truncated at the sensitivity limit, where truncation indicates that the undetected objects, even the number of undetected objects, are entirely missing from the dataset. In a supervised astronomical survey where a particular property (e.g. far-infrared luminosity, calcium line absorption, CO molecular line strength) of a previously defined sample of objects is sought, some objects in the sample may be too faint to detect. The dataset then contains the full sample of interest, but some objects have upper limits and others have detections. Statisticians refer to upper limits as left-censored data points.

Multivariate problems with censoring and truncation also arise in astronomy. Consider, for example, a study of howthe luminosity function of active galactic nuclei (AGN) depends on covariates such as redshift (as a measure of cosmic time), clustering environment, host galaxy bulge luminosity and starburst activity.