Recombination and the power of statistical tests of neutrality

Abstract

Two new test statistics were constructed to detect departures from the equilibrium neutral theory that tend to produce genealogies with longer internal branches (e.g. population subdivision or balancing selection). The new statistics are based on a measure of linkage disequilibrium between adjacent pairs of segregating sites. Simulations were run to determine the power of these and previously proposed test statistics to reject an island model of geographic subdivision. Unlike previous power studies, this one uses a coalescent model with recombination. It is found that recombination rates on the order of the mutation rate substantially reduce the power of most test statistics, and that one of the new test statistics is generally more powerful than the others. Two suggestions are made for increasing the power of the statistical tests examined here. First, they can be made more powerful if critical values are obtained from simulations that condition on a lower bound for the population recombination rate. Secondly, for the same total length sequenced, power is increased if independent loci are considered instead of a single contiguous stretch.