10 - How to get away with peeking at data

Summary

Examples

Paulo watched two male Mus domesticus (house mouse) interacting to establish dominance. For each 3 second interval, he recorded + if a mouse did a particular behavior, such as groom himself, or – if that behavior was not performed. This sequence was recorded for one of the behaviors of the finally dominant mouse.

– – – + – – + – + – – – – + – + – – + – + + – – + – – + + – + – –

Paulo wondered if the frequency with which the mouse performed this behavior changed at some point in the interaction; perhaps at the time he began to assume dominance. So Paulo looked through the data sequence for the time when the difference between the frequency before and the frequency after is the greatest. Then he used the hypothesis that the behavior is performed independently with the same probability for each interval to test if the frequency with which the behavior is performed before this time is different from that after this time. Somehow, he was not surprised that these frequencies turned out to be significantly different. After all, he peeked at the sequence to divide it at the point of maximal difference.

Jennifer and her colleagues observed fossil Forams (little shelled animals that lived at the bottom of the sea hundreds of millions of years ago) that had changed size through evolutionary time. Looking at a plot of size vs time before present, Forams seemed to be getting bigger. In addition, the plot of points seemed to bend at a particular time in the past, suggesting that the rate of increase after that time was more rapid. Had the evolutionary rate changed at that time? So they used the computational ANCOVA technique, described in Section 9.3, to test for equality of rate before and after that time. Somehow, they were not surprised that the rates were different. After all, they had peeked at the data to find the time before and after which rate difference was greatest.