29 - Desperate Data Analysis by a Desperate Job Candidate

Summary

I study the ways that emotions and other motivations bias moral reasoning, and I inadvertently demonstrated the thesis while trying to prove it. I had just finished my first postdoc and had failed to get an academic job. I found another postdoc and was desperate to get more manuscripts under review at top journals before sending in the next year’s applications. I had begun a line of experiments in which I exposed people to disgusting (or non-disgusting) images and stories and then measured their moral condemnation on subsequent stories. I was looking for carryover effects of disgust.

I recruited participants in a public park in Philadelphia. The means were different across the two conditions, but the t-test was not significant because the variance was high – there were several outliers. I scrutinized those outliers carefully and realized that one of them was a guy who was smoking marijuana when I recruited him. Doesn’t that justify excluding him? Maybe, but then what about the outlier on the other side, who was drinking beer while filling out the survey?

I wrestled with this problem for a while, searching for principles that would allow me to exclude the outliers that I wanted to exclude. I found a small set of principles that – with some stretching – allowed me to exclude three outliers that hurt my case while only losing one that helped me. I knew I was doing this post hoc, and that it was wrong to do so. But I was so coni dent that the effect was real, and I had defensible justii cations! I made a deal with myself: I would go ahead and write up the manuscript now, without the outliers, and while it was under review I would collect more data, which would allow me to get the result cleanly, including all outliers.