28 - When the Data and Theory Don’t Match

Summary

A few years ago, a study in my lab produced a pattern of results that was not only unexpected but inconsistent with a theory that my collaborators and I had proposed several years before. Making the situation even worse, the finding was directly implied by a competing theory that we aimed to refute. Our theory predicted that repeated exposure to two co-occurring stimuli would form a mental association between the two stimuli even when people reject the co-occurrence as meaningless or invalid. A useful example to illustrate this hypothesis is the concern that repeated claims of Barack Obama being Muslim may create a mental association between Obama and Muslim even when people know that the claim is factually wrong. This possibility is explicitly denied by theories assuming that newly formed memory representations depend on how people construe co-occurrences and whether they consider them as valid or invalid. Consistent with the latter theories, and in contrast to the predictions of our own theory, our study showed that the effects of repeated exposure to information about other individuals were generally qualified by the perceived validity of this information; there was no evidence for unqualified message effects that were independent of perceived validity. My graduate student and I replicated this pattern in three independent studies, so there was no question about its reliability. Yet, a major question was: What should we do with the data? Should we publish them and discredit our own theory? Or should we ignore the data and pretend that our theory is correct despite our discovery that one of its central predictions has failed?

We eventually decided to submit the data for publication, and after an initial rejection the paper was accepted pending minor revisions at another journal. It was not easy to state in the paper that our theory includes an incorrect assumption, but the data ultimately helped us better understand the phenomena our theory had been designed to explain. Since the paper came out, some people have asked me why we invested so much effort into conducting and publishing research that discredits our own theory. Looking back, I still think it was the right thing to do, because the data told us something important that was inconsistent with what we believed at that time. Two years later, someone else published a study on the same question using a different operationalization. Their results confirmed the original prediction of our theory, so it turned out that we were not completely mistaken with our initial assumptions. However, taken together, the two articles suggest that our theory is at least incomplete, in that it fails to specify an important moderator of the predicted effect (which still needs to be identified). And that’s important to know if our goal is to advance science instead of pursuing our own personal agenda.