35 - “Getting It Right” Can Also Be Wrong

Summary

My student and I sat at my desk discussing the results – or rather the lack thereof – of our recent lab study. We had already conducted one experiment that provided support for our hypothesis, which we regarded as new and interesting. That first study, however, involved purely attitudinal measures, and the obvious next step was to assess actual behavior. We selected a behavioral task that appeared in the literature, made a few changes to our experimental protocol, and ran the study with ample participants per cell. Lo and behold, we found no support for our hypothesis. So there we were in my office, discussing our nonsignificant results and our next step. As good Popperians, we remained true to our hypothesis; we discussed the next study to be conducted, one that we hoped would make use of a “better” task to assess our participants’ behavior – that is, a task that would produce our expected effect. This time we would “get it right.”

We didn’t seriously question our hypothesis; rather we scrutinized features of the experiment. We went back to the literature to seek an alternative behavioral measure and found one we believed would serve our purpose better. We are now conducting this revamped study. Of course the purpose of research should be ascertaining whether there is a true, reliable effect; but clearly our purpose was to find an effect, and in particular the effect we hypothesized. If we got it right in the current study and find our hypothesized effect, we will likely write up the two confirmatory studies, one assessing attitudes and the other behavior, and submit our paper for publication. The null effects of the intermediary experiment will not be discussed or mentioned, but will be ignored and forgotten. And null findings in this newest study wouldn’t necessarily preclude revising our protocol once again to “get it right.”