Our paper (Krupnikov and Levine, Reference Krupnikov and Levine2014) includes 27 theoretically-informed within-sample comparisons and 27 cross-sample comparisons in Tables 3–6. As part of an independent replication effort, Alex Coppock discovered an error in the program we used to conduct the cross-sample comparisons. In this document we report updated results of each of these 27 comparisons. For completeness, we also include the within-sample results, as they were also part of the independent replication effort (these estimates produced identical substantive results in all cases except one).
This document provides revised versions of Tables 3–6 in the original paper. Comparisons in bold signal those in which the substantive interpretation of the comparison is now different.
The main substantive conclusions that refer to within-sample comparisons from the article still stand. In particular, it is still the case that the “YouGov samples consistently produced results that were in line with theoretical expectations, especially once we accounted for relevant moderators.” (77). This was also the case for our student sample, with only one exception across all four sets of experiments. It is also the case that “the MTurk results look different, at least for the experiments that required a bit more “buy-in” from our subjects (77).” That is, in both sets of experiments that test savviness and also in the experiment that required subjects to read an article we still do not see evidence that low experience MTurkers responded as we would theoretically expect. Moreover, and again against theoretical expectations, we do see differences among how Independent and Republican-identifying MTurkers responded to the news article.
In terms of the cross-sample differences, as before it is still the case that “there were various points where the effect sizes differed.” Our conclusion that “undergraduate samples can produce results that differ from those with adult samples” and that “convenience sample adults may also not produce replicable results” (77) still stands. That said, we have less evidence than in the original paper that our estimates across samples are statistically distinct from one another.
We are grateful to Alex Coppock for identifying this issue, and his reconsideration of our data underscores the importance of public data-sharing.
Revised Table 3 (Studies 1, 2, 3)
Revised Table 4 (Studies 4, 5, 6)
Revised Table 5 (Studies 7, 8, 9)
Revised Table 6 (Studies 10, 11, 12)
