Measuring Evangelicals: Practical Considerations for Social Scientists–CORRIGENDUM

THE ERRORS IN ANALYSIS

In the process of replicating the findings of this article, Dr. Tobin Grant alerted us (2018) to several errors that were made in the calculations and interpretations in our article. Specifically, Dr. Grant was unable to replicate the percentage of evangelicals from the General Social Survey (GSS) presented in Figure 1 when using the new self-identification measure, as well as in some of the t-test results we show via comparison of confidence intervals in Figures 2 and 3. After some investigation, we confirm that there were coding errors that produced incorrect findings in our published article, as well as errors in interpretation. We would like to describe and correct those mistakes.

Figure 1. Percentage of evangelicals using both affiliation and self-identification measures.

Figure 2. Mean partisanship of evangelicals using both affiliation and self-identification measures.

Figure 3. Mean church attendance of evangelicals using both affiliation and self-identification measures.

CORRECTED ANALYSIS

Figure 1 reports the percentage of evangelicals using both affiliation and self-identification measures. The self-identification measure excludes African Americans from being classified as evangelical Protestants, as described in the body of the article. In calculating the proportion of individuals who were classified as evangelical using this approach for the samples in both the GSS and Cooperative Congressional Election Study data sets, we inadvertently filtered out African-Americans from the denominator. Thus, the denominator was too small and hence the estimate of evangelicals was too large. This alters the findings somewhat. The biggest difference is that for the GSS the number of self-identified evangelicals is now a smaller percentage. This difference is approximately 2–3% lower in the new approach than the flawed calculation. The numbers calculated using the affiliation approach have not changed as that code was correctly specified in the original version. The corrected graph is below.

This corrected graph also includes 84% confidence intervals. This is a visual change from the previous paper that included 95% confidence intervals throughout the paper. In practice, comparing two 84% confidence intervals is the statistical equivalent to a single 95% difference of means test. This change helps us correct an error in interpretation because in the prior version we calculated statistically significant differences based off of the comparison of two 95% confidence intervals, which was too strict of a test. It is not equivalent to use the overlap of two 95% confidence intervals to indicate a 95% difference of means test (Schenker and Gentleman 2001). While the calculated means for the two groups might have 95% confidence intervals overlap, that is only a necessary but not a sufficient condition to indicate statistical significance using a t-test. The assumption that overlapping confidence intervals indicates that the two samples are statistically indistinct resulted in the incorrect portrayal of results in Figures 1, 2, and 3.

For Figure 1, in the correction, we find that the differences in means between the two approaches are not statistically different for the 2008 and 2016 waves of the GSS. In the prior version, we indicated statistically insignificant differences in all five waves of the GSS.

In the other two visualizations, that portray the mean partisanship (Figure 2) and mean church attendance (Figure 3), the text indicated that in the instances where the confidence intervals overlapped that there was no statistical difference at the 0.05 threshold. That note is incorrect. Per Payton et al. (2003), the note should read that the differences are not significant at the p  =  0.01 level (2 × 95% confidence intervals is the equivalent of a 99% test). An updated Figure 2 is below with 84% confidence intervals, which provide a very close approximation to the more appropriate p < 0.05 level in a t-test. Note that in the corrected version the differences in mean partisanship are statistically indistinguishable for both the 2008 and 2010 waves of the GSS using the t-test, while the published article indicates that all five waves were statistically indistinguishable.

A revised Figure 3, which is the mean church attendance for each of the two measures, is also included below. Here the differences in means are statistically indistinguishable for GSS 2008, GSS 2012, GSS 2014, and GSS 2016. The only change here from the published version is that the GSS 2010 is statistically distinguishable at the 0.05 level.

Finally, we wanted to note two small errors in the manuscript. In the text, Figure 4 is described as a “logit model,” but it should be “gaussian model.” We also state that the Cooperative Congressional Election Study began in 2008, but the inaugural year was 2006.

STATISTICAL SIGNIFICANCE VERSUS SUBSTANTIVE DIFFERENCE

Though this update produces some statistically significant differences in approaches to measuring evangelicals, the differences are substantively small. For instance, the difference in party identification for each of the two measures in the GSS in 2016 is 0.26. That is one-quarter of one point on a scale that runs from 1 (Strong Democrat) to 7 (Strong Republican)—those who self-identify are 4% more Republican than those who have an evangelical affiliation. For Figure 3, the differences in using these approaches for the GSS is never greater than 0.2 on a scale from that runs from 0 to 6. In two cases the difference in means is less than 0.1.

While oftentimes t-tests are used to indicate whether the means from two samples are statistically distinct, this statistical test results in a reductive outcome: the means are different or they are not. In our case, using the two different measurement techniques resulted in some instances where the means of the two approaches were statistically different from each other. So, while the update is an important corrective, we believe that the small substantive differences between the self-identification and affiliation approaches allow for the central theme of our paper to persist.

IMPLICATIONS FOR UNDERSTANDING EVANGELICALS

These corrections do soften the overall conclusions of the paper. For instance, we wrote in the abstract, “We find almost no statistical differences between the two measurements in prominent demographic, political, or religious factors.” Instead, a more accurate description would be that we do find some small statistical differences between the two measures (primarily in the area of political ideology). Additionally, the new self-identification measure provides a smaller sample size of evangelicals (on average) than the previously utilized affiliation approach. This indicates that new self-identification measures are not the functional equivalent of the affiliation measure, though a close approximation on prominent measures of political and religious behavior.

These errors in computation temper our findings. The self-identification approach generates an evangelical sample that is somewhat smaller, and slightly more Republican than the previously employed affiliation strategy. Nevertheless, scholars should understand there are some tradeoffs based on measurement strategy. We believe we have provided a resource in this article and errata letter to understand this.

While we wish that we had avoided these mistakes in the first place, we are grateful for colleagues and editors for identifying the mistakes and allowing us to publicly correct the record. It is our hope that subsequent researchers will use this contribution as a means to understand how to wisely approach measuring evangelicals, the comparison between methods, as well as the tradeoffs when using the self-identification strategy versus the affiliation approach in public opinion research.