The available evidence suggests the percent measure should not be used to study inequality: Reply to Norton and Ariely

Abstract

In this reply, we reiterate the main point of our 2012 paper, which was that the measure of inequality used by Norton and Ariely (2011) was too difficult for it to yield meaningful results. We describe additional evidence for this conclusion, and we also challenge the conclusion that political differences in perceived and desired inequality are small.

We are pleased that Norton and Ariely have written a comment on our paper (Eriksson and Simpson, 2012). Their comment makes two related points: that there is more agreement than disagreement between our papers, and that a fruitful direction for future research is to understand why the different measures we used generate different results. We believe that both points need qualification in the light of our paper’s main arguments and findings (including some additional findings that we did not report).

To summarize, we argued that Norton and Ariely’s (2011) findings are limited because they are derived from a method—measuring perceptions of wealth inequality in terms of percentages of quintiles—that is so conceptually and computationally demanding that it does not tap into people’s actual perceptions of inequality. To support our claims, we introduced a logically equivalent but less demanding measure that asked respondents about the average wealth in each quintile. What we found:

• The two measures gave radically different results: whereas respondents’ estimates of wealth inequality using the percent measure were highly inaccurate, the same respondents made far more accurate estimates of inequality using the average measure.

To further illustrate that the percent measure does not track respondents’ actual perceptions, we assessed perceptions of inequality in other domains as well. One domain was popularity of web pages, where common sense says that the most popular web pages, such as Facebook and Google, receive many orders of magnitude more clicks per day than the least popular web pages (e.g., our personal web pages). For another domain, school teachers’ salaries, common sense says that inequality is much more modest, because not even the highest paid teachers are very highly paid. What we found:

• The average measure yielded inequality perceptions that qualitatively corresponded to the high or low levels that actually exist in a given domain while the percent measure yielded low levels of perceived inequality regardless of domain (see Table 1, Eriksson and Simpson, 2012).

Why does the percent measure fail to track perceptions of inequality? In order to transform perceptions into a response to the percent measure, a respondent must first estimate averages in each quintile and then calculate percentages. (We encourage readers to try to answer the measure for themselves.) The fact that most of of our respondents gave completely inconsistent responses to the two measures suggested that they did not attempt to calculate the percentages or could not do so.

In light of the above, we do not believe the two points of Norton and Ariely’s comment are justified. First, consider the extent to which the measures agree. As we noted repeatedly in our paper (see pp. 742, 744), both measures showed that respondents want less inequality than they believe exists. But it is worth reiterating that the average measure did not lead to anything near the egalitarian preferences elicited by the percent measure.¹ To underscore how the measures lead to very different conclusions, consider findings on political differences from our second study, not included in our paper. For the percent measure, the median ratio of wealth that respondents thought should be owned by the poorest versus the richest was 1:3 among liberals and 1:5 among conservatives. These results suggest not only that respondents are incredibly egalitarian, but also that there is a high level of consensus between liberals and conservatives in ideal inequality. But we argue that these findings reflect less a consensus across the political spectrum than a common inability to answer the percent question. The average measure showed much less egalitarianism and much more dissensus between liberals and conservatives, who gave median ratios of 1:50 and 1:200, respectively.² Further, the two measures generated an even larger difference in perceptions of inequality, the domain on which our paper was mainly focused. While the percent measure yielded estimates that the wealthiest Americans are only about 20 times richer than the poorest, the average measure yielded a ratio of 1:1,500, as noted earlier.³

The second issue is how future research should be guided by what we have learned thus far. Norton and Ariely suggest some reasons that the two measures yield different results and that future work might address. However, their suggestions do not fit the data we presented in our paper.⁴ Norton and Ariely do not mention the explanation given in our paper: the percent question is simply too demanding for respondents to answer. Because our original paper provided only indirect evidence that participants did not or could not do these computations we conducted a new study to provide more direct evidence for this claim. Specifically, we gave 100 users of Amazon’s Mechanical Turk the following information about the distribution of wealth in a country:

The richest 20% of households have an average wealth of 1,000,000 dollars per household. The 2nd richest 20%: 200,000 dollars per household. The middle 20%: 50,000 dollars per household. The 2nd poorest 20%: 10,000 dollars per household. The poorest 20%: 1,000 dollars per household.

We then asked respondents whether, given this information, it is possible to tell what percent of the country’s total wealth each quintile controls. We asked those who stated that they could do the calculations to do so. We found:

• Of the 100 respondents, 37 thought the percentages could not be derived from the information given; 48 thought the percentages could be derived but said they didn’t know how to do it; 15 thought they knew how to do it, but 6 of them did it incorrectly, typically saying that the percentages were 20, 20, 20, 20, 20.

These results strongly suggest people are generally unable to transform perceptions of typical households’ wealth into percentages, as we suggested.

We sum up by noting that the original Norton and Ariely (2011) paper has received a very high level of mostly uncritical attention, probably because of the importance of the topic, the surprising findings, and the fact that both authors are well-known for doing excellent research. In this case, however, the available evidence suggests that their original conclusions are artifacts of an invalid measure. Our original paper was motivated by our concerns that policy recommendations might be founded on unsupported conclusions and that the paper might set a precedent for future research to employ similar invalid measures of perceived and desired inequality.