“Decisions from experience” = sampling error + prospect theory: Reconsidering Hertwig, Barron, Weber & Erev (2004)

Abstract

According to prospect theory, people overweight low probability events and underweight high probability events. Several recent papers (notably, Hertwig, Barron, Weber & Erev, 2004) have argued that although this pattern holds for “description-based” decisions, in which people are explicitly provided with probability distributions over potential outcomes, it is actually reversed in “experience-based” decisions, in which people must learn these distributions through sampling. We reanalyze the data of Hertwig et al. (2004) and present a replication to determine the extent to which their phenomenon can be attributed to sampling error (a statistical rather than psychological phenomenon) versus underestimation of rare events (i.e., judgmental bias) versus actual underweighting of judged probabilities. We find that the apparent reversal of prospect theory in decisions from experience can be attributed almost entirely to sampling error, and are consistent with prospect theory and the two-stage model of decision under uncertainty (Fox & Tversky, 1998).

Several recent papers have contrasted “decisions from description” with “decisions from experience”. In the former, people are explicitly provided probability distributions over potential outcomes; in the latter, probabilities and outcomes are not provided but must be learned by sampling from these distributions. Perhaps most notable is an experiment presented by Hertwig, Barron, Weber and Erev (2004, hereafter HBWE) that was also described in detail by Weber, Shafir and Blais (2004) and Hertwig, Barron, Weber and Erev (2005). In this experiment students either saw six decision problems (e.g., gain 4 points with probability .8; gain 0 otherwise; for a list of all lottery pairs see Table 1) or sampled outcomes (with replacement) from unlabeled buttons associated with these pairs of payoff distributions. After sampling draws from each button as many times as they wished, participants indicated which lottery they preferred to play once for real money. The authors characterize their results as follows: “In the case of decisions from description, people make choices as if they overweight the probability of rare events, as described by prospect theory....in the case of decisions from experience, in contrast, people make choices as if they underweight the probability of rare events.” They conclude their abstract with a “call for two different theories of risky choice.” Although we are sympathetic to the investigation of decisions from experience, we believe that the call for two different theories of risky choice is premature as HBWE’s results are driven almost entirely by sampling error and are consistent with prospect theory (hereafter PT, Kahneman & Tversky, 1979; Tversky & Kahneman, 1992) when it is applied to the probability distributions of outcomes that participants actually sampled.

Table 1 Percentage of choices consistent with prospect theory (PT) for each decision problem in Hertwig et al. (2004). H = option with the higher expected value; L = option with the lower expected value (the first number is the prize amount in points, the second number is the objective probability). Underlining indicates the choice predicted by PT, assuming objective probabilities and median value- and weighting-function parameters reported by Tversky & Kahneman (1992). Entries in the last three columns indicates percentages of responses compatible with PT for decisions from description, and for decisions from experience assuming objective probabilities and probabilities experienced by participants, respectively.

To motivate our approach we note that economists since Knight (1921) have distinguished decisions under risk, in which objective probabilities of outcomes are known by the decision maker, from decisions under uncertainty, in which they are not. The experiment of HBWE thus compared decisions under risk (where lotteries were explicitly described) to decisions under uncertainty (where lotteries were unlabeled buttons). In decisions under uncertainty people’s beliefs may differ from so-called “objective” probabilities for a variety of reasons. First, experienced probabilities may differ from objective probabilities—as HBWE note, it follows from the binomial distribution that people are more likely to under-sample than over-sample low-probability events, and this sampling error will be more pronounced the lower the probability and the smaller the size of the sample taken.¹ Second, people judge the probabilities that they have experienced incorrectly. Thus, the question arises to what extent HBWE’s finding of “underweighting” can be attributed to: (1) sampling error (the difference between so-called “objective” probabilities and probabilities of outcomes that participants actually experienced); (2) judgment error (the difference between experienced probabilities and judged probabilities); and (3) probability weighting (the difference between judged probabilities and their impact on choices). Breaking down the source of “underweighting” is critical to the interpretation of HBWE’s results because sampling error is a statistical rather than psychological phenomenon, judgment error is a bias in belief rather than preference, and the weighting of these judged probabilities provides the most apt comparison to prospect theory.²

If the predominant source of HBWE’s effect is sampling error, one would expect most choices to accord with prospect theory weights applied to the probabilities that participants actually observed. To test this notion we obtained HBWE’s raw data and tallied the participants in the experience condition (unlabeled buttons) whose choices were consistent with PT assuming “objective” probabilities (that were known only to the experimenter) versus “experienced” probabilities (proportions of events that participants observed).³ For each gamble pair we calculated the percentage of participants who chose the higher PT-valued gamble assuming median value- and weighting-function parameters reported by Tversky and Kahneman (1992). The results (see Table 1) are striking and consistent over all lottery pairs: although most participants “violate” PT when the analysis is applied to “objective” probabilities as presented in HBWE, most conform to the predictions of PT when applied to probabilities that participants experienced, at a similar rate to decisions from description.

In order to distinguish judgment error from probability weighting we replicated the method of HBWE, adding an elicitation of judged probabilities. We asked 46 students at Ben Gurion University to (1) sample outcomes from each pair of unlabeled lotteries, (2) choose their preferred lottery, and (3) recall the possible outcomes of each lottery and then estimate their respective probabilities, counterbalancing the order of judgment and choice tasks (a fuller account of this experiment will appear elsewhere; details can be obtained from the authors). The median correlation among respondents between judged and experienced probabilities was .97 and the median mean absolute error was .06, suggesting that participants were very accurate.⁴ When Tversky & Kahneman’s (1992) median value- and weighting-function parameters were applied to “objective” probabilities, most choices (60%) violated PT as in HBWE; however, when this analysis was applied to judged probabilities, most choices (63%) conformed to PT. Thus, participants apparently weighted judged probabilities in decisions from experience just as they tend to weight chance probabilities in decisions from description, consistent with Tversky & Fox’s (1995) “two-stage model” of decision under uncertainty (see also Fox & Tversky, 1998; Wakker, 2004). We expect that this model would have fit even better had predictions been based on individually measured PT parameters, which are known to vary somewhat between participants, rather than group medians (see Gonzalez & Wu, 1999).

We note that HBWE do find some interesting patterns in their data. Notably, participants with small incentives are content to make decisions based on small samples of information, and more recently sampled information may have greater impact on their decisions. However, our internal analysis of HBWE’s data and our replication show that the so-called “underweighting” of low probability events in decisions from experience is driven almost entirely by a tendency to undersample low-probability events—a statistical property of the binomial distribution—and has almost nothing to do with underestimation of observed probabilities or a tendency to underweight these probabilities. Likelihood judgments correspond very closely with experienced probabilities and choices are consistent with overweighting low probabilities, as characterized by prospect theory. Thus, the call for “two different theories of risky choice” seems premature, and future research on decisions from experience might instead explore models of search rules (what information do people seek), models that terminate search (how much information do they seek), and models of bias in likelihood judgment.