Herbert Simon’s spell on judgment and decision making

Abstract

How many judgment and decision making (JDM) researchers have not claimed to be building on Herbert Simon’s work? We identify two of Simon’s goals for JDM research: He sought to understand people’s decision processes—the descriptive goal—and studied whether the same processes lead to good decisions—the prescriptive goal. To investigate how recent JDM research relates to these goals, we analyzed the articles published in the Journal of Behavioral Decision Making and in Judgment and Decision Making from 2006 to 2010. Out of 377 articles, 91 cite Simon or we judged them as directly relating to his goals. We asked whether these articles are integrative, in the following sense: For a descriptive article we asked if it contributes to building a theory that reconciles different conceptualizations of cognition such as neural networks and heuristics. For a prescriptive article we asked if it contributes to building a method that combines ideas of other methods such as heuristics and optimization models. Based on our subjective judgments we found that the proportion of integrative articles was 67% of the prescriptive and 52% of the descriptive articles. We offer suggestions for achieving more integration of JDM theories. The article concludes with the thesis that although JDM researchers work under Simon’s spell, no one really knows what that spell is.

1 Introduction: Simon’s spell

After receiving his Nobel prize in economics, Daniel Kahneman wrote that he and Amos Tversky “explored a territory Herbert Simon had defined and named—the psychology of bounded rationality” (2003, p. 697). Gerd Gigerenzer and another Nobel laureate in economics, Reinhard Selten, claimed that by bounded rationality Simon did not mean irrationality as in Tversky and Kahneman’s heuristics-and-biases program or Thaler’s behavioral economics but instead “…models…[that] use fast and frugal stopping rules for search” (2001, pp. 5–6).

Some judgment and decision making (JDM) researchers side with one of these two seemingly contradictory interpretations of bounded rationality, others attempt to show that they are not really contradictory, and yet others suggest moving beyond trying to define bounded rationality. One way or another, we researchers in JDM work under Simon’s spell.

This article aims at initiating an informed and constructive discussion about the relation between Simon’s work and JDM research and how it can be used to push JDM research forward. In the next section we analyze writings of Simon and other researchers about his work and identify two of his main goals for JDM. In Section 3, in order to investigate how recent JDM research relates to these two goals, we analyze the 377 articles published in the Journal of Behavioral Decision Making and Judgment and Decision Making from 2006 to 2010. We argue that Simon’s goals are not pursued in an integrated way and speculate on why is this the case and how we can achieve more integration of JDM theories. In Section 4 we return to Simon’s spell and claim that JDM researchers do not really know what it is.

2 Simon’s two research goals for JDM

Simon was so prolific that many disciplines, from political science and psychology to operations research and computer science, claimed him as their own. He, however, saw himself as an interdisciplinary researcher, focused on understanding human behavior and in particular decision making, who “rapidly became, and remained, obsessed with human decision making…” (Simon, 2001, p. 501).

Simon popularized the concept of bounded rationality with two articles in flagship journals, in economics in 1955 and psychology in 1956. He received the economics Nobel prize in 1978 and in the abstract of his obituary, his longstanding colleague James March noted: “In particular, he persistently sought to clarify the real processes of human decision making, thus was pre-eminently a behavioral economist” (Augier & March, 2002, p. 1).

Real processes mean cognitive processes in practice, not in neoclassical economic theories where decision makers have access to all information as well as unlimited time and computational ability. Simon saw cognitive processes such as recognition and search as central to decision making. He studied such processes for more than six decades—in 1939 at the age of 23 he directed a study of the administration of state relief programs and had the opportunity to observe human behavior as it really occurred. Describing decision processes was Simon’s first research goal for JDM.

A second JDM research goal of Simon, prescribing how to make good decisions, was one he pursued as an engineer. It is summarized in his 1968 book The Sciences of the Artificial which is based on lectures that Simon delivered at the Massachusetts Institute of Technology. This book makes clear that Simon was strongly interested in principles and algorithms for making good decisions. For example, propounding the principle that the structure of information in the world is often hierarchical, he argued that simple local algorithms can work well enough.

Simon’s two goals are shared in some sense by virtually all JDM researchers. On its home page, the Society for JDM (SJDM) says that it is “dedicated to the study of normative, descriptive, and prescriptive theories of decision”. In the December 2006 newsletter of the Society, Jonathan Baron discusses the history of the terms descriptive and prescriptive decision theory which seem to go back to at least the mid-1980s. Did Simon then truly make a unique contribution when proposing these two goals?

We think so. As Baron mentions in the 2006 SJDM newsletter, “…Simon did not clearly distinguish prescriptive from descriptive...” (p. 6). That is, Simon sought to understand the real decision processes and he studied whether the same processes lead to good decisions. Whereas some JDM researchers sometimes seem to view or have viewed heuristics, such as Tversky’s elimination-by-aspects, as a bad habit that needs to be corrected,¹ Simon did not. In his 1955 paper, for instance, he proposed a sequential, aspiration-level-based model for selling a house. In The Sciences of the Artificial he wrote: “the presence of uncertainty…places a premium on robust adaptive procedures instead of optimization strategies that work well only when finely tuned to precisely known environments” (p. 35).

Basing prescriptive decision theory on the heuristics people naturally use for making decisions is still a relatively marginal idea but also one that has led to a number of good results (for a review see Katsikopoulos, 2011) and it is an idea that came from Simon. Of course, other key JDM figures such as Ward Edwards and Robyn Dawes also advocated that a bit of expert knowledge can be used to support decision making. But Herbert Simon reserved a big role for the processes people use. What about the bulk of JDM today?

3 Recent JDM research: How does it relate to Simon’s goals?

3.1 Simon-related articles

We analyzed the 377 articles published in the Journal of Behavioral Decision Making and Judgment and Decision Making from 2006 to 2010. At first glance, most of these articles relate to bounded rationality in the general terms that Simon described it: “…theories that incorporate constraints on the information processing capacities of the actor may be called theories of bounded rationality” (1972, p. 162). We used stricter criteria for judging an article to be Simon-related or not. First, we looked at citations and found 44 articles that cite Simon. Second, we judged 47 additional articles as relating directly to one or both of Simon’s goals even without citing him.

For examples of descriptive articles consider two special issues in Judgment and Decision Making edited by Newell and Broeder (2008) and Marewski, Pohl and Vitouch (2010). The two issues have the same structure: Models from Gigerenzer, Todd and the ABC research group’s (1999) adaptive toolbox, such as take-the-best and the recognition heuristic, are compared in terms of their descriptive adequacy to more mainstream psychological models such as evidence accumulation models and exemplar models.

As an example of a prescriptive article consider Serwe and Frings (2006). In order to predict the winner of a match at the Wimbledon tennis tournament, they proposed the following: “Pick the player who is recognized by more laypeople”. This fits with Simon’s approach of basing prescription on people’s heuristics given that people, under some conditions, use the recognition heuristic (Hilbig, 2010, among others, argues that these conditions are narrow). For more examples see the special issue in honor of Paul Meehl by Morera and Dawes (2006) in the Journal of Behavioral Decision Making. The articles in this issue study the accuracy of linear models with regression or unit weights which have also been proposed as descriptive models.

The 91 articles are cited in the reference list and are also provided in the two tables below where Table 1 summarizes the results for the Journal of Behavioral Decision Making (45 articles) and Table 2 for Judgment and Decision Making (46 articles).

Table 1: The 45 articles published in the Journal of Behavioral Decision Making from 2006 to 2010 that cite Simon or that we judged as relating to his prescriptive or descriptive goal. For each article, we judged whether it relates to Simon’s prescriptive goal (1) or not (0) and whether it is integrative (1*) (see text for definitions). The same coding was used for our judgments of relation to Simon’s descriptive goal. We also judged whether articles targeted the topics of formal modeling, experimental methods of process tracing, the distinction between intuition and deliberation and the distinction between maximizing and satisficing (1 for yes and 0 for no). The main results are: (a) 6 out of 9 prescriptive articles are judged to be integrative, (b) 17 out of 40 descriptive articles are judged to be integrative and (c) the most popular topic is by far formal modeling (21 articles).

Table 2: The 46 articles published in Judgment and Decision Making from 2006 to 2010 that cite Simon or that we judged as relating to his prescriptive or descriptive goal. For the coding used for our judgments see the caption of Table 1. The main results are: (a) 12 out of 18 prescriptive articles are judged to be integrative, (b) 20 out of 31 descriptive articles are judged to be integrative and (c) the most popular topic is by far formal modeling (28 articles).

For each article, we judged whether it relates to Simon’s prescriptive and descriptive goals (marked by 1) or not (marked by 0). Simon pursued his descriptive and prescriptive goals by strongly relying on theory and our main interest was to judge whether Simon-related JDM articles are theoretically integrative (marked by 1*). The integration of theories has been argued to be “psychology’s most important task” (Gigerenzer, 2010, p. 733; for an example of what he means see Luan, Schooler and Gigerenzer, 2011) and there is a history of complaints that it has not been achieved (Watkins, 1984; Mischel, 2009). These authors have made the point for descriptive theories and we make it for prescriptive theories as well. We are interested in the integration of theories only, not in the integration of other parts of research such as experimental methods for testing theories or analyzing data. To ensure that our judgments were consistent, we specified a number of criteria for integrative articles:

For a descriptive article to be judged as integrative, we requested that it contributes to building a theory that reconciles different conceptualizations of cognition such as neural networks, evidence accumulation models and heuristics. As an example consider Newell and Broeder (2008) who used a number of concepts from cognitive psychology to place a number of formal JDM models in the same theoretical framework (Figure 1, p. 200). In our definition of integrative descriptive articles we use a somewhat strict meaning of reconciling models. For instance, we did not judge as integrative those studies that compared the descriptive adequacy of competing cognitive models but did not create a super model that incorporated concepts from the various models. Of course, an article was not judged as integrative if it studied just one descriptive model.

For a prescriptive article to be judged as integrative, we requested that it contributes to building a method that combines elements of other methods such as heuristics and optimization models or incorporates different yardsticks of rationality such as Hammond’s (2007) coherence and correspondence. For examples of integration see the Judgment and Decision Making special issue edited by Dunwoody (2009), as in Tape (2009) who argues that “using the best of both approaches [coherence and correspondence] could…lead to striking advances in medical science” (p. 134). As with descriptive articles we use a somewhat strict meaning of combining methods. For instance, we did not judge as integrative those studies that compared the performance of competing methods but did not create a hybrid method that included ideas from the various methods. Of course, an article was not judged as integrative if it studied just one method for making decisions.

In Section 3.2. we discuss whether the 91 articles in Tables 1 and 2 were found to be integrative. Before that we organize the content of the articles. The vast majority of the articles target one or more of the following topics: formal models of decision making, experimental methods of process tracing, the distinction between intuition and deliberation and the distinction between maximizing and satisficing. Formal modeling is by far the most popular topic, targeted by 49 articles (9 articles target process tracing and the intuition/deliberation distinction and 4 target the maximizing/satisficing distinction).

Simon himself worked on all of these topics. For example, he was skilled in mathematics and computing and wrote extensively on formal economic models of decision making such as utility theory. Simon also worked on experimental methods such as protocol analysis (Ericsson & Simon, 1993). And he was preoccupied with the relative merits of a theoretically optimal solution versus a practically good enough solution as can be seen in The Sciences of the Artificial, especially Chapter 5. Finally, Simon touched on the now popular distinction between intuition and deliberation, stating, for example: “we cannot, of course, rule out the possibility that the unconscious is a better decision-maker than the conscious” (1955, p. 104).²

3.2 Are the Simon-related articles integrative?

We emphasize that our judgments are subjective and should not be taken at face value but rather be used in order to stimulate further thinking. As seen in Tables 1 and 2, the results are: (i) The number of descriptive articles is much higher than the number of prescriptive articles (71 vs. 27) and (ii) the proportion of integrative articles is higher for prescriptive than for descriptive articles but not much higher than half of the articles (67% and 52% respectively).³

It seems that a number of articles focuses on producing novel effects, not on developing theory. This is consistent with complaints about a general lack of theory in social (Fiedler, 1991) and cognitive psychology (Gigerenzer, 1998). According to our definitions, such articles cannot be integrative. We next provide more subtle examples of the lack of theoretical integration.⁴

Serwe and Frings (2006) came up with the prescriptive rule “Pick the player who is recognized by more laypeople” for predicting the outcomes of tennis matches and showed that it matched the accuracy of official rankings. But they did not go a step further and investigate how the rule could be combined with other methods. Yet this step is necessary because we know that recognition-based rules do not always perform well (Andersson & Rakow, 2007). For example, it would be worthwhile to evaluate the performance of a super method that uses recognition-based rules in the first rounds of a tennis tournament and switches to the official rankings in latter rounds. In general, no method performs well for all decision problems in the real world and it is important to think about how to construct hybrids that capitalize on diverse methods such as heuristics and optimization (for a family of such hybrids see Katsikopoulos, 2011, Figure 2, p. 25).

As an example of lack of integration in descriptive articles, consider the numerous debates on the descriptive adequacy of the adaptive toolbox and its alternatives. A sign of segregation in these debates is that researchers have sometimes been divided into “advocates” and “critics” or “proponents” and “critics” of various models (as in Marewski et al., 2010). Such divisions are not helpful. In fact they are harmful in the sense that importance is placed on who said what instead of what has been said. An extreme version of the problem is focusing on who said what when (more recently, Marewski, Pohl & Vitouch, 2011, p. 371 have also discussed this problem). For example, one critique of the current theory of the recognition heuristic is that it is inconsistent with its previous versions. Leaving the validity of this critique aside, we cannot see why researchers should not try to improve their understanding of a topic in the course of time. As Herbert Simon is anecdotally quoted to have said, articles are often progress reports, not the final word. An integrative approach to the descriptive theory of JDM should focus on the current shared understanding of a topic and on improving it in the future, not on paying a lot of attention to who was right or wrong in the past.

The last example refers not to a particular article but to an exchange in the Journal of Behavioral Decision Making. After Gloeckner, Betsch, and Schindler (2010) argued that it is not precisely specified how people select a heuristic from the adaptive toolbox, Marewski (2010) responded that it is Gloeckner et al.’s (2010) own parallel constraint satisfaction (PCS) model that cannot solve this selection problem, to which Gloeckner and Betsch (2010) replied that PCS does not need to solve the selection problem. In this discussion the researchers’ different conceptualizations of cognition are not reconciled. There is little progress on what kind of concepts in general would allow models to solve the selection problem, only arguments why their model can do so or the others’ model cannot.

In the next section, we speculate on the reasons for the lack of integration.

3.3 Why Is there a lack of integration and what can we do about it

We first consider descriptive Simon-related articles. We begin with an observation by Gigerenzer (2010). He pointed out that in psychology “We teach our students how to test theories, not the art of theory construction in the first place” (2010, p. 733, emphasis added). The same can be said for JDM. It seems to us that the level of effort in analyzing data is not matched by the level of effort expended in thinking about the primitive concepts of theories. Let us explain this with some examples.

Reisen, Hoffrage and Mast (2008) and Gloeckner (2009) developed methods for uncovering people’s decision models that jointly use measures such as choices, reaction times, eye movements, verbal protocols and information search in a Mouselab environment.

These are serious efforts to solve a difficult problem. Contrast it with work on modeling, say, the recognition heuristic. It is clear that such models need to be grounded on memory research (Dougherty, Franco-Watkins & Thomas, 2008; Erdfelder, Küpper-Tetzel & Mattern, 2011). The discussion on how to do this revolves around questions such as how to model recognition in a way that is not all-or-none (as it was done originally).

Such questions are important. But it is not what we mean by thinking hard about primitive concepts. What we have in mind are questions such as “When should JDM models use variables that are observable (e.g., yes/no judgments of recognition) or latent (e.g., strength of memory trace)?” There are no obvious answers as the following disagreement illustrates. In Luce’s view (1997, p. 83), “latent, not directly observable structures … afford too many options, leading to badly underdetermined models with many free parameters”. On the other hand, Newell, Lagnado and Shanks (2007) are happy with the move in the Rescorla-and-Wagner learning model, from predicting “response probabilities” to “internal association strengths or weights,” because “it is a person’s belief about an association between an action and an outcome that we want to understand, rather than the superficial manifestation of that belief” (p. 155). Both views have merit but are in contrast and have to be reconciled. In our observation, little effort has been made towards such integration.

Based on the above, we conclude that in JDM we do not always think hard about the concepts on which our descriptive models are based. But without such thinking, theory integration is bound to be incomplete.

Next we consider prescriptive Simon-related articles. As with descriptive articles, the lack of integration likely has to do with research tradition: A single view of prescriptive theory, the one based on Bayesian inference and expected utility theory, has dominated (Katsikopoulos & Fasolo, 2006). This can to a large extent be attributed to Ward Edwards, arguably the father of JDM research (Heukelom, 2010).

For example, in a panel discussion on what constitutes a good decision at the 1983 Subjective Probability and Utility in Decision Making conference in Groningen, Edwards announced that “no principle other than maximizing SEU [subjective expected utility] deserves a moment of consideration” (Vlek, 1984, p. 7). This conviction is apparent throughout Edwards’ career and writings (Weiss & Weiss, 2009; Katsikopoulos, 2010). Findings that question the dominance of the prescriptive theory of Bayesian inference and expected utility theory, such as those of Payne, Bettman and Johnson (1993) and Gigerenzer et al. (1999) are often dismissed (Hogarth, 2011) and have only recently begun to be viewed as serious challenges.

What can we do to promote theoretical integration in JDM research? As a starting point, we can use Gigerenzer’s (2010) suggestions of encouraging submissions of integrative articles in leading journals and teaching theory construction in graduate school. In our view, a precondition for constructing good theories is precise conceptual thinking. For instance, a textbook that cultivates this kind of thinking and illustrates it with examples from JDM research is Cognitive Modeling by Busemeyer and Diederich (2010). Furthermore, the pursuit of integration with disciplines outside psychology can be fostered by recalling Simon’s commitment to particular research problems in JDM, not blindly to a single discipline that is part of JDM. He found a nice way of saying that it is necessary to consider how to integrate one’s point of view with that of other disciplines: “If you see any of these disciplines dominating you, you join the opposition and fight it for a while” (quoted by Augier & March, 2002, p. 2).

4 What exactly is Simon’s spell? A critique

There are not many critiques of Herbert Simon’s idea of bounded rationality, at least not in the JDM literature. It is not difficult, however, to find weaknesses in this idea: If we look beyond the seductive image of Simon’s grand project of finding out how mortals really reason, we notice that the idea of bounded rationality needs more work in order to be theoretically insightful and practically relevant: Simon had a tremendous influence and he essentially did so with a single but powerful notion—that in practice decision makers need to prune the full decision tree (Moses, 2004, p. 4). What he did not tell us, however, is what are the principles that guide the pruning, how do they change from one task to the next and what are the reasons for these changes.

In sum, Simon’s idea needs to be fleshed out with more empirical evidence and theoretical modeling. This current vagueness of bounded rationality may be the reason why we in JDM labor under Herbert Simon’s spell but do not really know what the spell is. Coming to know it is a great task for the future.