The role of process data in the development and testing of process models of judgment and decision making

Abstract

The aim of this article is to evaluate the contribution of process tracing data to the development and testing of models of judgment and decision making (JDM). We draw on our experience of editing the “Handbook of process tracing methods for decision research” recently published in the SJDM series. After a brief introduction we first describe classic process tracing methods (thinking aloud, Mouselab, eye-tracking). Then we present a series of examples of how each of these techniques has made important contributions to the development and testing of process models of JDM. We discuss the issue of large data volumes resulting from process tracing and remedies for handling those. Finally, we argue for the importance of formulating process hypotheses and opt for a multi-method approach that focuses on the cross-validation of findings.

1 Introduction

Theories of judgment and decision making (JDM) can be classified into two general types: formal, or as-if, models, which specify relationships between input task and context parameters and output JDM behavior; and process models, which in addition seek to model explanatory psychological mechanisms underlying such input-output relationships. Within the formal modelling tradition theories are evaluated via analysis of their predictions concerning outcome judgments and decisions, and subsequent rigorous experimental tests of such predictions. Alternative models are evaluated in terms of the testable predictions that distinguish them (for an exemplary illustration of this research strategy see Birnbaum, 2008). Process models, on the other hand, can be tested and evaluated in terms of both JDM behavior and process tracing methods, which elicit and analyze observations of a range of verbal and nonverbal antecedents and concomitants of judgments and decisions.

Imagine you conduct a risky decision making experiment in which you observe choices and collect verbal protocols. You find that the choices conform to, say, prospect theory (Kahneman & Tversky, 1979), but the verbal protocol has only infrequent uses of what you coded as evidence for prospect theory (e.g., reference point setting; coding as gains or losses; probability). What would you conclude: that decision making conforms to prospect theory (according to the choices); or that it fails to do so (according to the verbal protocols)? We surmise that the protocols would be seen as a subordinate, a supplementary source of data in this case. They would tend to be dismissed, if inconsistent with the choice output. In other words, we would hardly be likely to reject prospect theory on the basis of verbal protocols, or any other process data. Process data somehow seem to be a subordinate source of evidence. For models that aim at predicting outcomes (as-if-models) this is appropriate, but not for models that aim to explain both outcome and process (process models). The priority of output data is based on a natural sequence of testing dependent variables: predicting choice data is a first criterion for any model, be it a process model or an as-if-model. But beyond this first step there is no reason to prefer outcome over process. Rather, for process models process data should be equally important, because they are richer than input-output data and can provide important evidence of explanatory mechanisms (see also Rubinstein, 2003; Manski, 2004). An instructive example is the work of Glöckner and Herbold (2011), who show that, although prospect theory is certainly a good as-if-model, its process assumptions have to be rejected in favor of alternative models.

In this paper we describe methods devised in the field of judgment and decision making (JDM) to enable the recording of traces of underlying processes and cognitive representations. We will describe these methods on three dimensions: their theoretical contribution, their core methodology and their key results. For a more comprehensive overview we refer the interested reader to a recently published handbook on process methods (Schulte-Mecklenbeck, Kühberger, & Ranyard, 2011).

2 Prominent process tracing methods

We first give an overview of prominent process tracing methods, organized on a set of critical dimensions, as outlined in Table 1. Table 1 highlights features of models based on process tracing data, and process tracing technology as a whole. The features are briefly described below.

Table 1: Tools x dimensions matrix of important process tracing methods.

Availability of tool. This refers to whether the tools are freely available or have to be bought from a commercial source.

Location of tool use. Whether the data need to be collected in a laboratory, or alternatives are possible, e.g., the Web (e.g., MouselabWeb).

Number of participants tested concurrently. This feature refers to the extent to which multiple participants can be run concurrently.

Flexibility of tool. Some tools need extensive pre-structuring of tasks with limited content (e.g., Mouselab), while others need hardly any structuring and allow for variable content (e.g., verbal protocols).

Contribution to model building. Tools differ in their contribution to the development of specific process models. Some tools are intimately connected to a particular theory or model (e.g., active information search and its link to risk defusing operators, see Huber, Huber, & Schulte-Mecklenbeck, 2011), while others do not have such strong links (e.g., eye-tracking).

Table 1 could include further dimensions on which process tracing tools for JDM research might differ (e.g., the produced data volume, or the availability of procedures for analysis). However, the aim of Table 1 is to highlight that different features of tools can have a direct impact on the way they are used in the model building and testing process, rather than to provide a comprehensive overview. To this end we do not deal in detail with tools such as response time, skin conductance or the large area of neural methods, but refer the to Schulte-Mecklenbeck, Kühberger, and Ranyard (2011) instead.

3 Examples of important contributions to the development and testing of process models of JDM

In what follows we sketch out how the tools listed above are used (method), and give examples of their theoretical contribution and results of their application.

3.1 Active information search

Method

Active Information Search (AIS) is normally based on verbal descriptions of decision problems. Participants are given only the most basic description of the situation, and have to ask questions in order to receive additional information. This technique enables the observation of the participant’s information needs. In the standard method the type, frequency, and sequence of the collected information is recorded. Computerized versions of AIS can also record reading time (see Huber, et al., 2011 for details).

Theoretical contributions and results

A central idea of Active Information Search (AIS, Huber, Wider, & Huber, 1997) is that the structuring of the task is an integral part of the decision process. This is overlooked if participants are presented with a pre-structured, completely formulated problem (e.g., an alternative x dimensions matrix). One of the main findings of AIS studies is that the role of probability information is overweighed in traditional lottery tasks: if people can decide which information to ask for, they will construct their problem definition freely through the question-answering process and, most importantly, they often do not ask for information about probabilities. Rather they are seeking control over the outcomes. This phenomenon is called risk defusing (Huber, Beutter, Montoya, & Huber, 2001).

3.2 Eye movements

Method

Recording corneal reflection (video based eye-tracking) is the most commonly used approach (Duchowski, 2002). Due to the high sampling frequencies that are currently available (often over 1000Hz), the detailed registration and analysis of rapid micromovements (i.e., saccades), as well as fixations (i.e., resting of the gaze on a single location) are possible. To infer cognitive processes the tempo, amplitude, duration, or latency of such saccadic movements, and the duration, frequency, and scanning path of fixations are of central interest.

Theoretical contributions and results

Eye movements offer evidence of attention processes, as well as information acquisition. There is a long history of studies in psychology linked to this method (see Rayner, 1998). In JDM research, eye-tracking has up to now mainly been used to investigate information acquisition. Currently, we witness an innovative use of this technology for distinguishing different modes of thought (intuitive and deliberative) using process data (Glöckner & Witterman, 2010; Horstmann, Ahlgrimm, & Glöckner, 2009). A good example of using looking patterns to differentiate modes of thought can be found in the work of Horstmann et al. (2009). Fixation duration is the main dependent measure when a more deliberative condition is compared to a more intuitive one. The authors found no duration differences for instructed modes of thought, but induced deliberation led to a higher number of fixations, and to more complete and repeated information acquisitions. This attests to the utility of eye movement recording for distinguishing modes of thought.

3.3 Mouselab, MouselabWeb and information boards

Method

Payne (1976) pioneered the development of this technique (actually in combination with thinking aloud) which provides data concerning the content, amount, and sequence of the information acquired. Participants search for information, for instance, by opening envelopes that contain cards with text on them (information boards), or open cells on a matrix displayed on a computer screen. Well known tools in this category are the Mouselab system (Bettman, Johnson, & Payne, 1990; Payne, Bettman, & Johnson, 1993), and MouselabWeb (Willemsen & Johnson, 2011) which conveniently add time measurements to the above mentioned variables.

Theoretical contributions and results

The Priority Heuristic (PH, Brandstätter, Gigerenzer, & Hertwig, 2006) is an instructive example in this category because this choice model includes an explicit description of process steps. All steps predicted by the model can be easily tested with, e.g., MouselabWeb. Johnson, Schulte-Mecklenbeck and Willemsen (2008) translated the steps suggested in the PH model into a production system-like list of process steps (Read, Calculate, …). Based on the different process steps and the detailed descriptions in the model, hypotheses can be formulated, e.g., the minimum outcome should be inspected more frequently, and longer, than the maximum outcome. Through this process approach Johnson et al., were able to demonstrate that the collected process data were largely inconsistent with the PH predictions.

3.4 Mouse tracking—response dynamics

Method

Freeman and Ambady (2010) developed a software package called Mousetracker which provides a low hurdle entrance into the area of response dynamics recording. The program tracks trajectories of mouse movements in choice situations. MouseTracker comes with a setup tool for different experimental designs, a data recording program, and a package for analyzing and exporting collected data. This is worth mentioning, because there is a lack of such a complete, freely available, package in many process tracing applications. Mouse trajectories are recorded in sufficient detail to allow for analysis of decision times, trajectories (with maximum deviation measures) of the mouse and areas under the curve calculation for later comparisons between conditions.

Theoretical contributions and results

The ability to assess temporal dynamics of mental processes is the key benefit of collecting mousetracking data. This technique was pioneered by Spivey (2005; 2007), and it measures perception as a dynamic process which builds up gradually including top-down (e.g., prior knowledge) as well as bottom-up (e.g., sensory) processes. The validity of the approach has been shown in such diverse areas as speech perception (Spivey, 2005) and stereotyping (Freeman, Pauker, Apfelbaum, & Ambady, 2010). Johnson and Koop (2011) present the first study in JDM with this tool using gain and loss gambles. When choosing, mouse trajectories were used as an index of attractiveness. A direct trajectory was found to the less risky gain, when this option was chosen. When the more risky gain was chosen, however, this was combined with a slight tendency towards the less risky option first and then the move to the riskier one. For the loss domain a reversal of this finding was reported, although less pronounced. Thus, attractiveness may be known nearly from the onset of the decision process, which is not entirely consistent with the prospect theory (Kahneman & Tversky, 1979) view in which evaluation follows an editing stage.

3.5 Think aloud

Method

Participants’ verbal reports of thoughts concerning judgment and decision making are elicited. Ranyard and Svenson (2011) identified four broad procedures depending on whether verbalizations are structured or unstructured, and collected concurrently (in parallel to the JDM task) or retrospective (immediately after task completion). The most influential procedure has been the concurrent, unstructured method known as thinking aloud. Ericsson and Simon (1980; 1993; Fox, Ericsson & Best, 2011) advocate the use of a rigorous procedure including training participants in the verbalization process with unrelated tasks, and reminding them to continue speaking in case of pauses. Their main recommendation is that participants are not directed to describe specific types of information or to explain their thoughts. Task performance is presented and recorded with audio and/or video enabling the collection of data in the laboratory as well as in the field.

Theoretical contributions and results

Montgomery and Svenson (1989) carried out an early think aloud study of real estate decisions. The verbal data obtained was used to test predictions of a multi-stage model of decision making known as the dominance structuring model (Montgomery, 1983), involving the early identification of a promising alternative and later re-evaluation of aspects of the alternatives. It was predicted, and found, that well before an option was finally chosen it received more attention and was more positively evaluated than other alternatives. More generally, an extensive range of information can be elicited from verbal reports, including evaluations of information presented, conscious contents of mental representations such as goals and plans, strategies consciously applied, and feelings of specific emotions. Clearly, cross-validation of findings from subjective verbal data is necessary. Harte, Westenberg and van Someren’s (1994) approach is useful, whereby the consistency of verbal data with task analysis is examined, using an independently derived process model. In addition, findings from verbal data concerning decision processes should be checked for consistency with decision behavior and with non-verbal process measures such as InterActiveProcess Tracing (Reisen, Hoffrage & Mast, 2008) discussed below.

Before concluding this section, we would like to point to a recently developed tool, Flashlight (Schulte-Mecklenbeck, Murphy, & Hutzler, 2011), which offers a combination of the level of detail found in Mouselab, and the flexibility of eye tracking. As Flashlight has only been used in a proof of concept stage we did not include it into Table 1 above. However, Flashlight has potential since it is freely available, can be used over the Internet, and has great flexibility in stimulus selection.

3.6 The big issue: What to do with all the data?

Often researchers are enthusiastic when new process tracing methods are introduced or when an older method witnesses a revival. However, once the hurdle of data collection is passed, which is surely higher than for simple input-output data, researches are often facing a new, unfamiliar problem: the sheer magnitude of the collected data is overwhelming. To put this somewhat into perspective: in a questionnaire study we collect one response per task (the choice) which is often accompanied by several additional responses like, e.g., confidence ratings. We manipulate several variables and use repeated measures, so we end up with, say, one hundred data points per participant. The picture changes considerably when we collect for instance eye-tracking data. With a current eye-tracking system several hundred data points are collected per participants for each second of an experiment. Given that each task lasts several minutes and we run more than one task we end up with tens of thousands of data points per participant. Whoever has opened a 100 Megabyte raw data file, knows the feeling. Different approaches have been developed to tackle this problem. We want to highlight three of these approaches:

1. 1) Indices: Payne (1976) introduced the idea of building a ratio between within and between option transitions more than 30 years ago. This simple ratio summarizes an important property of the overall acquisition behavior of the participant. Böckenholt and Hynan (1994) criticized this measure for ignoring the actual setup of the stimulus (the number of alternatives and attributes), the authors introduced a search metric (SM) that solves this problem and which is the preferred index at the moment.

2. 2) Metrics: Riedl, Brandstätter and Roithmayr (2008) introduced an approach which takes into account different metrics (a ratio of the time spent on the different options, a search index as described above, or whether participants weight the different options or not) which are building blocks of an overall list of strategies. This list of strategies can then be applied to the collected data, moving from the process level to the strategy level.

3. 3) Algorithms: Day (2011) describes a new approach to looking at process data which applies statistic algorithms to a dataset. This has the benefit of being able to deal with huge datasets in an easy way and works well with eye-tracking data as demonstrated by the author. In addition to different methods for analyzing process data, important techniques for visually representing them have continued to be developed. For example, Johnson et al. (2008) have devised icon graphs which incorporate time, frequency and transition measures into one, easy to read display.

4 Discussion

We have summarized tools for collecting process data with a focus on information acquisition. By using some examples, we have shown how these data have been used for important developments in JDM with respect to model building. We argue that the potential of process tracing methods has not yet been fully realized. Significant advances can be expected given that two issues are resolved: First, process tracing is at its best when clearly formulated hypotheses exist that directly relate to process data. Second, as any single method has its weaknesses, specific combinations of methods can compensate for some of these weaknesses. We end this contribution by briefly elaborating these two issues.

4.1 Explicit hypotheses

By definition process data are more directly focused on JDM processes than input-output data and therefore are a more empirically valuable source of evidence regarding process models. In terms of our example: a choice can simultaneously conform to many models (SEU, prospect theory, use of a minimax heuristic, use of the priority heuristic, etc.), and often cannot be unambiguously interpreted. For both as-if and process models this problem has been resolved with outcome data via the careful analysis of alternative model predictions to identify critical tests that differentiate them in terms of predictive validity (see, for example, Tversky, 1969). More recently Glöckner and Betsch (2008) have illustrated how diagnostic task selection can facilitate this strategy. In addition to this, process models can be investigated more thoroughly by analyzing their implications for both process and outcome, with process data being afforded equal status with outcome data. Furthermore process data are more specific and offer more scope for falsification. They thus have more empirical substance with respect to process models. The quest then is for explicit hypotheses, that are as directly as possible related to process data. A good example of a valuable application of this perspective is the work on the priority heuristic described earlier.

4.2 Multi-method approach

Process tracing tools have been applied to JDM tasks in a variety of ways. They have been used as focal tools for model building and development (rarely), or as peripheral tools enabling additional tests to be made (frequently). Using a multi-method approach by combining tools will significantly enhance their utility. This quest for the multi-method approach is not new. From time to time researchers argue for using process tracing tools in combination with input-output analysis (Ford, Schmitt, Schechtman, Hults, & Doherty, 1989; Payne & Venkatraman, 2011). We also argue for the combination of different process tracing data. A good example is Reisen, Hoffrage, & Mast (2008), who combined active information search, Mouselab, and retrospective verbal protocols in their tool called InterActiveProcess Tracing.

Following Payne and Venkatraman (2011), the last 60 years of JDM have witnessed four revolutions: (i) formal modeling of human judgment; (ii) the adoption of the information processing approach; (iii) the emotional revolution, and (iv) the brain revolution. Formal modeling can be carried out with input-output data only; no process tracing is necessary. The information processing approach focuses on process, rather than products, and process tracing is a focal method in this tradition. Consequently, process tracing tools, mostly of information acquisition (e.g., verbal protocols, information boards) have been used. Recent technical developments, most notably with information board studies have enhanced the value of these methods. Emotional processes are largely beyond the reach of these classical tools and new tools are needed. For instance, pupil dilation or skin conductance studies can offer insights into such areas (Figner, Mackinlay, Wilkening, & Weber, 2009; Wang, Spezio, & Camerer, 2010). In addition, there is a variety of factors limiting the value of explicit measurements (e.g., in the form of verbal data), that have plagued JDM research, most notably in relation to dual process models. These are limits in motivation, opportunity, and ability to report, and limits in awareness (Wilson & Brekke, 1994). We think that process tracing tools overcoming these limits by tapping implicit processes (e.g., eye movements, or mousetracking) have considerable heuristic value since they allow the concurrent mapping of explicit and implicit processes.

Probably a combination of tools from different revolutions is most promising. Such combinations will be advantageous for at least two reasons: first, different tools can better compensate for the weaknesses of one another; second, collecting process data pertaining to different paradigmatic traditions urges researchers to specify models incorporating the different traditions. In all likelihood, a more complete picture of human judgment and decision making will result from such models.