Process dynamics in delay discounting decisions: An attractor dynamics approach

Abstract

How do people make decisions between an immediate but small reward and a delayed but large one? The outcome of such decisions indicates that people discount rewards by their delay and hence these outcomes are well described by discounting functions. However, to understand irregular decisions and dysfunctional behavior one needs models which describe how the process of making the decision unfolds dynamically over time: how do we reach a decision and how do sequential decisions influence one another? Here, we present an attractor model that integrates into and extends discounting functions through a description of the dynamics leading to a final choice outcome within a trial and across trials. To validate this model, we derive qualitative predictions for the intra-trial dynamics of single decisions and for the inter-trial dynamics of sequences of decisions that are unique to this type of model. We test these predictions in four experiments based on a dynamic delay discounting computer game where we study the intra-trial dynamics of single decisions via mouse tracking and the inter-trial dynamics of sequences of decisions via sequentially manipulated options. We discuss how integrating decision process dynamics within and across trials can increase our understanding of the processes underlying delay discounting decisions and, hence, complement our knowledge about decision outcomes.

1 Introduction

Many everyday choices involve options that pose a conflict between immediate, but small gains, and delayed, but larger or more beneficial gains. This conflict occurs on many time scales. For example, you might wonder whether to enjoy spending your money now or saving it for a pension. Or you might be seduced to take the hearty burger – which is immediately very tasty - instead of the light salad – which might be better for your figure in the long-term. In such intertemporal choices (for a review, see Frederick, Loewenstein & O’Donoghue, 2002), humans discount the offered gain by the delay of delivery. This delay discounting is well described by utility discounting models which assume that the greater the delay in delivery of a reward, the more the utility of a reward is discounted. Hence, these discounting models represent the subjective value of a reward as a function of its delay (see Doyle, 2013 for an overview). While these models offer a good description of the average outcome of the decision process – the final choice – they mostly leave open how the decision process itself unfolds in time. Dissecting this process, however, is necessary in order to fully understand the way decisions are made as well as the sources of failures and deviations from the average discounting model (Lempert & Phelps, 2016). To fill this gap, recent developments aim to uncover the process dynamics leading to a final decision in delay discounting (Dai & Busemeyer, 2014; Rodriguez, Turner & McClure, 2014).

Here, we will combine the static description of decisions offered by delay discounting models with the process dynamics as described by attractor models. Such attractor models have been used successfully to study perceptual decisions (see Tuller, Case, Ding & Kelso, 1994 for the model also sketched out here), and recently also higher level decisions, e.g., risky decisions (van Rooij, Favela, Malone & Richardson, 2013) or the development of preferences (O’Hora, Dale, Piiroinen & Connolly, 2013). Applying an attractor model to delay discounting decisions enriches the mere uncovering of discounting curves by adding a level of description for the processes leading to a final choice and, thus, allowing for intuitive, heuristic theorizing about this process and possible failures resulting from it. With its simple mathematics and metaphorical heuristic power, an attractor model offers a bridge between the simple yet static models of intertemporal decisions incorporated by discounting curves and other more dynamic yet also more complex models of decision processes, e.g., neural network models. In the remainder of this paper, we will first derive our attractor model of decision making in delay discounting. Based on this model, we will then derive qualitative predictions about the dynamics of the decision process within single choices as well as across sequences of choices, and about the resulting deviations in the final decision. Finally, we will test these predictions with a newly developed discounting paradigm in four behavioral experiments.

1.1 Deriving an attractor model of decision making in delay discounting

Discounting models are based on the common ground of a psychometric perspective (Takahashi et al., 2008) that follows psychophysical reasoning (Fechner, 1860): Their common aim is to derive the best function that describes choice outcomes. Such a function provides a clear decision criterion determining an individual’s choice of the sooner-smaller (SS) option over the later-larger (LL) option and vice versa for a given combination of times and values. The function does this by specifying indifference points on a graph of subjective value as a function of delay, that is, points at which the LL option has the same subjective value as the SS option. A decision involving combinations of values and delays that lie far away from these indifference points should be relatively easy (i.e., induce low levels of decision conflict) because the two options are very different with regard to their subjective value. In contrast, a decision involving combinations of values and delays that lie close to indifference will be relatively difficult (i.e., induce more conflict).

In this regard, high-conflict decisions are comparable to perceptual decisions when observing ambiguous figures like the Necker Cube. The Necker Cube offers two equal perceptual perspectives – an upwards and a downwards one – between which an observer’s perception alternates continuously. Similar to the visual system choosing between the two possible perspectives on the ambiguous Necker Cube, an intertemporal decision can be interpreted as an ambiguous state where the cognitive system chooses one of two conflicting delayed options. In the perceptual decision making literature, attractor models have proven themselves as a useful tool to understand the processes underlying choice variability in the perception of ambiguous stimuli (Hock, Schöner & Giese, 2003; Noest, Ee, Nijs & Wezel, 2007). This is because attractor models provide a high-level, abstract description of the otherwise complex, non-linear dynamics lying at the core of the different classes of more complex models (Onnis & Spivey, 2012) that have been used in the past to study ambiguous perception such interactive activation and competition networks (Rumelhart & McClelland, 1986) or parallel constraint satisfaction networks (Feldman, 1981). Because attractor models provide a simple yet elegant mathematical description of the dynamics of a perceptual decision process, they are also well-suited to reason about and to model value-based decision processes. Value-based decisions have also been studied based on network models, e.g., parallel constraint satisfaction networks (Glöckner & Betsch, 2008; Glöckner, Hilbig & Jekel, 2014), suggesting that the attractor model could again provide a parsimonious high-level description of the underlying decision processes. In order to test this assumption for delay discounting decisions, in the following we will first apply such an attractor model¹ (Tuller et al., 1994) to delay discounting from which we will then derive and test empirical hypotheses about observable behavior.

In such an attractor model, the two options are represented by two attractors, that is, two valleys in a potential- or energy-landscape (Figure 1). The deeper a point in such an energy landscape, the more stable is the respective potential state of the system and the higher is the certainty that the system makes its decision. In comparison to the stable end-states representing the two options, the starting state of the system is unstable and lies in the middle between the two valleys. In this indecisive state, no decision has been made by the system and both choices are still possible. To make this abstract description more concrete, we can apply this attractor-model to the activation of neural representations of the SS and the LL option. The two attractor-states reflect the exclusive activation of the SS or the LL representation, indicating complete certainty of the system that this option is the one to choose. The states in between these stable attractors, e.g., the start state, reflect varying amounts of concurrent activation in which the system has not settled into a decision yet (see the appendix for a formal implementation of such a neural system).

Figure 1: Sketch of the attractor model for decisions with two possible choices representing a sooner smaller (SS) and a later larger (LL) option. The potential-landscape defines a one-dimensional state-space of the system representing all potential states of decisiveness for/against a certain option. The depth of the potential wells (as shown in the five insets on the left) defines the stability/attractiveness of each of these states. The relative depth of the well is represented by the control parameter c. This parameter depends on the relative difference in subjective value (attractiveness) of the options for a subject and hence configures the system for each potential combination of SS and LL options (as indicated by the continuous variation of c on the right): An increase in attractiveness for the SS option (e.g., because the SS option’s value becomes higher while both options’ delays are held constant) results in a negative control parameter which, in turn, increases the depth of the attractor representing the SS option. In contrast, an increase in attractiveness for the LL option (e.g., because the LL option’s delay is reduced while both option’s values are held constant) results in a positive control parameter which, in turn, increases the depth for the attractor representing the LL option. Hence, the control parameter c is primarily dependent on the values and delays of the presented options, but also on a subject’s tendency to discount, as indicated in Figure 2. Within this potential landscape, the current system state (marked by a red dot) tends to move to the bottom of the potential wells and travels through all intermediate states on its way to a stable final choice. The deeper a potential well of an option (compared to the alternative), the more probable, direct, and quicker is the movement of the current system state into this potential well.

In an attractor-model, the depth of the attractors and hence the stability of its end-states varies with the properties of the system’s environment. In our case, the crucial environmental properties are the value and the delay of the two options which – in combination – determine each option’s relative attractiveness², that is, its subjective (discounted) value. The difference in relative attractiveness between the SS and LL option (which elegantly combines all choice-relevant environmental information about the options’ values and delays) is represented by the control parameter, which we will call c. In the example of neural representations for the SS and the LL option, c reflects the relative difference in the input to the neural representations. Accordingly, the special case where c = 0 (Figure 1 middle panel) represents a decision where both options receive an identical input and are thus equally attractive. In such high conflict situations, a neutral starting state would keep the system indifferent until slight differences in input (or random noise) tips the system to one side or the other, resulting in a more or less arbitrary decision. For c < 0, the SS option is more attractive, for example, because the SS option’s reward is almost as large as the LL option’s (Figure 1, left panel). In contrast, for c > 0, the LL option is more attractive, for example, because the LL option’s delay is almost as low as the SS option’s (Figure 1, right panel). Low conflict decision situations with large differences in relative attractiveness and values of c ≪ 0 and c ≫ 0 result in unambiguous decision situations with no decision conflict at all.

The connecting element between our attractor model and discounting curves is the attractor layout where c = 0 (Figure 2). At this point, both the attractors representing the SS and the LL option are equally deep and, hence, equally attractive – a state to which we refer as the indifference point (Figures 1 and 2). By determining the precise location of these indifference points for the different attractor landscapes resulting from changes in the environment (such as the time interval between the two options in an intertemporal decision, see Figure 2), we arrive at the discounting curve over all delays.

Figure 2: Illustrative delay discounting curve (in blue) and the respective underlying attractor layouts (grey-scaled with darker shades of grey representing deeper attractors) of subjects in a previous experiment by Scherbaum, Dshemuchadse, Leiberg & Goschke (2013). Blue circles mark indifference points, that is, the subjective value of an immediate SS option that has the same subjective value as an LL option that is delayed by a given interval. (Only the time intervals 1, 4, and 7 are depicted here.) For a time interval of 4, for example, the indifference point indicates that an SS option has to yield a value_SS = 0.6 × value_LL in order to be (subjectively) equally attractive as the discounted LL option. The range of attractor configurations defined by the control parameter c (the grey scaled insets, see also Figure 1) is aligned so that the attractor layout for two equally attractive options (control parameter c = 0) is located at the indifference point. For combinations of options lying above the indifference point (e.g., interval 4 and value_SS = 0.8 × value_LL,) c is smaller than 0 and the SS option is more attractive than the LL option. For combinations of options below the indifference point (e.g., interval 4 and value_SS = 0.4 × value_LL,) c is larger than 0 and the SS option is less attractive than the LL option.

Importantly, indifference points vary between individuals which means that people differ with regard to how much they discount a given value due to the delay of its delivery. These individual differences are also reflected in the attractor model. The precise configuration of the attractor landscape for a given combination of options depends on a person’s individual level of discounting, because this level of discounting defines the subjective values of the options. Hence, the attractor model is aligned at each interval to a subject’s individual discounting curve and in that way the control parameter c integrates the properties of the environment (the options’ relative values) and individual differences (the subjective value of the options’ discounted values as determined by the discounting curve). The attractor model thus integrates the amount of discounting as indicated by discounting functions and the overall process description of a choice, as indicated by a given attractor configuration which defines how the system’s current state develops over time until a final choice is made.

The temporal extension that the attractor model offers also provides a means to study the temporal dynamics of intertemporal decisions on two time-scales. One, already mentioned, is an intra-trial time-scale that focuses on how a final choice comes about over the course of one decision. The other is the inter-trial time-scale that focuses on how consecutive decisions may influence one another. Classic discounting-curves are blind to both kinds of time-scales. The former extension that pertains to the intra-trial dynamics of decision making focuses on how the system state gradually moves into one of the two attractors over the course of a trial until it makes a final choice.

However, this more process-oriented perspective on intertemporal decision making is not a new one. For example, sequential sampling models (Dai & Busemeyer, 2014; Rodriguez et al., 2014) also describe how decisions come about by describing how information about the offered options accumulates gradually until one of them hits a threshold, thus, eliciting a decision. The major step forward offered by the attractor model over sequential sampling models is the extension to the inter-trial time-scale: While sequential sampling models focus on intra-trial dynamics exclusively, the attractor model makes strong predictions about the inter-trials dynamics occurring over the course of multiple decisions. It inherently assumes that the attractors are formed by the offered options and, hence, these attractors are not present between trials. This leads the system state to stay in the area where it ended up previously – in the vicinity of the vanished attractor representing the recent choice – and to relax only slowly to the neutral start state under no input. For example, if the model choses the SS option in a first decision trial, it would remain in the vicinity of the SS attractor in the inter trial interval. In a second decision trial, it would hence start the decision with a bias to the SS decision, even if this trial comprises of a more attractive LL option (Figure 3).

Figure 3: Inter-trial dynamics in the attractor model. Choosing SS in a first trial leads to a bias in a second trial due to slow relaxation of the system state during the inter trial interval..

In sum, the attractor model implies that the previous system state automatically carries over to the next trial, thus making the system’s behavior in a given trial inherently dependent on the history of its previous decisions (Scherbaum, Dshemuchadse & Kalis, 2008; Townsend & Busemeyer, 1989).

In the following, we will use the attractor model to derive predictions about intra-trial and inter-trial dynamics in real decision-makers. The simplicity of the model allows us to derive these predictions in a purely qualitative, argumentative manner. However, Appendix I shows that the exact same hypotheses can be derived by means of computational simulation based on a competitive neural-network.

1.2 Predictions from the attractor model

The intra-trial process dynamics focus on the movement of the system state into the attractor of the final choice. Here, we derive two predictions about the decision process. The first prediction concerns decisions in which both options are equally attractive (i.e., high conflict decisions) compared to decisions in which one option is clearly more attractive than the other one (i.e., low conflict decisions): For high conflict decisions, the system should settle only slowly into one of the two equally deep attractors compared to a quicker movement into the deeper attractor for low conflict decisions.

The second prediction concerns only low conflict decisions: When the system chooses the more attractive option – the deeper attractor – it should settle into this attractor quickly, compared to choosing the less attractive option. Note that choosing the less attractive option is possible if the system already is in the vicinity of this option’s attractor. This could happen because of previous choices of this option leading the system to stay within the vicinity of an option’s attractor (compare Figure 3). As explained above, this is a typical phenomenon for inter-trial dynamics as explained in the following.

The inter-trial process dynamics focus on the stability of choice patterns across consecutive decisions, and hence, on staying within the vicinity of one attractor or switching from one attractor to the other one as induced by changes of attractiveness and stability. Here, we derive a third prediction for decisions with either one or two potentially attractive options. For no conflict decisions with only one attractive option we expect the system to always choose the same option, irrespective of previous decisions. In contrast, for high conflict decisions with two potentially attractive, but not necessarily equal options we expect the system’s choice to depend not only on the attractiveness of the currently presented options but also on contextual factors, that is, the system’s history of previous choices – a phenomenon that is also known as path-dependence or order-effects.

For example, a choice of the SS option in one trial should bias the starting state of the system for the subsequent trial which can result in a renewed choice of the SS option – even when its attractiveness has been slightly reduced (Noest et al., 2007). In its most extreme form this phenomenon is also known as hysteresis (Tuller et al., 1994): If a sequence of choices starts with a very attractive SS option, this option will be chosen initially; decreasing the attractiveness of the SS option systematically (decreasing c) will only then lead to a switch to the LL option when the SS option’s attractiveness approaches zero. This is because the system state stays in the vicinity of the initially chosen SS option and leaves this attractor only when it has vanished completely (Figure 4, blue arrows). In contrast, if a sequence of choices starts with an attractive LL option and then the attractiveness of the LL option is systematically decreased (increasing c), the switch to the SS option happens only after the LL option’s attractiveness has reached (almost) zero, because the system stays in the vicinity of the attractor of the LL option until it vanishes completely (Figure 4, red arrows). Simply put, this means that the outcome of a given choice depends not only on the current properties of the options but also on the history of previous choices.

Figure 4: Changes in the attractor model as a function of the control parameter c. Brightness reflects the attractor layout as depicted in Figure 1 (black = deep attractor). Arrows mark the final choice of the system; arrows’ directions indicate the direction of change of c from two different starting states. If a sequence of choices starts with an attractive SS option (blue), the system will stay with the SS option despite changes in c making the SS option less and less attractive from trial to trial. Only when the SS option becomes very unattractive, the system switches to choose the LL option. If the sequence starts with an attractive LL option (red), the system stays with the LL option despite the changes in c. In the area near c = 0 (the indifference point with two equally stable attractors), the system has two stable states: In this range, whether the system chooses the SS or the LL option depends on the parameter’s history. Hence, the switch point between the two states depends on the state the system initially settled into. At c ≪ 0 and c ≫ 0, the weaker attractor completely loses stability and hence, only one stable state exists and the system reliably chooses only one option.

If we find such predicted behavior in real decision makers, it will stress why it is important to extend discounting functions with a compatible process account: While discounting functions readily describe the average outcome of multiple decision processes, they do not take the system’s choice history into account and, hence, cannot predict such a pattern (but see Scholten & Read, 2006, 2010). While an attractor delay discounting model parallels discounting functions in the way that it also defines a clear decision criterion, namely, through the layout of the attractor, it extends them through the idea that the system’s actual behavior is produced by the interaction of this attractor layout with the current state (and, hence, history and potentially other influences) of the system which can potentially lead to final choices that deviate from the discounting model. In the following, we will test the attractor model and the predictions for the process dynamics of intertemporal choice derived from it empirically. Note that, while we applied a metaphorical approach to derive our predictions in the previous paragraphs, Appendix I describes a formalized derivation of these predictions from a computational model.

1.3 A dynamic investigation of delay discounting

To validate the predictions of the attractor model, we needed a delay discounting setup that could provide us with markers for the expected phenomena. Concerning the predictions on the intra-trial process dynamics, a fruitful approach to study the temporal dynamics of decision processes behaviorally is mouse tracking (Scherbaum, Dshemuchadse, Fischer & Goschke, 2010; Freeman, Ambady, Rule & Johnson, 2008; Dale, Kehoe & Spivey, 2007; Kieslich & Hilbig, 2014; Koop & Johnson, 2011). In this methodology, subjects make binary choices by moving a computer mouse into the left or right corner of the computer screen instead of pressing left or right buttons. Importantly, the movement trajectories recorded with this method not only provide the final outcomes but also capture features of the decision process. In particular, for several cognitive tasks (Scherbaum et al., 2010; Song & Nakayama, 2009; Spivey, Grosjean & Knoblich, 2005) as well as for a standard intertemporal choice task (Dshemuchadse, Scherbaum & Goschke, 2012), strong competition between alternative responses produces less direct movement trajectories. Hence, regarding prediction 1 we expect more direct movements to the final choice in low conflict decisions and less direct movements to the final choice in high conflict decisions. Regarding prediction 2, we expect choices of the attractive option to be more direct than choices of the unattractive option.

Concerning the prediction of hysteresis for the inter-trial process dynamics, standard intertemporal choice tasks (for example, as used in Soman et al., 2005) pose a problem, since the options’ values and times are presented verbally and explicitly on a screen. This makes sequential manipulation of values or times simply too obvious (as indicated by the results of Robles & Vargas, 2008). Hence, for our investigation we use a recently developed, non-verbal delay discounting task (Scherbaum et al., 2013). In this task, subjects collect coins of different sizes with an avatar which they move on a checkered playing field by clicking with the computer mouse (Figure 5). The playing field stayed constant across trials — except the options which changed from trial to trial — and the avatar started each trial from the position of the previously chosen option. The goal was to collect as much reward as possible in the allotted amount of time. In each trial of the task, subjects have to choose between two reward options of different magnitude (small vs. large) at different distances (near vs. far fields). So unlike standard intertemporal choice tasks, this paradigm allows us to observe intra-trial dynamics through mouse movements (Experiment 1) and investigate inter-trial effects of hysteresis by manipulating (without any notification) the distance between the two options for sequences of trials (Experiments 2–4).

Figure 5: The dynamic delay discounting paradigm. Subjects moved an avatar across a playing field by clicking with the mouse into horizontally and vertically adjacent movement fields (white border). They had to collect rewards (red border), with one reward being near but small (small coin) and one reward being far but large (large coin). The remaining time (“Zeit”) within one block and the collected credits (“Gewinn”) were presented next to the playing field. Zoomed Inset: Mouse movements were measured from the position of the avatar to the first click into a movement field. Movement deviations in the direction of the unchosen option were used as a measure of the attractor layout underlying the decision.

2 Experiment 1

Experiment 1 aims to validate the predictions for low vs. high conflict trials and for attractive vs. unattractive choices. By investigating these two predictions of intra-trial dynamics, it offers a fine-grained validation of the model. For Experiment 1, we report newly analyzed data from a study using the dynamic delay discounting paradigm described above (Scherbaum et al., 2013).

2.1 Methods

2.1.1 Subjects

25 students (15 female, mean age = 23.04 years) of the Technische Universität Dresden took part in the experiment. All subjects had normal or corrected to normal vision. They gave informed consent to the study and received a 3 € show-up fee and the money they collected within the experiment.

2.1.2 Apparatus and stimuli

The experiment was presented on a 17-inch screen (1280 x 1024 pixels, 75 Hz). As presentation software, we used Psychophysics Toolbox 3 (Brainard, 1997; Pelli, 1997) in Matlab 2006b (the Mathworks Inc.), running on a Windows XP SP2 personal computer. Responses were carried out by moving a high precision computer mouse (Logitech Laser Mouse USB). Mouse movement trajectories were sampled with a frequency of 92 Hz. Recording started at presentation of the choice options and lasted until the first click into a movement field (see procedure).

Subjects’ moved an avatar on a playing field divided into 20 x 20 fields (Figure 5). To move the avatar, subjects clicked with the mouse in one of four horizontally or vertically adjacent movement fields, as signaled by a white border surrounding the fields. On each trial two reward options were presented as coins on fields marked with a red border: One reward was near but small, the other reward was far but large. The two options’ positions were always chosen so that the first move into one direction decreased the distance to one option but increased the distance to the other option. This way, the first move of the avatar already represented a clear preliminary decision for one option and against the other option.

For both options, the size of the coin represented the reward valueand the horizontal and vertical distance of the reward field to the field of the avatar represented the distance of the option. Reward values ranged from one to ten credits and distance ranged from two to fifteen fields. For better comprehensiveness in the context of intertemporal choice, we maintain in the following the standard description of the time dimension using “soon”, “late”, “delay”, and “interval”, although in our scenario time delay is represented by spatial distance.

Next to the playing field (Figure 5) subjects could see the remaining time within one block and the collected credits in Euro (1 credit = 1/10 € cent).

2.1.3 Procedure

Subjects’ task was to collect as much reward as possible within the allotted time limit. In each trial, they had to choose between two reward options (one soon but small, one late but large; see design). They collected the selected reward by moving their avatar with the mouse across the playing field. Moving over longer distances took more time, and thus was more costly with respect to the limited time.

A trial started with an inter trial interval (ITI) of 1.3 seconds. Within this interval, the mouse cursor was locked in the center of the field containing the avatar. After the ITI, the two options were presented. As soon as the two options appeared, subjects could click on the adjacent movement fields to move their avatar towards the chosen option. When the avatar reached one option, both options disappeared, the value of the selected option was shown to the subject, and the next trial started.

The experiment consisted of three blocks, with one block lasting eight minutes. This amount of time allowed subjects to work through the complete design matrix of trials (see design) at least one time. Between blocks, subjects were informed about the credits collected and were instructed to rest briefly before the self-paced start of the next block.

Before the start of the experimental blocks, subjects worked through a test block of one minute to get used to the virtual environment and handling of the mouse.

2.1.4 Design

Reward values ranged from one to ten credits, with the reward values of the two options in each trial adding up to eleven credits to keep the overall value of each trial constant (smaller/larger reward pairs were: 1/10, 2/9, 3/8, 4/7, 5/6). Distances ranged from one to fourteen fields with the nearer option being at a distance (D _S) of 1, 3, or 7 fields and the additional interval to the farer option (D _L - D _S) being 1, 2, 4, or 7 fields. Reward values, smaller distance, and interval to the larger distance were varied orthogonally with a random order of trials. Importantly, the reward/distance combinations were chosen such that consistently choosing only one option, either the SS or the LL option, yielded worse results than a dynamic strategy that optimizes each choice by the current options’ value/time ratio.

The combination of 5 value combinations, 3 distance of the SS options and 4 intervals between the SS and the LL option yielded a complete sequence of 60 trials. We generated 8 such sequences, with a randomized order of trials within each sequence. This resulted in 480 trials that subjects could potentially complete in the whole experiment. From pretests, we knew that subjects would run into the time limit of the experiment (3 blocks, 8 minutes each) before completing all potential trials.

2.1.5 Data preprocessing

Choice categorization.

To distinguish low conflict decisions from high conflict decisions (prediction 1) we categorized trials as low vs. high conflict trials in three steps. First, we determined the indifference point for each interval, that is, the subjective (discounted) value of a delayed option that is equivalent to the value of a (hypothetical) immediate option. As an estimate of the indifference point, we determined the point of inflection of a logistic function fitted to the individual choices (SS vs. LL) as a function of increasing value differences (Ballard & Knutson, 2009; Dshemuchadse et al., 2012). Second, we took these indifference points and calculated the distance in subjective value of each specific trial to this indifference point – that is, the difference between the indifference point and the small/large reward ratio of the trial. We then categorized all trials by performing a median split on their distance relative to the indifference point: trials with a low distance to a given subject’s indifference points were categorized as high conflict and trials with a large distance to a given subject’s indifference point were categorized as low conflict (Robles & Vargas, 2008; Scherbaum et al., 2013). Hence, high conflict trials overall yielded a low distance to the indifference point (M= 0.14, SD = 0.01) whereas low conflict trials overall yielded a high distance to the indifference point (M = 0.47, SD = 0.02).

To distinguish choices of the attractive option from choices of the unattractive option (prediction 2), we calculated the ratio of costs (invested time) and benefits (received credits) for both alternatives in each trial, choosing the SS or the LL option (Scherbaum et al., 2013). We inferred costs from the number of fields necessary to reach the option and from the movement-speed of the respective subject. If subjects chose the option with the better benefit/cost ratio, we defined that trial as attractive-chosen; if subjects chose the option with the worse benefit/cost ratio, we defined that trial as unattractive-chosen.³

Mouse movements.

Subjects’ movements were recorded from the center of the avatar’s current field until the click into the respective movement field. For analysis we followed the procedure of the original study (Scherbaum et al., 2013): Trajectories were time-normalized in length to 33 time slices per trial and aligned to common starting and end positions such that an ideal movement would be a straight line and less direct movements would show a deflection of the trajectory from this straight line into the direction of the alternative option. With regard to statistical testing, this allows to analyze the mean deflection – the area under the curve – in a trial as a marker for less direct movements (Dshemuchadse et al., 2012; Scherbaum et al., 2010; Spivey et al., 2005).

2.2 Results

On average, subjects completed 351 out of the possible 480 trials (SE = 7). Hence, subjects completed each of the 60 possible combinations of delays and values about 6 times. In the following, we will present the results of the reanalysis of existing data with a focus on the dynamic predictions derived from the attractor model. Analyses on choice behavior and discounting functions can be found in the report of the original analyses (Scherbaum et al., 2013).

The first prediction derived from the model was that high conflict trials in which the two options are equally attractive should show more heavily deflected movements than low conflict trials in which one of the options is clearly more attractive. Accordingly, a one-sided t-test on mean deviation for low conflict vs. high conflict trials revealed the expected larger deviations for high conflict trials (M = 1.89 px, SE = 0.22 px) compared to low conflict trials (M = 1.43 px, SE = 0.19 px), t(24) = 2.05, p = 0.025, d = 0.41 see Figure 6, left).

Figure 6: Results of Experiment 1. Mouse movements from the starting position (center of the position of the avatar) to the position of the first click (in the first movement field). A straight line along the Y-axis would indicate a direct movement while deviations to the right indicate a deflection of the movement to the unchosen alternative option and hence conflict in the decision process. Shaded areas represent standard errors. Left: Movements for low vs. high conflict decisions. Right: Movements for low conflict decisions in which the attractive option was chosen or the unattractive option was chosen.

The second prediction derived from our attractor model of delay discounting was that, in low conflict trials in which one of the options is clearly more attractive than the other one, choosing the unattractive option should show more heavily deflected movements than choosing the attractive option. Accordingly, a one-sided t-test on mean deviation for low conflict attractive-chosen vs. unattractive-chosen trials revealed the expected larger deviations for unattractive-chosen trials (M = 2.75 px, SE = 0.45 px) compared to attractive-chosen trials (M = 1.43 px, SE = 0.19 px), t(24) = 3.27, p = 0.002, d = 0.65, see Figure 6, right).

2.3 Discussion

In Experiment 1, we reanalyzed data from a previous study (Scherbaum et al., 2013) to validate our predictions from the attractor model of delay discounting. Low conflict choices showed more direct movements than high conflict choices and choices of the attractive option showed more direct movements than choices of the unattractive option. This indicates that for intra-trial processes, the attractor model correctly describes the process dynamics. The next two experiments will focus on the inter-trial dynamics and explore the prediction of hysteresis by systematically varying the relative attractiveness of the two options across trials, which would provide further evidence for the validity of the model.

3 Experiment 2

Experiment 1 corroborated the predictions of the model for intra-trial dynamics. This leaves untested the decisive extension to inter-trial dynamics, for which the model predicted hysteresis. Experiment 2, will test this prediction by slowly varying the interval between the SS and the LL option. If hysteresis was present, this should lead to subject’s choices showing stronger discounting behavior for sequences with a decreasing delay to the LL option in comparison to those with an ascending delay.

3.1 Methods

3.1.1 Subjects

20 students (11 female, mean age = 24.4 years) of the Technische Universität Dresden took part in the experiment. All subjects had normal or corrected to normal vision. They gave informed consent to the study and received a 3 € show-up fee and the money they collected within the experiment.

3.1.2 Apparatus, Stimuli and Procedure

The setup followed the setup of Experiment 1. Subjects’ task was again to collect as much reward as possible within the allotted amount of time. In each trial, they had to choose between two reward options

3.1.3 Design

To study choice sequences, we consecutively increased or decreased the delay of the LL option to the avatar while keeping all other factors constant within the sequence. Hence, we orthogonally varied the difference in delay between the SS and the LL option (D _L – D _S = 1 to 12 fields, manipulated sequentially), the direction of these sequences (direction = ascending or descending), and the delay to the SS option (D _S = 2 or 3 fields). This resulted in four possible sequences (direction x D _S) of 12 trials (D _L). Orthogonally to these combinations, we varied the value V of the options, (V_S = 1, 2, 3, 4, and 5 credits; V_L = 10, 9, 8, 7, and 6 credits), following the original experiment (Scherbaum et al., 2013). Hence, a set of trials consisting of all combinations included 5 (values) x 2 (SS delay) x 2 (sequence direction) x 12 (delay difference) = 240 trials. We created 2 such sets, with each set’s order of sequences being randomized individually, resulting in 480 trials. Similar to the original experiment, subjects did not finish all of these trials due to the time limit, but worked on each sequence of trials at least once.

3.2 Results

On average, subjects completed 379 out of 480 possible trials (79 %, SD= 11.4 %), with the SS option being chosen in 44.3 % (SD = 13.6 %) of these trials. We hence successfully acquired each complete combination of trials (240 trials) at least once in each subjects’ dataset. As the order of sequences was randomized across subjects, repeated trials from partially completed sequences were also included into the analyses.

We checked for the effects of the sequence manipulation by performing an ANOVA on discounting curves with the factors interval and direction (ascending/descending; see Figure 7, top right). As expected, we found a main effect of the factor interval (F(11,209) = 102.87, P < 0.001, η ²= 0.84), indicating discounting, a main effect for the factor direction (F(1,19) = 18.34, P < 0.001,η ²= 0.49), indicating hysteresis, and an interaction interval x direction (F(11,209) = 5.06, P < 0.001, η ²= 0.21), indicating different amounts of hysteresis for different intervals. In line with the model’s predictions, we found hysteresis in responses for intermediate intervals and intermediate value differences (Figure 7, left), with stronger hysteresis for shorter intervals. Importantly, our data showed that hysteresis is a general effect that is not only driven by a minority of extreme subjects: When checking individual subjects for hysteresis (Figure 7, bottom right) we found that 16 out of 20 subjects showed the hysteresis effect (i.e., a hyperbolic k-value that was lower for ascending sequences than for descending sequences), while only 4 subjects showed no or a slightly reversed effect.

Figure 7: Results of Experiment 2. Left: Median response patterns across intervals for different ratios of small and large values. Hysteresis is most prominent for intermediate intervals and value ratios. Top right: Grand average discounting curves indicating the subjective value of a delayed value as a function of interval/delay and order of presentation. Shaded areas (left) and error bars (right) indicate standard errors. Bottom right: Discounting as indicated by hyperbolic k-values of each subject for sequences with ascending and descending order of presentation.

3.3 Discussion

As predicted by the model, we found hysteresis in Experiment 2: When the delay between the two options increased from trial to trial, subjects stuck to their choice of the LL option. In contrast, when the delay between the two options decreased from trial to trial subjects stuck to their choice of the SS option. Hence, in for intermediate intervals, the choice depended not only on the current values and delays of the SS and the LL option but also on the history of previous choices. This lead to a shift of the discounting curve indicating less discounting for increasing intervals compared to stronger discounting for decreasing intervals. Concerning their choice behavior within a sequence, subjects perseverated in their choices: They tended to stick to the option chosen at the beginning of a sequence and had difficulties to release it even when its objective attractiveness decreased. This rather unintuitive finding validates the attractor model, but stands in stark contrast to psychometric models of delay discounting which assume that a system’s decision in a given moment is exclusively determined by the two options‘ values and their delay. The results of Experiment 2 imply that a systems’ choice history is a third factor contributing to the outcome of the decision process.

Experiment 2 also leaves a couple of questions unanswered. For example, while our ascending vs. descending sequence of presentation manipulation did indeed shift discounting curves, it did so only for the lower part of intervals. This result might indicate that higher intervals exhibit only a small area where two attractors are present at the same time. Moreover, one might argue that the found perseverative choice behavior may be related to a lack of precision in the presentation of the options’ values: Whereas the variation in values were presented as relatively small differences in the size of the coins, the variation in delays was represented by the readily observed distances of the options. To rule out such an alternative explanation for the found hysteresis as a perceptual effect (Kelso, 1997; Tuller et al., 1994) and to provide further evidence for its existence, we performed Experiment 3 where we used a different format of presenting the options’ values.

4 Experiment 3

Experiment 3 was identical to Experiment 2 except for the way in which the options’ values were presented: Instead of representing the value of each option implicitly by the size of a coin, the value was now indicated explicitly by a number printed on a coin with fixed size. We expected to replicate Experiment 2. Hence, we hypothesized that subjects would, again, show hysteresis, i.e., stronger discounting behavior for sequences with a decreasing delay to the LL option in comparison to sequences with an ascending delay.

4.1 Methods

4.1.1 Subjects

20 students (12 female, mean age = 25.0) of the Technische Universität Dresden took part in the experiment. All subjects had normal or corrected to normal vision. They gave informed consent to the study. As in Experiment 1 and two, they received a 3 € show-up fee and the money they collected within the experiment.

4.1.2 Setup and Design

The setup and design were the same as in Experiment 2, with one exception: Instead of representing the value of each option by the size of a coin, we used red numbers printed on a coin of constant size to explicitly show the reward value of the option (1 up to 10 credits).

4.2 Results

On average, subjects completed 372 out of 480 possible trials (77.4%, SD = 10%), with the SS option being chosen in 49.4% (SD = 12%) of these trials. Again, this meant that we were successful in acquiring at least one complete sequence of trials for each subject.

We checked for the effects of the sequence manipulation by performing a repeated measures ANOVA on discounting curves with the factors interval and direction(ascending/descending; see Figure 8, top right). As expected and similar to Experiment 2, we found a main effect of the factor interval (F(11,209) = 20.28, P < 0.001, η ²= 0.52), a main effect for the factor direction (F(1,19) = 163.01, P < 0.001, η ²= 0.9) and an interaction interval x direction (F(11,209) = 7.28, P < 0.001, η ²= 0.28). Checking individual subjects for hysteresis indicated that 18 out of 20 subjects showed a hysteresis effect (i.e., a higher hyperbolic k-value for ascending than for descending sequences), while only 2 subjects showed a slightly reversed effect (Figure 8, top right).

Figure 8: Results of Experiment 3. Left: Median response patterns across intervals for different ratios of small and large values. Hysteresis is most prominent for intermediate intervals and value ratios. Right top: Grand average discounting curves indicating the subjective value of a delayed value as a function of interval/delay and order of presentation. Shaded areas (left) and error bars (right) indicate standard errors. Right bottom: Discounting as indicated by hyperbolic k-values of each subject for ascending and descending order of presentation.

To check for differences between the graphical presentation of rewards as coins of different size (Experiment 2) and the presentation of rewards a numbers (Experiment 3), we compared Experiments 2 and 3 statistically by performing an ANOVA with the within-subject factors interval(1-4) and direction (ascending/descending) and the between-subject factor presentation(size of coin vs. numerical). As expected, we found a main effect of the factor interval (F(11,418) = 250.1, P < 0.001, η ²= 0.87), a main effect for the factor direction (F(1,38) = 37.61, P < 0.001, η ²= 0.5) and an interaction interval x direction (F(11,478) = 11.54, P < 0.001, η ²= 0.23). There was no main effect of the factor presentation (F(1,38) = 2.34, P = 0.14, η ²= 0.06) and no significant interaction with this factor (all P > 0.18).

4.3 Discussion

Experiment 3 replicated Experiment 2 by finding the expected hysteresis effect for intermediate intervals and value ratios. We found no relevant difference to Experiment 2, where setup and design were similar except for the way in which the values of the options were presented. The asymmetry in the shifting of discounting curves was also present in the second dataset, indicating that the highest intervals lead only to small areas in which two attractors are present at the same time.

While Experiments 2 and 3 both validated the inter-trial process dynamics predicted by our attractor model of delay discounting, they left one important question regarding one of its core assumptions unanswered. In the attractor model, the observed hysteresis effect results from the model’s persistence at an abstract level of integration that represents the options’ integrated attractiveness or their subjective utility. However, one might argue that the exact same dynamics could also result from persistence at earlier levels of the decision process. For example, more complex models of intertemporal choice (for example connectionist models of delay discounting as described in Scherbaum, Dshemuchadse & Goschke, 2012) not only contain levels that represent the abstract attractiveness of the two options but also include subordinate levels that represent preceding steps of information processing, for example the perception and representation of the delay and value features of each option. From this point of view, it would be possible that the persisting choice behavior we observed in experiments two and three did not actually result from persistence at the superordinate level of the SS vs. LL choice. Instead the same behavioral pattern could also be caused by a persisting activation of the previously chosen option’s features at a lower level of processing (i.e., the representation of its value or delay). As this would prime the perception of the features of the following options towards the previously chosen ones, this mechanism would also lead to a preference for the more similar option and hence to persisting choice behavior. As our model explicitly predicts hysteresis to take place at the more abstract, choice-based level of representation, this alternative explanation needs to be excluded.

While we could exclude simple response priming, since options were rearranged in every trial, we conducted Experiment 4 to exclude feature priming at lower levels as an alternative explanation.

5 Experiment 4

Experiment 4 was designed to locate the level of the found trial-by-trial persistence in choice behavior: A feature priming explanation which attributes choice persistence to the mere priming of concrete values and delays at lower levels of representation; and the more abstract explanation of our attractor model which attributes choice persistence to a prevailing tendency to choose the SS or the LL option on an aggregate level representing the overall attractiveness of both options. To this end, we implemented pairs of trials into our paradigm that consisted of a prime trial and a target trial. Prime trials were designed such that they contained one clearly attractive option, e.g., the SS option. In the subsequent target trials, the features – value and distance - of the previously primed option (i.e. SS), were applied to the alternative option (i.e., LL). Hence, we transferred the features of one option (SS) in prime trials to the opposite option (i.e., LL) in the target trial. This feature repeating option (i.e., LL) was then paired with a new alternative option (i.e. SS; note that the opposite scheme was applied to trial pairs where the prime was favoring LL choices).

From a feature priming account, one could expect a repetitive choice of the option exhibiting the previously chosen features. Hence, in our paradigm feature priming predicts a switch of the chosen option from trial to trial, i.e., from SS to LL or from LL to SS, since features are transferred from one option in the prime trial to the opposite option in the target trial. In contrast, choice persistence at an integrated attractiveness level predicts that subjects stick to their choice of the previously chosen option – even though this option’s features would have changed. This would lead to a repetition of the chosen option, i.e., SS to SS or LL to LL. Importantly, the attractor model predicts that this effect would be present for the whole range of controlparameter c in which both options show some attractiveness – hence, two attractors exist. Hence we also slightly varied the relative attractiveness of the two options in the target trial.

An important prerequisite for this prime-target scheme was to precisely manipulate subjects’ choices. Hence, we first needed to estimate each subject’s overall level of discounting to later fit the presented options to these individual subjective values accordingly.

5.1 Methods

5.1.1 Subjects

33 students (26 female, mean age = 23.93) of the Technische Universität Dresden took part in the experiment. All subjects had normal or corrected to normal vision. They gave informed consent to the study. As in Experiment 2, they received a 3 € show-up fee and the money they collected within the experiment. After checking our manipulation in the prime trials, we had to exclude 6 subjects from subsequent analyses since they showed a reversed effect of the bias manipulation during prime trials and, hence, lacked a sufficient number of valid prime-target trial pairs for further analysis. Hence, we performed all subsequent analyses on 27 subjects (observed effects did not change qualitatively when the 6 additional subjects were included).

5.1.2 Setup and Design

The setup was similar to Experiment 3, with two modifications. First, the experiment now consisted of four blocks (8 minutes each) where the first block served to estimate the individual rate of discounting of each subject. The subsequent three blocks then implemented the paired trial scheme (i.e., combinations of prime and target trials) to test whether the repetition bias observed in experiments two and three was due to feature or to choice repetitions. Second, while each option’s reward value was again depicted explicitly through red numbers printed on coins of equal size, we now used a credit system ranging from 10 to 99 credits instead of 1 to 10 credits in experiments two and three. This allowed us to match the values presented in the prime trials as precisely as possible to the individual subjective values of each subject that were measured during the first block.

To estimate individual discounting in the first block, we independently varied the delay of the SS option (D _S= 2 or 8 fields) and the difference in delay between the SS and the LL option (D _L - D _S = 3 or 6 fields). The value of the LL option (V_L = 50 to 99 credits) was chosen randomly, the value of the SS option was then derived systematically by setting it to V_S = 20, 50, 70, 80, 85, 89, 93, 97% of V_L (Dshemuchadse, Scherbaum & Goschke, 2012). This resulted in 32 different types of trials. Each trial could be repeated 6 times, leading to a pool of 192 trials in the first block of the experiment.

In the subsequent three blocks, we used pairs of prime and target trials where the features of the option chosen in a given prime trial (i.e., its’ value and relative distance) were transferred to the opposite option in the subsequent target trial.

With regard to options’ delays, in prime trials, we varied the delay of the SS option (D _S= 2 or 8 fields) and the difference in delay between the SS and the LL option (D _L – D _S = 3 or 6 fields). In target trials, we varied the delay of the SS option (D _S = 2, 5, or 8 fields) and the distance to the LL option (D _L= 8 or 14 fields).

With regard to options’ values, we combined the individual discounting functions defining the subjective value SVas measured in block 1 with an additional bias SV _bias = 0.15 or -0.15, in favor of the SS or the LL option. In prime trials, this yielded values of the SS option (10 < V_S < 50) and the LL option (40 < V _L < 99), with V_S < V _L. In target trials, the value of the favored option of the prime trial was then used as the value of the opposite option and the value of the remaining option was determined by SV _biasto favor the SS (SV _bias= 0.15) option, no option (SV _bias= 0), or the LL option (SV _bias= - 0.15). This manipulation in the target trial served to create both, high conflict and less conflict trials. Hence, it allows us to investigate that the priming effect/choice persistence not only affects indifferent/high conflict decisions, but also decisions with an expected choice outcome.

This procedure resulted in 48 different combinations of prime and target trials, with 10 possible repetitions leading to a pool of 480 trials (see Appendix II and Table A1 for more details).

5.2 Results

On average, subjects completed 408 out of 480 possible pairs of prime and target trials (85.05%, SD = 10.47%).

To check for the effects of priming on target trials, we excluded all prime-target trial pairs where subjects did not choose the option favored by the biasing manipulation. For the remaining trials (65%), we checked whether subjects stayed with the option chosen in the prime trial or whether they stayed with the features chosen in the prime trial and hence, switched options. To this end, we performed an ANOVA with the factors Bias _prime (SS vs LL) and Bias _target(SS, none, LL) on the amount of choices of the SS option in target trials (Figure 9). This yielded a significant effect of Bias _prime (F(1,26) = 6.15, P < 0.05, η ²= 0.19) and of Bias _target (F(2,52) = 40.94, P< 0.001, η ²= 0.61), but no significant interaction (F(2,52) = 1.72, P = 0.19, η ²= 0.06). After choosing SS in the prime trial, choosing the SS option in the target trial was more likely (M = 48.59%, SD = 3.11%) than after choosing the LL option in the prime trial (M= 37.60%, SD = 0.34%). This result indicates that the current choice was influenced by a persisting bias of the previous choice on the option level and by the current attractiveness of an option, and that feature priming did not influence subjects’ choices.

Figure 9: Results of Experiment 4. Mean probability of choosing the SS option as a function of priming in the previous trial and relative attractiveness/dominance in the current trial. Dominance showed the expectable main effect, while priming an option shows the effect as predicted by the attractor model. In primed SS trials the priming trial successfully led to increased choices of the soon option. In primed LL trials, the priming trial successfully led to increased choices of the late option. Error bars denote standard errors.

5.3 Discussion

Experiment 4 provides evidence that choice persistence can be attributed to persistence on the level of the representation of choice options – and not to a mere persistence on the level of the representation of option features. Instead, current choices tend seem to be drawn towards the option chosen previously (SS or LL), even if this option’s features (i.e., its value and distance) have switched to the opposite option (from SS to LL or vice versa). This finding supports the assumption that our attractor model targets the correct level of representation with attractors representing the SS and the LL choice at an abstract, integrative level of representation instead of in terms of their features at conceptually lower levels of representation.

6 General discussion

In this article, we aimed to integrate a process-oriented attractor model into the outcome-based, psychometric modelling of delay discounting decisions. Through this integration, we opened up a window on the process dynamics leading to a final choice and the dynamics across consecutive choices. While these dynamics only recently came into the focus of research, they nevertheless must be studied in order to better understand both consistent decision making as well as decision failures and deviations from the average discounting model (Lempert & Phelps, 2016). We presented the attractor model for delay discounting (Scherbaum et al., 2008; see also Svyantek, Deshon & Siler, 1991; Townsend & Busemeyer, 1989) that is based on models of perceptual decision making (Hock et al., 2003; Noest et al., 2007; Tuller et al., 1994). From this model, we derived qualitative predictions regarding both the intra-trial and the inter-trial process dynamics underlying choice behavior and validated these predictions in four experiments based on a delay discounting game (Scherbaum et al., 2013): While Experiment 1 used mouse movement trajectories to provide evidence for the validity of the model’s predictions regarding intra-trial dynamics, experiments two and three demonstrated the importance to consider inter-trial dynamics by detecting hysteresis in intertemporal choice, that is, a persisting influence of previous on current choices. Finally, Experiment 4 showed that the model’s level of abstraction – representing the SS and the LL option as attractors – matches the phenomena of interest.

In the following, we will first evaluate the model and then continue with an evaluation of the empirical findings.

6.1 The attractor model of delay discounting

The attractor model represents the two possible choices and their attractiveness by two attractor basins of varying depth. We show, that this approach is fully compatible with discounting functions: While the latter merely define the configuration where the two options possess the same subjective value – as measured by equal final choice outcomes for the SS and the LL option – the attractor model additionally adds the stability of decision states – the depth of the attractors – and hence provides the dynamics leading to the final choice outcomes. Thereby, it offers a heuristic intuition about the process dynamics leading to a final choice (Duran & Dale, 2014; Frank, Richardson, Lopresti-Goodman & Turvey, 2009; van Rooij & Bongers, 2000; van Rooij et al., 2013). The model’s simplicity and abstractness supports theorizing without the demand of complex technical details (Onnis & Spivey, 2012). Nevertheless, a formalized implementation of the model is possible, and in Appendix I, we demonstrate that it leads to the exact same predictions as described above. Moreover, the formalized implementation illustrates not only that the attractor model is compatible with discounting curves but also that it builds a bridge to more complex models of decision making which are equivalent on a conceptual level: The attractor dynamics can be seen as an abstraction of the neural interaction dynamics within more complex neural networks (Onnis & Spivey, 2012) which have, in turn, been linked to the stability of firing patterns of neural assemblies representing the various options and their properties in the brain (Meyer-Lindenberg, Ziemann, Hajak, Cohen & Berman, 2002; Scherbaum et al., 2008). Specifically, the dynamics of the attractor model describe the dynamics that arise from interactive activation and competition networks (Rumelhart & McClelland, 1986) as they are applied in judgement and decision making, especially in the form of parallel constraint satisfaction networks models (Freeman & Ambady, 2011; Glöckner & Betsch, 2008). While the latter models allow for the description of the decision process on a fine-grained level, the attractor model provides a more abstract (and mathematically less complex) description of the dynamics of these processes while still allowing to predict and study phenomena like hysteresis which are typically associated with more complex dynamical system.

The simplicity of the attractor model is a feature that is shared with other recent models of delay discounting, i.e., drift diffusion models and linear ballistic accumulator models (Dai & Busemeyer, 2014; Milosavljevic, Malmaud, Huth, Koch & Rangel, 2010; e.g., Ratcliff & Smith, 2004; Rodriguez et al., 2014). Concerning the intra-trial dynamics, the attractor model is comparable to these models (Wang, 2008). However, it adds the non-linear dynamics which lead to hysteresis, which diffusion and linear accumulator models do not explain without additional assumptions. Therefore, the attractor model can be seen as an instrument to describe and integrate the behavioral dynamics of decision making on both the intra-trial and the inter-trial time-scale, thus, building a bridge between simple yet static discounting curves and dynamic yet complex neural process models.

6.2 Mouse movement deflections in delay discounting

The data on the intra-trial choice dynamics as measured by mouse movements matched the predictions of the model. First, we found greater mouse movement deflections in high conflict compared to low conflict trials, which matches previous findings on process dynamics in intertemporal choice (Dshemuchadse et al., 2012). Second, we found greater mouse movement deflections for low conflict trials in which the unattractive option was chosen compared to low conflict trials in which the attractive option was chosen. While the choice of the unattractive option is an anomaly for discounting-models, it is perfectly compatible when the attractor dynamics are taken into account: a slight bias at the start of trial suffices to lead the system state into the weaker attractor. Such an initial bias could be induced by noise or – as the other experiments suggest – by the choice in the previous trial leading to choice perseveration (compare Scherbaum et al., 2013).

6.3 Sequential effects in choices

Choice perseveration in sequences of decisions are best illustrated in their strongest manifestation as hysteresis as observed in Experiments 2 and 3. While perseverative choice patterns have been observed previously in studies focusing on value based decision making (Coulson & Nunn, 1999; Svyantek et al., 1991) or social perspective taking (Duran & Dale, 2014), to the best of our knowledge this is the first report of hysteresis in delay discounting. It is likely that earlier investigations have not observed this effect due to the classic paradigm (for example as used by Scholten & Read, 2010) employed in these studies where the explicit presentation of options’ attributes makes sequential manipulations obvious. In fact, this issue might even have led to opposite effects (Robles & Vargas, 2008) similarly to what has been found for perceptual decisions (so-called enhanced contrast due to repeated exposure, see Tuller et al., 1994). In contrast to the classic paradigm, our delay discounting game impeded the detection of sequences as it not only varied the delay (i.e., distance) of two consecutive options but also their positions on the x-y-plane. Importantly, we found hysteresis in two experiments with different option presentations, attesting to the robustness and validity of the finding.

This finding of hysteresis adds to the importance of complementing the outcome-based, psychophysical approach to delay discounting (Takahashi, Oono & Radford, 2008) with a more process-oriented approach that not only focuses on the outcome of the decision process but also takes into account how the process unfolds over time. Such a process-oriented approach allows to integrate different features of the choice situation and contextual factors influencing the final decision – which have led to disconcertion in the research community (Lempert & Phelps, 2016) and a vast multitude of different discounting models (Doyle, 2013) – into a single theoretical framework (Scholten & Read, 2010; Townsend & Busemeyer, 1995).

6.4 Limitations

6.4.1 Generalizability of the model and the evidence from the discounting game

With regard to our experimental approach, the advantages inherent to our discounting game come at the cost of the question whether or not the attractor model and the evidence for it presented here are generalizable to other, more standard intertemporal choice tasks and to intertemporal decisions in general. Classic intertemporal choice tasks present subjects with a number of hypothetical or partly factual intertemporal decisions between sooner/smaller and later/larger options with values and delays presented in writing and with ranges varying from days (e.g., Kirby, Petry & Bickel, 1999) to years (e.g., Read, Frederick, Orsel & Rahman, 2005). Our paradigm is very similar in that we ask subjects to decide between nearer/smaller and farer/larger options, with distances and values clearly visible to the subject and distances reflecting the time to reach the options and hence representing the delays of the options. Three differences between the classical and our paradigm are apparent, however. First, the values and times used in our paradigm are on a smaller scale than in most delay discounting tasks. Second, the values and times are presented implicitly (as the size and distance of coins) while they are typically presented explicitly (as amounts of money and delays in days, weeks or months) in classic intertemporal choice tasks. Third, time is limited in our paradigm and it hence poses an objective constraint on how to spend time by reaching an option. This is in contrast to classical intertemporal choices where the value of time is per se a subjective factor.

In response to these concerns, we suggest that the paradigm used in this set of studies is indeed a valid instrument to study intertemporal decision making. First, with regard to the small and accumulative amounts of money we used as rewards in our task, the discounting game joins a wide range of delay discounting paradigms employing various types of gains ranging from primary rewards such as food or drinks (McClure, Ericson, Laibson, Loewenstein & Cohen, 2007), one-out-of-many-choices-rewards (Ballard & Knutson, 2009; McClure, Laibson, Loewenstein & Cohen, 2004), or hypothetical rewards (Green, Fristoe & Myerson, 1994; Robles & Vargas, 2008; Takahashi et al., 2008). In this diverse collection, we see our approach as reasonable in so far as it implements real rewards that the subject receives in total by the end of the experiment. The comparatively small temporal frame the discounting game operates on would indeed be problematic had this resulted in our subjects not discounting at all. However, our original study indicated that the discounting behavior observed in the discounting game is similar to the behaviour observed in more standard paradigms of delay discounting. For example, we showed that subjects completing the delay discounting computer game also weighed delays and reward attributes for each option which was reflected in a bias towards sooner/smaller choices (Scherbaum et al., 2013). This is also reflected in the fact that disadvantageous choices of the SS option show significantly lower distances between the avatar and the SS option than advantageous ones. This finding indicates that also in the discounting game subjects are seduced by immediate or very short-term options (Kalenscher & Pennartz, 2008). Of course, it is out of question that certain timescale-dependent effects (Soman et al., 2005) might not be captured within the short temporal horizon of the discounting, but such effects were not in the focus of the current studies. Nevertheless, the general comparability of the weighing of both, the small values and time intervals in the discounting game and the larger values and intervals in more classical tasks is also reflected in a high correlation of r = 0.64 between the k-values measured in our discounting game and a commonly used, paper pencil intertemporal choice questionnaire (Kirby et al., 1999) that we observed in our previous study (Scherbaum et al., 2013).

Second, concerning the format of presentation of options’ values and times in terms of sizes and distances, it is indeed well known that different formats of presentation can influence the decision process, as is evident e.g., in the so called date-delay effect (Read et al., 2005; Scherbaum et al., 2012). However, we tried to rule out at least some of the concerns resulting from this observation by comparing the effects of presenting the two options’ values implicitly (as the size of the coins) vs. explicitly (as written numbers on same-sized coins). Having found no difference in discounting behaviour and hysteresis between these two modes of presentation makes us confident that these effects are robust with respect to the mode of presentation.

Third, the limited time frame that we used may have encouraged subjects to use certain strategies like always choosing the SS option in order to make as many choices as possible. As this kind of strategic behaviour would stop subjects from making deliberate, considerate choices based on the careful weighing of the two options’ values and delays, this would render the decisions made in the discounting game incomparable to classical intertemporal decisions. Crucially, the choice patterns we observed stand against this possibility. We designed the combination of options presented to our subjects during the discounting game in such a way that persistently choosing one (e.g., the SS) option led to suboptimal results whereas a mixed, trial-dependent strategy resulted in the highest overall gain (for further details, please see the methods section of experiment 1). As the choice patterns we observed empirically closely resembled this optimal behaviour, we can conclude that subjects did not rigidly stick to one or the other option throughout the experiment but that they carefully weighed both options’ values and delays on a trial-by-trial basis instead.

Over and above these rather technical points illustrating the comparability of our dynamic and more classic measures of intertemporal decision making, we have gathered recent evidence that behaviour in the discounting game is linked to a problematic behaviour in the real world that has been consistently linked to impairments in delay discounting in the past: drug abuse. In this study (Scherbaum, Haber, Morley, Underhill & Moustafa, under review) we had a group of heroin addicts and a group of matched controls complete the discounting game. Consistent with previous evidence linking classic measures of intertemporal choice to addiction (Kirby et al., 1999), we found increased discounting for heroin addicts as compared to the control group, thus, providing further evidence for the validity of our paradigm.

All of these arguments combined make us confident that the informative value of the work presented here – the attractor model and the behavioural patterns we observed in support of it — are not restricted to the novel paradigm we employed, but that they are also valid and applicable to more classical investigations of intertemporal choices. What is more, the paradigm used here has a number of advantages compared with more classical intertemporal choice tasks. First, the it allows us to study phenomena that are difficult to study in the classical paradigm: While the sequential manipulation of delays that is necessary to study hysteresis is quite easy to see through in classic versions of the intertemporal choice paradigm, the possibility to rotate the options presented on a trial by trial basis makes these types of sequential manipulations much harder to detect in the discounting game. Second, whereas classical tasks are based on hypothetical choices or on partially realized choices (often, one trial is selected by a lottery), every trial counts for our subjects, thus, motivating them to stay focussed and on task throughout the experimental session. Third and last, but not least, the paradigm is built around an immersive, playful environment and subjects report that they were so engaged by the game that they almost forgot about participating in an experiment.

6.4.2 Level of generality of the attractor model

The attractor model builds a bridge from discounting functions to more complex models of delay discounting, for example accumulation models with different timescales (Scherbaum et al., 2012), parallel constraint satisfaction networks (Glöckner & Betsch, 2008), or decision field models including several transformations (Dai & Busemeyer, 2014). The compatibility of the attractor model with more complex implementations of the decision process is illustrated further by the neural network implementation of the attractor model in Appendix I. Importantly, this implementation describes the decision dynamics on an aggregate level of detail. In particular, this model does not give any indication of a number of details of the decision process that are a matter of debates in the literature, for example, whether the attractiveness of an option is derived by calculating the subjective value of each option first and then performing alternative-wise comparisons of both options (Dai & Busemeyer, 2014; Mazur, 1987; Samuelson, 1937) or by attribute-wise comparisons of both options’ delays and values (Scholten & Read, 2010). However, elucidating these open questions is beyond the scope of our attractor model. Instead, the value of the abstract model presented here is that it provides an intuitive and easy-to-work-with instrument for theorizing about the intra- and inter-trial dynamics of the decision process. Of course, the merit of a model at this level hinges on additional empirical tests of its validity of the sort we have reported here.

6.5 Conclusion and prospects for research on delay discounting and decision making

The model and paradigm presented here led us to predictions and findings of behavioral patterns that could not be expected from the discrete models and behavioral methods typically used in intertemporal choice research. Instead, the predictions we tested in this article are typical patterns produced by dynamic systems which can be readily described with attractor models. Hence, our application of the attractor model to delay discounting (as initially sketched in Scherbaum et al., 2008) corroborates other applications of dynamic systems theory to value-based decision making in general (Coulson & Nunn, 1999; Koop & Johnson, 2011; McKinstry, Dale & Spivey, 2008; Oullier & Kelso, 2006; Oullier, Kirman & Kelso, 2008; Roe, Busemeyer & Townsend, 2001; van Rooij & Bongers, 2000; van Rooij et al., 2013; Svyantek et al., 1991; Townsend & Busemeyer, 1995). In their combination, this growing body of work showcases that opening a window into the processes dynamics underlying value-based decisions is an important and fruitful endeavor. Therefore, the value of the dynamic approach presented here goes far beyond our findings for delay discounting and that it offers prospects for both the theoretical discussion and the empirical investigation of delay discounting and value-based decisions in general.

On a more abstract, theoretical level, our approach extends the focus of decision research from discounting curves and decision outcomes to dynamic properties of the decision process unfolding both within and across trials (as asked for e.g., in Townsend & Busemeyer, 1995). Within trials, deviations from the ideal choice line are a sensitive measure (Frisch, Dshemuchadse, Görner, Goschke & Scherbaum, 2015; Scherbaum, Gottschalk, Dshemuchadse & Fischer, 2015) that can provide information about the overall stability of decision states. Similarly, the strength of the hysteresis effect is an indicator of the overall stability of the attractor layout. Considering these process dynamics could hence provide explanations of dysfunctional behavior, e.g., too stable attractors in obsessive-compulsive disorders (Rolls, Loh & Deco, 2008) or relatively flat and hence volatile attractor layouts in impulsive decision makers (Kräplin et al., 2014, 2015; Wittmann & Paulus, 2007).

For empirical investigations, the presented paradigm offers an immersive way to study behavioral choice patterns both within trials (i.e., via mouse tracking) and across trials (e.g., by allowing unobtrusive sequential manipulations). Such measures could uncover differences between populations or conditions that might be occluded when only measuring choice outcomes or focusing on the outcome of (averaged) single choices without considering sequential effects (see Spencer, Smith & Thelen, 2001 for a comparable approach to the study of development of spatial memory). For example, changes in the hysteresis effect could be indicative of developmental differences in the attractor layout of younger and older populations (Li, Lindenberger & Sikström, 2001) which would be occluded when only looking at discounting curves without considering sequential effects.

In summary, the combination of an attractor model with a dynamic behavioral paradigm integrates process dynamics into discounting functions and thereby supports the understanding of decision failures and deviations from static descriptions. The way how we make delay discounting decisions necessarily tells more than the final result of this process.