Risky choice frames shift the structure and emotional valence of internal arguments: A query theory account of the unusual disease problem

Abstract

We examine a Query Theory account of risky choice framing effects — when risky choices are framed as a gain, people are generally risky averse but, when an equivalent choice is framed as a loss, people are risk seeking. Consistent with Query Theory, frames affected the structure of participants’ arguments: gain frame participants listed arguments favoring the certain option earlier and more often than loss frame participants. These argumentative shifts mediated framing effects; manipulating participants initial arguments attenuated them. While emotions, as measured by PANAS, were related to frames but not related to choices, an exploratory text analysis of the affective valence of arguments was related to both. Compared to loss-frame participants, gain-frame participants expressed more positive sentiment towards the certain option than the risky option. This relative-sentiment index predicted choices by itself but not when included with structure of arguments. Further, manipulated initial arguments did not significantly affect participant’s relative sentiment. Prior to changing choices, risky choice frames alter both the structure and emotional valence of participants’ internal arguments.

1 Introduction

We argue that when evaluating a risky decision, like choosing to settle a lawsuit or negotiating a contract, a person considers a series of arguments in favor of and against the options. Specifically, we examine how the structure of these arguments is affected by how the risky choice is framed — presenting equivalent risky decisions framed as either gains or losses. Consistent with Query Theory, we posit that risky choice frames differentially shift participants’ arguments to favor one option. When framed as a gain, a certain option attracts participants arguments favoring it, perhaps because it has more positive emotional valence than the risky alternative. But when framed as a loss, the certain option repels arguments favoring it because it has a more negative emotional valence than the risky alternative. According to Query Theory (Johnson, Häubl & Keinan, 2007; Weber et al., 2007), initial arguments in favor of an option underlies value construction: the initially favored option crowds out arguments for the other option, leading people to generate more arguments in favor of the early favorite, thus choosing it more often.

In our studies, we find that the structure of arguments is related to choice: risky choice frames affect which option receives arguments in its favor, which mediates the effect of frame on choice. In addition, we find that emotions, as measured by PANAS (Positive And Negative Affect Schedule), are predicted by frame but not predictive of choice. Given this unexpected result, we performed an exploratory sentiment analysis, which measured the affective valence of participant’s arguments. Unlike PANAS, this sentiment index was predicted by frame and predictive of choices. These results suggest that, on the way to changing choices, risky choice frames change both the structure of participants’ internal arguments their arguments’ emotional valence.

Risky choice framing effects appear in a variety of domains: financial decisions (Levin, Schneider & Gaeth, 1998), medical decisions by both patients and doctors (McNeil, Pauker, Sox & Tversky, 1982), and negotiations (Neale & Bazerman, 1985). Given the impact of risky choice frames in disparate domains, it is important to understand the underlying psychological mechanisms to debias decisions.

2 Literature Review

Risky choice frames can change what people chose (Levin et al., 1998). When risky choices are framed as gains, participants select the certain option more often than when risky choices are framed as losses (Kahneman & Tversky, 1979; Tversky & Kahneman, 1981), even when both frames are logically equivalent. For instance, in the Unusual Disease Problem,¹ participants chose between two public health programs for the treatment of a disease: a certain option — e.g., where 200 of the 600 people will live — and a risky option — e.g., where there is a 1/3 probability that all 600 will live and a 2/3 probability that no one will live. When the treatment outcomes are framed as gains — e.g., 200 people will be saved — participants choose the certain option more often than when logically equivalent treatment outcomes are framed as losses — e.g., 400 people will die. Given that frames alter decisions, we can infer that they also alter participants’ cognitive and/or emotional processes preceding a decision.

Risky choice framing effects were first explained by the Prospect Theory value function, which predicts risk aversion for gains but risk seeking for losses (Kahneman & Tversky, 1979). That is, for gains people prefer a sure gain to an equal expected value risky gain, but for losses people prefer a risky option to an equal expected value sure loss. Applied to the Unusual Disease Problem, in the gain frame risk aversion predicts choice of the certain option, but in the loss frame risk seeking predicts choice of the risky option.

Different risk preferences for gains and losses, however, seems inconsistent with choice patterns in variants of the Unusual Disease Problem. For instance, the framing effect diminishes when the risky options are truncated, e.g., participants are shown “a 1/3 probability that all 600 will be saved” but not a “2/3 probability that no one will be saved”. While Prospect Theory does not explain this result, other theories do (Kühberger, 1998; Kühberger & Gradl, 2013; Mandel, 2014; Rahimi-Golkhandan, Garavito, Reyna-Brainerd & Reyna, 2017; Tombu & Mandel, 2015). A widely cited encoding account of the Unusual Disease Problem — Fuzzy Trace Theory (Rahimi-Golkhandan et al., 2017) — suggests that people encode only the gist of the options. In the gain frame, the gist of the certain option is “some people are saved” while the gist of the un-truncated risky option is “some people are saved, or no one is saved.” When comparing these two gists, “some people are saved” is better than “some people are saved, or no one is saved”, making the certain option more desirable. In the loss frame, however, the gist of the certain option — “some people will die” — is less desirable than the gist of the risky option — “some people will die or no one will die” (Reyna & Brainerd, 1991). Fuzzy Trace Theory explains the effect truncating risky options by people encoding a different gist leading to different choices. Changing the gist of certain option also changes choices (Kühberger, 1995). Risky choice framing effects depend on how the options are described.

While Fuzzy Trace Theory and other theories explain choices on variants of the Unusual Disease Problem, we argue that there are multiple distinguishing features of Query Theory that make it an attractive explanation. First, it explains a broad set framing effects in other domains — attribute framing (Hardisty, Johnson & Weber, 2010), the endowment effect (Johnson, Häubl & Keinan, 2007), and the accelerate/delay effect in intertemporal choice (Weber et al., 2007) — making it a parsimonious explanation. Second, it allows for individual differences in framing effects: labeling an environmentally friendly option as either a tax or an offset affected Republicans’ attention but not Democrats’ (Hardisty, Johnson & Weber, 2010). Third, unlike other explanations of the Unusual Disease Problem, it makes novel predictions about how shifting initial the order of arguments will affect choice. Fourth, our measure of participants’ internal argumentative processes — thought listings — provides, via natural language processing, a granular measure of the emotional valence of participants’ arguments.

Query Theory assumes people evaluate options by sequential queries of memory, retrieving arguments in favor of one option then arguments in favor of the other option. Importantly, the option with early arguments in its favor is more likely to be chosen. Initially retrieving arguments in favor of an option interferes with the subsequent retrieval of arguments in favor of the other option. Differences in argument retrieval are related to choice: if someone retrieves arguments in favor of one program earlier, there will be more arguments for choosing it, making it will more likely be chosen. Therefore, a key question in Query Theory is: which option has initial arguments in its favor. The answer depends on the problem (Weber & Johnson, 2009). For instance, the status quo affects thought order (Johnson et al., 2007; Weber et al., 2007). When making an intertemporal choice where the immediate option was the status quo, people queried arguments for taking the immediate option earlier than the delayed option. Further, arguments in favor of the status quo were more accessible: participants were faster to recognize queries, which were consistent with the status quo than thoughts inconsistent with the status quo. Which option has initial arguments in its favor varies based on individual differences: labeling a carbon user fee as an offset or a tax shifted the structure of Republicans’ arguments but not Democrats’ (Hardisty, Johnson & Weber, 2010).

Given that the Unusual Disease Problem does not have an obvious status quo and does not alter attribute labels, we hypothesize that another element of the options drives initial arguments: the emotional valence of the choice options (Gonzalez, Dana, Koshino & Just, 2005; Tombu & Mandel, 2015; Willemsen, Böckenholt & Johnson, 2011). In the gain frame, the certain option (“saved” has a positive valence) has a more positive emotional valence than the risky option (the positive valence of “saved” is negated by “no one”). Therefore, the certain option attracts participants initial arguments. But in the loss frame the certain option (“die” has a negative valence) has more negative emotional valence than the risky option (“die” is negated by “no one”) meaning arguments should shift towards the risky option (Kühberger & Gradl, 2013; Tombu & Mandel, 2015; Wallin, Paradis & Katsikopoulos, 2016).

Further, risky choice framing effects are related to specific emotions participants are experiencing. For instance, the emotions experienced by participants moderated the effect of frames: participants with elevated levels of distress were more likely to show risky choice framing effects compared to those who were experiencing low levels of distress (Druckman & McDermott, 2008). Manipulated emotions also related to risky choice framing effects (Druckman & McDermott, 2008). People manipulated to feel distressed showed a risky choice framing effect but the effect was attenuated in those manipulated to feel enthusiasm.

Other work suggests that manipulating emotions may also manipulate which option is favored by early arguments. For instance, sadness and its associated sense of loss triggers an implicit goal of reward replacement (Lerner, Small & Loewenstein, 2004). This implicit goal relates to intertemporal choices between monetary amounts received at different points in time. Participants who watched a sad movie clip made more impatient intertemporal choices than participants who watched a neutral or a disgusting movie clip (Lerner, Li & Weber, 2013). Consistent with participants’ emotions shifting argument order, participants manipulated to experience sadness were more likely to list arguments in favor of the immediate reward option than those manipulated to experience disgust or neutral-feelings. That is, emotions changed choices by changing which option received favorable initial arguments.

Instead of focusing on manipulated emotions unrelated to the choice, we focus on observing the emotions that result from risky choice frames. This shifted focus is due to the fact that we hypothesize that initial arguments in the UDP are due to the emotional valence of the choices. We measure the effect of frames on emotions in two ways (Watson, Clark & Tellegen, 1988): First, using items from the PANAS (Watson, Clark & Tellegen, 1988) that relate to gambling behavior (Raghunathan & Pham, 1999) and risky choice framing effects (Druckman & McDermott, 2008).

And second, we measured the valence of participant’s arguments via exploratory text analysis. Using natural language, text analyses predict movie ratings from review text, depressive tendencies in patients from their essays, and investing behavior from twitter activity (Chatterjee & Perrizo, 2016; Da, Engelberg & Gao, 2015; Rude, Gortner & Pennebaker, 2004; Turney, 2002). Relatedly, the perceived risk of an activity is predicted by its semantic relatedness to emotional words, e.g., the more semantically related an activity is to fearful words the higher its perceived risk (Bhatia, 2018). Measuring only the emotional valence of participant’s thoughts misses out on multiple emotional dimensions, e.g., how certain, how in control, and how much responsibility people feel (Lerner, et al., 2015). However, emotional valence, by itself, is related to risky choice framing effects (Tombu & Mandel, 2015; Wallin et al., 2016). For instance, framing effects were attenuated, holding the risky choice constant, when the gain framed certain option had a negative valence and the loss framed certain option had a positive valence (Tombu & Mandel, 2015). While the emotional valence of participant’s thoughts is only a single dimension of emotions, prior research suggests it captures choice relevant information.

In sum, prior work on framing effects suggests that risky choice frames affect the structure and emotional valence of participants arguments. Our work has three goals: first and foremost, to test a Query Theory account of risky choice framing effects; second, to examine how risky choice frames affect participant’s reported emotions; and third, to present an exploratory text analysis of the content of participant’s arguments. Studies 1 and 2 showed that early arguments favoring an option, as predicted by Query Theory, were related to risky choices. Studies 3 and 4 replicated Studies 1 and 2 in a more vivid scenario that attempted to increase participant’s emotional reactions to the stimuli. Further, these studies showed that a conventional measure of specific emotions — items from the PANAS scale — were altered by gain/loss framing but were not related to choices or the structure of participants arguments. However, a series of exploratory analyses showed that the sentiment of participants arguments was altered by frames and related to both choices and the structure of participants arguments — e.g., which option was the early favorite and which option had more thoughts favoring it. We jointly analyzed emotional valence and the structure of participants arguments in a path analysis. This model suggested that risky choice frames alter both the sentiment of participant’s arguments and the argument’s structure but that the sentiment of arguments accounts for no additional variance beyond the structure of arguments. Another analysis suggested that frames affected the emotional valence of participants thoughts before affecting the structure of their argument.

3 Studies

3.1 Hypotheses

In Studies 1 and 2, we focused on how of risky choice frames affect participants’ arguments and subsequent choices. That is, these studies test if risky choice framing effects can be accounted for by Query Theory. Prior Query Theory studies showed that, while deliberating their decision, certain aspects of an option lead participants to argue in its favor earlier and more often (Johnson et al., 2007; Weber et al., 2007). In the Unusual Disease problem, we hypothesize that the certain option is initially considered in the gain frame (200 people saved) because certain gains are attractive, but not in the loss frame (400 people die) because certain losses are aversive making the risky prospect to become the initial leader.

Hypothesis 1:

Frames shift the structure of participant’s arguments.

The framing of the problem — as a gain or a loss — alters the order in which people consider options. Specifically, in the gain frame participants should list arguments favoring the certain option — those that make a case for the certain option being good or the risky option being bad — earlier and more often relative to the risky option than participants in the loss frame.

Whereas prior work demonstrates that risky choice frames direct attention in the form of acquiring information about options, we investigate how frames direct arguments participants make while deliberating (Willemsen et al., 2011).

Consistent with prior Query Theory papers we hypothesize that frames affect arguments and choices.

Hypothesis 2:

The structure of participant’s arguments predicts choices and mediates the effect of frames on choice.

Framing alters initial arguments and the number of arguments, which then alter choice. The effect of frame on choice will be mediated by differences in the thought listings. Study 1 tested hypotheses 1 and 2.

3.2 Study 1: A Query Theory Account of the Unusual Disease Problem

3.2.1 Method

Participants

We recruited 180 participants (93 female), who ranged in age from 19 to 65 (M = 33.7, SD = 10.84), 143 reported an education level of some college or higher, from Amazon Mechanical Turk, an online labor pool. Participants received $1 for completing the study. To ensure participants read the instructions and prompts carefully, we included an attention check question. Participants who failed the attention check (10 participants) were not included in the final analysis.

Procedure

Participants first listed thoughts about purchasing a vehicle to familiarize themselves with the thought listing interface (Hardisty et al., 2010; Johnson et al., 2007; Weber et al., 2007). Next, participants read either a gain or loss framed version of the Unusual Disease problem. As opposed to having 600 expected fatalities, we changed this to 720 to make the proportions more difficult to calculate.²

The exact wording used for the gain and loss frames follows:

Imagine that the U.S. is preparing for the outbreak of an unusual Asian Disease, which is expected to kill 720 people. Two alternative programs to combat the disease have been proposed. Assume that the exact scientific estimate of the consequences of the programs are as follows (participants did not see the text in square brackets):

[Gain Frame:]

If Program A is adopted, 240 people will be saved. [Certain Option]

If Program B is adopted, there is 1/3 probability that 720 people will be saved, and 2/3 probability that no people will be saved. [Risky Option]

[Loss Frame:]

If Program A is adopted 480 people will die. [Certain Option]

If Program B is adopted there is 1/3 probability that nobody will die, and 2/3 probability that 720 people will die. [Risky Option]

After participants read the Unusual Disease problem, they were instructed: “Before you indicate your preference for these programs, please tell us everything you are thinking of as you consider this decision between Program A and Program B. We would like you to list any thoughts, both positive and negative, that you might have about this decision. We will ask you to enter your thoughts one at a time.” On average, participants listed 3.94 thoughts (SD = 1.96). After their thought listing, participants choose one of the two programs: Program A (the certain option) or Program B (the risky option). Next, participants were shown their previously written thoughts, one at a time, and asked to place them in one of five categories: “An advantage of Program A”, “A disadvantage of Program A”, “An advantage of Program B”, “A disadvantage of Program B”, or “neither”. Finally, participants provided demographic information.

3.2.2 Results

Choices.

Replicating prior Unusual Disease studies, gain-frame participants were more likely to choose the certain option (71%) than participants in the loss frame (55%) (β = 0.69, SE = .33, p = 03).³

Thought Listings.

As in prior Query Theory studies, we combined thoughts participants rated as “An advantage of Program A”, and “A disadvantage of Program B” into pro-certain thoughts. Similarly, we combined thoughts participants rated as “An advantage of Program B”, and “A disadvantage of Program A” into pro-risky thoughts. (Thoughts participants rated as “neither” were not used, as they are not choice relevant arguments.)

As with prior studies, we characterized the structure of participants’ thought listings using the Standardized Median Rank Difference (SMRD) and the Balance of Thoughts (Johnson et al., 2007). SMRD was calculated as 2(MR_r−MR_c)/n, where MR_c is the mean rank pro-certain thoughts and MR_r is the mean rank of pro-risky thoughts and n is the total number of pro-certain and pro-risky thoughts. SMRD ranges from +1 (all pro-certain thoughts listed before all pro-risky thoughts) to −1 (pro-certain thoughts listed after all pro-risky thoughts). A SMRD of zero corresponds to pro-risky and pro-certain thoughts having the same median rank. We used SMRD instead of other measures — e.g., which option a participant’s first thought argued in favor of — because it has been used in all prior Query Theory papers and gives a broader view of the structure of participants arguments. Balance of Thoughts was simply the difference between the number of pro-certain and pro-risky thoughts. For example, if a participant listed 2 more pro-certain thoughts than pro-risky thoughts, then their Balance of Thoughts was +2. For the twelve participants rated all thoughts as only belonging to the “neither” category we set both SMRD and Balance of Thoughts to 0.

Consistent with Hypothesis 1 — Frames shift the structure of participant’s arguments — participants had a higher SMRD — i.e., they listed pro-certain arguments earlier — in the gain frame (M = .40) than in the loss frame (M = .14) (t(168) = 1.96, p = .052, Cohen’s D = 0.30). Participants in the gain frame had a significantly more positive Balance of Thoughts (M = 1.18) than participants in the loss frame (M = 0.41) (t(159.12) = 2.5, p = .013, Cohen’s D = 0.39). Further, as predicted by Query Theory, SMRD and Balance of Thoughts were correlated (r = .59, t(168) = 9.53, p < .001).

Mediation.

Hypothesis 2 predicts that the structure of arguments mediates the effect of frame on choice. To simplify our mediation and prediction analysis, we created a Structure of Thoughts index that z-scored both SMRD and Balance of Thoughts and took their average. Structure of Thoughts — which combines rank and order information — predicted choice (β = 1.56, SE = .25, p < .001).⁴

To fit the mediation model, we used the lavaan R package (Rosseel, 2012) with 5000 bootstrapped iterations. We used a probit model because the dependent variable, choice, is binary (Imai, Keele & Tingley, 2010).

Figure 1A shows the results of the mediation model. Consistent with hypothesis 2, there was a significant effect of frame on Structure of Thoughts (β = 0.33, SE = 0.13, p = .013). Also, the effect of Structure of Thoughts on choice was significant (β = 0.30, SE = 0.032, p < .001). After including the indirect path, the effect of frame was not significant (β = 0.06, SE = 0.06, p = .33). Importantly, the indirect effect from frame through Structure of Thoughts to choice was significant and positive (β = 0.098, SE = 0.039, p = .01). The significant indirect path suggests that the structure of participant’s arguments mediates risky choice framing effects.

Figure 1: Path models predicting choice by frame for Study 1 (Panel A) and the vivid stimuli of Study 3 (Panel B). Direct effects without mediator are in parentheses. ^* p < .05 ^*** p < .001.

3.3 Study 2: Manipulated Thought Order Changes Choices

While Study 1 established that Query Theory processes mediate risky choice framing effects, it did not manipulate initial arguments. Study 2 directly tests hypothesis 3.

Hypothesis 3:

Manipulating the order in which participants attend to arguments will attenuate framing effects.

Specifically, when participants are forced to consider the options in an unnatural order — in the gain frame considering the risky option first, in the loss frame considering the certain option first — the effect of frame on choice will be attenuated. Consistent with prior studies, we predicted attenuation not reversal of framing effects because the frame induced natural order is difficult for participants to suppress and therefore still operates (Johnson et al., 2007; Weber et al., 2007).

3.3.1 Method

Participants

We recruited 584 participants (263 female), ranging between the ages of 18 and 72, (M = 33.26, SD = 10.46) with 124 having an education level of at least some college from Amazon Mechanical Turk. An attention check question removed 50 participants.

Procedure

Participants saw either the gain-framed or loss-framed choice and were instructed to first generate thoughts supporting the certain option followed by thoughts supporting the risky option, or vice versa. Otherwise the materials and procedures were identical to Study 1. Based on the results of Study 1, we categorized listing the arguments supporting choice of the certain first as the “natural” order for the gain frame and the “unnatural” order for the loss frame. In contrast, we considered listing arguments supporting the risky option first as the “natural” order for the loss frame and the “unnatural” order for the gain frame. That is, participants saw one of four between-subject conditions (2 frames by 2 argument-orders).

3.3.2 Results

To test the effect of manipulated argument order, we performed logistic regression with choice predicted by frame, argument listing order (natural or unnatural), and their interaction. (We used contrast coding for this analysis).

We replicated the standard risky choice framing effect. Gain-frame participants were more likely to choose the certain option (71.4%) than loss-frame participants (42.6%) (β = 0.61, SE = .09, p < .001). Analyses of simple effects showed that the effect of frame was significant both in the natural order (β = 1.76, SE = 0.27, p < .0001) and unnatural order (β = 0.70, SE = 0.25, p = 006). As shown in Figure 2A, however, there was a significant interaction: the effect of frame was larger in the natural order than in the unnatural order (β = 0.26, SE = 0.09, p = .004). Consistent with Hypothesis 3 and prior Query Theory studies, the effect of frame on choice was attenuated when participants listed arguments in an unnatural order.

Figure 2: Percentage of certain option choices for the natural and unnatural thought listing conditions in Study 2 (A) and Study 4 (B). Error-bars are 95% Confidence Intervals.

Because the natural order in the gain frame and the unnatural order in the loss frame both involve listing pro-certain thoughts first and the unnatural gain frame and natural loss frame both involve listing pro-risky thoughts first, we expected no main effect of unnatural versus natural thought order. As expected, there was no main effect of thought order on choice (β = 0.04, SE = .09, p = .66).

3.4 Study 3: Emotional Influences on Framing

Studies 1 and 2 demonstrated that the structure of participant’s arguments relates to framing effects; however, both did not explore how emotions relate to choices (e.g., Druckman & McDermott 2008) or the structure of arguments. Study 3 tested whether experienced emotions — as measured by the PANAS scale — related to framing effects.

Additionally, we created a new, more vivid, version of the Unusual Disease Problem. While prior work has manipulated the emotional valence of the two choice options by changing their description — in the gain frame adding “400 lives will not be saved” to the certain option — we attempted to increase participants’ overall level of emotional engagement of the scenario. Prior work suggests that people with reduced emotional reactions display smaller risky choice framing effects than people with normal emotional reactions (De Martino, Harrison, Knafo, Bird & Dolan, 2008). Given this result, we theorized that increasing participants’ emotional reactions via more vivid stimuli would enhance the effect of frames. As shown in Figure 3, we increased vividness and emotional engagement in three ways. First, we changed “unusual Asian Disease” to “Ebola”, which was salient due to a recent, at the time of the study, outbreak of Ebola in West Africa. Second, we indicated that the individuals infected by the virus were children. Third, we added photos: the gain frame showed a child recovering in a hospital bed, and the loss frame a child’s coffin being lowered into the ground. (While this may seem like a confound, the decision maker can logically infer in the gain frame that 480 children will die — and that there will be coffins — even though her attention — either verbally or by visual — is being focused on the children that are saved.)

Figure 3: Vivid, emotionally engaging gain-framed risky choice (Left) and loss-framed risky choice (Right)

3.4.1 Method

Participants

We recruited 258 (129 female) respondents, ranging between 18 and 72 years old (M = 34.77, SD = 10.86) with 163 reporting an education level of some college or higher, from Amazon Mechanical Turk. An attention check question removed 13 participants.

Procedure

The procedure was nearly identical to Study 1. We wanted to assess the emotional impact of the frame, so we measured the change in participants’ emotions using a subset of a modified version of the PANAS scale, as used by Lerner et al. (2013). Participants rated on a scale from 0 (Do not feel even the slightest bit) to 8 (Feel the emotion more strongly than ever) each of the following emotions both before and after making their decision: Positive Emotions (Amused, Cheerful, Happy); Neutral Emotions (Neutral, Unemotional); and negative emotions (Afraid, Angry, Bored, Depressed, Disgusted, Fearful, Furious, Gloomy, Indifferent, Mad, Nauseated, Nervous, Repulsed, Sad). Having a measure of these emotions both prior to and after making the choice allowed us to test if the two frames of the vivid Unusual Disease Problem differentially changed participants’ emotions. That is, we were concerned not with how participants were feeling prior to making the decision — incidental emotions — but were concerned with how risky choice frames shifted the choice relevant emotions — integral emotions (Lerner, Li, Valdesolo & Kassam, 2015). Integral emotions are important for the Unusual Disease Problem because, as we argued earlier, the valence of the options may focus participant’s attention.

Four participants who passed the attention check did not rate emotions both prior to and after their choice were removed from these analyses.

3.4.2 Results

Choices

As in Studies 1 and 2, the percentage of participants who chose the certain option was higher in the gain frame (M = 74%) than in the loss frame (M = 44%) (β = 1.25, SE = .27, p < .001).

Thought Listing

Consistent with Hypothesis 1 and Study 1, participants in the gain frame had a higher SMRD (M = .45) than in the loss frame (M = .01) (t(241.81) = 4.24, p < .001, Cohen’s D = 0.54). That is, they listed thoughts about the certain option earlier in the gain frame. Participants also listed significantly more thoughts in favor of the certain option relative to the risky option in the gain frame (M = 1.26) than in the loss frame (M = 0.15) (t(242.3) = 4.09, p < .001, Cohen’s D = 0.52). SMRD and Balance of Thoughts were again correlated (r = .61, t(243) = 12.08, p < .001).

Mediation

Consistent with the Study 1 results, the Structure of Thoughts index — the combination of SMRD and Balance of Thoughts — significantly predicted choice (β = 1.74, SE = 0.23, p < .001).

Consistent with Hypothesis 2, the results of the Study 1 mediation replicated. As seen in Figure 1B, there was a significant effect of frame on Structure of Thoughts (β = 0.52, SE = 0.11, p < .001). Also, the effect of Structure of Thoughts on choice was significant (β = 0.29, SE = 0.25, p < .001). After including the indirect path, the effect of frame was still significant (β = 0.14, SE =0.06, p = .02). Importantly, the indirect effect from frame through Structure of Thoughts to choice was, once again, significant and positive (β = 0.151, SE = 0.035, p < .001). When the sparse stimuli of the Unusual Disease problem were made more vivid, Query Theory processes still mediated the effect of frame on choice. However, the failure of Query Theory processes to completely mediate the effect of frame suggests that frames also change other processes.

PANAS Analysis

Prior work found that specific emotions, as measured by the PANAS items, predicted choices in the Unusual Disease problem (Druckman & McDermott, 2008). (Note: Traditionally PANAS items of a certain valence are combined). For example, participants’ anger correlated with choosing the risky option, independent of frame. Their analysis, however, only investigated how incidental emotions predicted choice, not how frames change emotions. We adapted the methods used by Druckman and McDermott to explore whether frames changed the PANAS emotions and how such changes would relate to choice (Lerner et al., 2015).

To test for the effect of frame on specific emotions, we used a multilevel model — with varying intercepts for each participant — predicting the difference between post- and pre-choice emotions. The 19 emotions were dummy coded. That is, in the data each participant had 19 rows one for each emotion they rated with emotion type indexed by a dummy variable and another dummy variable indicating which frame the participant saw. For ease of exposition, we report the results of an ANOVA with type 3 sums of squares (The Sums of Squares for each variable were entered after all other variables, including interactions.) (Table A1 presents all coefficients.) Answering the Unusual Disease problem did not change all emotions equally, different emotions had different means (χ²(18)=1848.21, p < .001). Critically, changes in participants’ specific emotions were not altered by which frame they saw: there was no main effect of frame (χ²(1)=0.26, p = .61) or interaction between frame and emotion type (χ²(18)=17.60, p = .48).

Moreover, specific emotions measured by the PANAS did not predict choice. Specifically, we ran a regression that predicted choice with continuous predictors — post-choice emotion level minus pre-choice emotion level — for each of the 19 emotions. Because of the large number of predictors in this model, we used a regularized regression model — LASSO — to determine which emotions related to choice (Hastie, Tibshirani & Friedman, 2009). The LASSO shrinks coefficients based on cross validated prediction error. Unlike other regularization techniques, the LASSO allows for coefficient estimates to become zero making it a principled way to select model features. As seen in Appendix B, Figure A1, the LASSO model regularized the coefficients for each emotion to 0: no emotion predicted choice.

In sum, Study 3 replicates Study 1 and demonstrates that specific emotions, as measured with the PANAS scale, did not differ between frames and were unrelated to choice.

3.5 Study 4: Manipulated Thought Order Changes Choices But Not Emotions

Study 2 showed that an instruction that changed the structure of participant’s arguments influenced the effect of frame on choice. Study 4 replicates Study 2 — where initial arguments were manipulated — with Study 3’s vivid stimuli and the PANAS measures. That is, Study 4 tests whether the manipulated thought order of Study 2 changes emotions, as measured by the PANAS.

3.5.1 Method

Participants

For Study 4, we recruited 593 participants (310 female), age range from 18 to 74 (M = 34.71, SD = 11.62) with 182 with at least some college education, from Amazon Mechanical Turk. Participants who failed an attention check question (17 in total) were removed from the analyses.

Procedure

All procedures were identical to Study 2 except participants saw the vivid stimuli and provided the PANAS ratings as in Study 3.

3.5.2 Results

Choice

The choice results mirrored those of Study 2: the effect of frame was attenuated in the unnatural order compared to the natural order. As in Study 2, we performed a logistic regression with choice predicted by frame, thought order, and their interaction. Gain-frame participants were more likely to choose the certain option (70.0%) than loss-frame participants (43.6%) (β = 0.57, SE = .09, p < .001). Consistent with Figure 2B, the effect was of frame was significant in the natural order (β = 1.65, SE = 0.26, p < .0001) and unnatural order (β = 0.61, SE = 0.24, p = .001). However, the effect of frame depended on thought order (β = 0.26, SE = 0.09, p = .004). Consistent with the Study 2 results and Hypothesis 3, the effect of frame on choice was attenuated when participants listed thoughts in an unnatural order. As seen in Figure 2, the interaction between frame and thought order are nearly identical across Studies 2 and 4. As with Study 2, there was no main effect of thought order (β = 0.08, SE = .09, p = .36). When stimuli are more vivid, manipulated initial arguments still attenuates the effect of frame on choice.

PANAS analyses:

We ran a similar model as in the Study 3 PANAS analyses, that asked the additional question: does manipulated thought order change rated emotions? To examine this, we ran a multilevel linear regression, which predicted, for each of the 19 emotions, the difference between post and pre-test emotion by type of emotion, frame, thought listing order, and all two way interactions between the three variables and included a varying intercept for each participant. (We did not include a three way interaction because it frustrates the interpretation of the two way interactions). That is, in the data each participant had 19 rows one for each emotion they rated with emotion type indexed by a dummy variable, and two additional dummy coded variables (e.g., coded as 0 and 1), one for which frame the participant saw and one for which thought order the particpant was instructed to use. (Two participants did not rate the PANAS items both prior to and after their choice and were removed from this analysis). We report the results of an ANOVA with type 3 sums of squares because the interactions are of primary importance: we want to see if Thought Listing Order differentially affects changes in specific emotions. The two two-way interaction between Thought Listing Order and emotion type was not significant(χ²(18) = 27.77, p = .066).⁵. Thus, the effect of manipulated argument order on choices does not appear to be accompanied by a change in specific reported emotions.

As with the Study 3 analyses, when predicting changes in emotions there was a significant effect of specific emotion type — Amused, Cheerful, Happy, Afraid, Angry, etc. — (χ²(18) = 1818.02, p < .001). The effect of frame was not significant (χ²(1) = 0.98, p = .32). Unlike Study 3, there was an interaction between specific emotion and frame (χ²(18) = 74.91, < .001). As can be seen in the Table A2, loss frames induced larger decreases in happiness than gain frames; however, no change in emotion was related to choice.

The conclusion of these results corroborates the conclusion from the Study 3 results: that changes in emotions measured by PANAS do not mediate the effect of frame on choice. Unlike Study 3, emotions, as measured by the PANAS, were associated with the effect of frame but, like Study 3, were unrelated to choice (as seen in Figure A2). More importantly, this study demonstrates that exogenously manipulating thought order, which attenuates the framing effect, does so without changing experienced emotions measured by the PANAS.

4 Exploratory Sentiment Analysis

We argue that the PANAS is a problematic measure of emotions for two main reasons. First, the PANAS measures cannot distinguish if a change in emotion is associated with one option or the other. Feeling a negatively valenced emotion towards the risky option and a positively valenced emotion toward the certain option should drive choices towards the certain option. Second, emotions are evanescent and the PANAS is temporally separated from choice processes. Participants’ emotional reactions may have been over by the time we administered the PANAS (Ericsson & Simon, 1980). To address these two problems, we conducted an exploratory sentiment analysis of the participants’ type-aloud thoughts, which reflected choice-relevant recall. Thought listings are collected during the decision, addressing temporal separation, and each thought is coded by the participant as pertaining to the risky or certain option, allowing us to code which option was the subject of any potentially emotional content of each statement.

The sentiment analyses, however, have a clear weakness: they measure only emotional valence. Measuring only a single dimension of emotions is problematic because emotions with the same valence have different effects on risk taking. Prior work, however, only measured overall emotions; we measure valence about options. The analyses attempt to answer an empirical question: is the emotional valence of participant’s arguments predicted by frames and predictive of choice?

To test this question, we pooled data from Studies 1 and 3 — the two experiments which allowed subjects to adopt their own thought order. We also conducted a related analysis examining the effects of manipulated argument order on valence.

4.1 Method

We measured the sentiment — “the expression of subjectivity as either a positive or negative opinion” (Taboada, 2016, p. 326) — of participants written arguments using text analysis. We used the sentimentr package, which outperforms many other packages on common natural language processing benchmarks (Rinker, 2019). The analysis produces a unidimensional score with 0 representing a neutral valence. Positive valences are represented by positive numbers and negative valences by negative numbers. The degree of sentiment is represented by the distance from zero: “Too many will die” had a sentiment of −1.38 but, the less extreme, “I do not like gambling” had a sentiment of −0.11. The sentimentr package uses regular expressions to parse sentence boundaries and a popular sentiment dictionary (Hu & Liu, 2004) to determine both the sentiment of words and valence shifters. Valence shifters can flip the sentiment of a sentence. For instance, one participant wrote: “go with plan b because there is a possibility of nobody dying”. A context free sentiment analysis technique would assign the thought a negative sentiment, because it contains only one negative word: dying. In contrast, a sentence level sentiment analysis codes this thought as positive: the negative sentiment of “dying” is negated by “nobody”. An accurate measure of sentiment requires placing emotionally laden words in a broader, sentence level, context.

Using the sentence level sentiment of participants arguments, we developed two measures: one that differentiates between the sentiment of thoughts about the certain option and the sentiment of thoughts about the risky option — the relative-sentiment index — and one that adds all thought sentiments together regardless of which option the thought was about — the overall-sentiment index. The relative-sentiment index takes, for each participant, the average sentiment of their arguments about the certain option minus the average sentiment of their arguments about the risky option.⁶ If there were no differences in sentiment between arguments about the certain and risky option, the relative-sentiment index was 0. The overall-sentiment index adds the sentiments of arguments about the certain option to the sentiments of arguments about the risky option. For instance, a participant whose arguments about the certain option had a mean sentiment of +1 and arguments about risky option had a mean sentiment +2, would have an overall-sentiment index of 1+2 = 3.

While the two sentiment indexes use the same data, we predicted their relationship with choices would differ. The overall-sentiment index only measures the valence of people’s overall emotions and emotions with the same valence have different impacts on risky decision making: anger can lead to risk taking, but distress can lead to risk aversion (Druckman & McDermott, 2008; Lerner et al., 2015). Therefore, we did not predict that the overall-sentiment index would relate to choices. Because the relative-sentiment index compares differences in emotional valence about options, we predicted that it would relate to choices: expressing more positive sentiment about the certain option than the risky option should lead to choices about the certain option.

4.2 Results

Using the combined Studies 1 and 3 data, a linear regression showed that the relative-sentiment index was related to choice (β = −0.62, SE = 0.35, p < .001). Another linear regression showed that frames influenced the relative-sentiment index (β = 0.32 SE = 0.15, p < .033), but that vividness, Study 3 had vivid stimuli but Study 1, did not, (β = 0.058, SE = 0.14, p = .67) and the interaction between frame and vividness (β = 0.07, SE = 0.20, p = .7) did not. Even though frames do not change overall reported experienced emotions as measured by the PANAS scale, this result suggests that frames change option-evaluation related emotional valence.

In the Study 3 data, the relative-sentiment index was not related to the PANAS measures: a LASSO regression that predicted the relative-sentiment index with the PANAS scales represented as continuous predictors (post-choice emotion level minus pre-choice emotion level) for each of the 19 emotions regularized all parameters to zero. None of the 19 emotions predicted the relative-sentiment index. Also, relative-sentiment index predicted choice when included in the same regression as PANAS emotions (Figure A3). These results suggest that the relative-sentiment index captures unique variance in emotional processing.

Unlike the relative-sentiment index, the overall-sentiment index was not related to frames or choice. In a simple regression predicting choice, the overall-sentiment index was not significant (β = 0.01, SE = 0.1, p = .9). A linear regression predicting overall-sentiment index showed no main significant effect of frame (β = 0.17, SE = 0.16, p = .26), vividness (β = −0.23, SE = 0.14, p = .097), or interaction between frame and vividness (β = 0.30, SE = 0.2, p = .13).

Like the relative-sentiment index, the overall-sentiment index for the Study 3 data was unrelated to the PANAS. A LASSO regression regularized all parameters to zero: none of the 19 emotions predicted the overall-sentiment index.

Taken together, these results suggest that combining the emotional valence of participant’s arguments in a relative manner captures choice-relevant information, but combining valence in an absolute manner does not. While both sentiment indices were unrelated to specific emotions measured by PANAS, we do not consider this an issue. The PANAS measures are more temporally removed from choice than the relative-sentiment index; the PANAS measures emotions experienced but the relative-sentiment index captures which option the emotional valence was about.

4.2.1 Analyses of Both Relative-Sentiment Index and Structure of Thoughts

Given that relative-sentiment index and Structure of Thoughts are both calculated on all of participants thoughts, we cannot make causal claims about which came first with our data. However, exploratory path analyses allow us to examine whether relative-sentiment-index had an effect on choice that was distinct from the effect of frame and structure of thoughts. That is, when included in the same mediation model, does both the Structure of Thoughts and relative-sentiment index mediate choice?

We fit the path analysis using the lavaan R package with 5000 bootstrapped samples (Rosseel, 2012). Again, we used a probit model for regressions, which had choice as the dependent variable. Given that our vivid stimuli did not directly alter the content or structure of participant’s arguments, we used the combined Study 1 and 3 data but did not include the effect of stimuli vividness.⁷

Figure 4 displays the path model (Table A3 has all parameter estimates). Consistent with the results of Studies 1 and 3, the path from frame to Structure of Thoughts, the path from Structure of Thoughts to choice, and the direct path from frame to choice were all significant. Further, there is no direct effect of relative-sentiment index on choice (β = 0.02, SE = 0.02, p = .219). When included in a multiple mediation with the Structure of Thoughts, relative-sentiment index does not mediate the effect of frame on choice.

Figure 4: Path model for multiple mediation analysis. Paths where p > .05 are gray, paths where p < .05 are black. ^*** p < .001, ^** p < .01, ^* p < .05.

4.2.2 Reversing Argument Order Does Not Affect Relative-Sentiment Index

While the path model cannot test whether the emotional valence of arguments changes the structure of thoughts or vice versa, we reanalyze the Study 2 data for a more data driven test of this hypothesis. Specifically, if manipulated argument order also changed the relative-sentiment index, this would be evidence that altering the structure of participants arguments also changes the emotional content.

To examine this hypothesis, we ran an exploratory linear regression on the Study 2 data, which predicted the relative-sentiment index by frame, thought order, and their interaction.⁸ There was significant main effect of frame on relative-sentiment index (β = 0.23, SE = 0.042, p < .0001). However, there was no significant effect of thought order (β = 0.06, SE = 0.042, p = .171) or interaction between thought order and frame (β = −0.063, SE = 0.042, p = .162). The significant effect of frame on relative-sentiment index suggests that frames still affected the emotional valence of participants’ arguments, even when thought order is imposed. (This result holds for the combined Studies 1-3 data Appendix C — Figure A4). Also, a separate regression showed that the relative-sentiment index still related to the Balance of Thoughts (β = 0.19, SE = 0.043, p < .0001). That is, even when initial arguments were exogenously manipulated, the emotional valence of thoughts still related to the structure of arguments. Taken together, these results provide suggestive evidence that the structure of arguments does not exert a strong influence on the arguments emotional valence.

5 Discussion

Across four studies, we documented how risky choice frames affect the structure and content of participants arguments for and against options and how these measures relate to choices. Replicating prior work, gain-frame participants choose the certain option more often than loss-frame participants. Consistent with Query Theory, gain-frame participants listed arguments in favor of the certain option earlier and more often than loss-frame participants; this difference in argument structure mediated the effect of frame on choice. Consistent with a causal role of early arguments, when participants were instructed to list their thoughts in an unnatural order in Studies 2 and 4 (i.e., in the opposite order than the one identified to occur naturally/without instruction), the effect of frame on choice was attenuated. Comparing Studies 1 and 2 to Studies 3 and 4 suggests that our vivid stimuli did not appreciably change the structure or content of participants’ arguments nor their choices. More importantly, the PANAS measures of emotions were not influenced by frame or the manipulated argument order.

However, exploratory analyses showed that the sentiment of participants’ thoughts during option evaluation were influenced by frame. When included in a mediation analysis with the structure of thoughts, the sentiment of participant’s thoughts did not predict choice. Other analyses showed that exogenously manipulating argument structure towards an option did not change the expressed emotional valence. These results have multiple theoretical implications.

5.1 Theoretical Implications

Our results suggest that risky choice frames shift the structure of participants arguments. This result is consistent with Query Theory but not other theories.

While Query Theory naturally accounts for manipulated thought order attenuating framing effects, Fuzzy Trace Theory does not appear to be able to do so. Fuzzy Trace Theory suggests that people encode the gist of the options and compare them to one another — e.g., “some people are saved” is better than “some people are saved, or no people are saved”. To account for the results of Studies 2 and 4, Fuzzy Trace Theory would need to assume that gist encoding depends on the order in which the two options are considered.

Other results fit will with Query Theory: Prior evidence suggests that time pressure increases risky choice framing effects (Guo, Trueblood & Diederich, 2017). If people are serially evaluating arguments for and against options, then the options with initial arguments in their favor — those which are initially attractive — will be more likely to be chosen when there is time pressure, as time pressure may suspend evidence collection before arguments for the initial leader are exhausted and never switch to consideration of the alternative choice alternative. Based on our reading, Fuzzy Trace Theory would not predict these results without an additional assumption: peoples’ mental representations become more verbatim — they encode exact numeric information — the longer people deliberate and thus are less likely to show framing effects.

While Query Theory and Fuzzy Trace theory explanations are distinct, they are not mutually exclusive nor incompatible. Query Theory is based on attention to arguments and memory; Fuzzy Trace Theory is based on mental representation. The mental representations posited by Fuzzy Trace theory may be at a lower level of cognition than the type aloud protocol could measure.⁹ That is, mental representations consistent with Fuzzy Trace Theory may direct attention to arguments in favor of certain options. However, Fuzzy Trace processes are not sufficient to account for our results.

Our work also speaks to emotional accounts of the Unusual Disease problem. Specifically, we failed to replicate prior work showing that emotions, as measured by the PANAS, related to choice framing effects (Druckman & McDermott, 2008). However, our methodology differed. We measured how frames changed emotions, but Druckman and McDermott measured emotions after participants completed the Unusual Disease Problem. We performed an additional analysis looking at only the post choice emotions on the Study 3 data: the LASSO regression regularized all emotions and all interactions between emotions and frames to 0. Also, the subset of emotions used was different: prior work included enthusiasm and distress. While we did not include emotions related to enthusiasm, we did include emotions similar to distress: nervous, depressed, and afraid. Further, their analysis included multiple covariates — risk aversion tendency, gender, student status, expertise, and interactions between these covariates and frame — but ours did not. The inconsistencies between Druckman and McDermott’s results and ours may be due to differences in measurement and analyses. Future work should explore why these results were inconsistent.

Our exploratory sentiment analyses suggest that risky choice frames affect the valence of arguments expressed by the participants; related work explains framing effects by the emotional valence inherent in the choice options — e.g., the Explicated Valence Account (Tombu & Mandel, 2015). In further exploratory analyses, we investigated if the relative-sentiment index applied to choice options — that is, not the relative-sentiment of arguments, but the relative-sentiment of the choice options — can explain choices. When the description of the risky option is changed the relative-sentiment index makes predictions consistent with Fuzzy Trace Theory and the Explicated Valence Account. Moreover, as seen in Appendix C, when the certain option is made slightly risky the relative-sentiment index of the options correctly predicts choices but Fuzzy Trace Theory and the Explicated Valence Account do not (Tombu & Mandel, 2015).

Other, closely related work, measures the relationship between the content of arguments to their structure (Zhao, Richie & Bhatia, 2020). Instead of using sentiment analysis and relative-sentiment index to describe the contents of thoughts, they used semantic space models and computational memory models. Consistent with our account, they find that the structure of sequential arguments is best fit by a computational model closely related to Query Theory.

Our work has multiple limitations. We did not manipulate option order which may have introduced a very small main effect increase in choices of Program A, the first option considered if reading order were the only determinant of processing order, but this is the same main effect under both frames and gets canceled out when we compare choices under the two frames. Consistent with this explanation, other work found that “the effect of option order on choice…appears to be very weak” (Mandel & Kapler, 2018, p. 10). Nonetheless, future research should examine if option order relates to affect.

Also, by manipulating thought order in Studies 2 and 4 we may have, inadvertently, signaled to participants that they should choose the initially considered option. This demand effect, however, does not account for results from other, closely related, Query Theory studies. Specifically, arguments in favor of the option found to be the early leader were more accessible, as measured by decision times, than arguments in favor of the other option (Weber, et al., 2007). Other work has shown that participants who do not want to follow the externally imposed thought order do not do so (Hardisty, Johnson, Weber, 2010). Given these results, we are not concerned about exogenously imposed argument orders causing demand effects.

Relatedly, listing arguments in an unnatural order may make the arguments disfluent — e.g., subjectively difficult to process (Alter & Oppenheimer, 2009). This disfluency may lead to less articulated preferences, making choices regress towards a 50-50 split between options. While disfluency is a plausible explanation for the reversal study, it does not explain the mediation results or the enhanced accessibility of frame consistent arguments (Weber et al., 2007). We argue that Query theory is an attractive account of risky choice framing effects but, like any complicated psychological phenomenon, they likely arise for multiple reasons (McGuire, 1997).

Our measure — relative-sentiment index — captures emotional valence only, missing out on many dimensions of emotions. relative-sentiment index’s relationship to frames and choices, highlights the tradeoff between precisely measuring a specific emotion separated from choice and measuring only an aspect of an emotions — e.g., the valence — in close proximity to the decision making process. Most prior studies have measured specific emotions, likely leading to theoretical accounts of emotions and decision making that focus on the overall feelings of the decision-maker (Lerner et al., 2015). We argue that work should acknowledge that measuring specific emotions is often decoupled from actual choice processes and highlight the limitations imposed by many measures of emotions.

Beyond generating insight into how the structure and content of participant’s arguments relate to risky choice framing effects, our results also give insight into other framing effects: the endowment effect, default effects in intertemporal choice, and tax versus offset framing. To our knowledge, ours is the first study to measure participants’ relative emotional valence about the options in close temporal proximity to choice. Our results suggest that other framing effects may be driven, in part, by the relative emotional valence about options attracting initial arguments. For example, conservatives consider “tax” to be a dirty word, so their arguments about an expensive plane ticket with a carbon tax may have been more negatively valence than a plane ticket with a carbon offset (Hardisty et al., 2010).

5.2 Conclusion

Taken together, our results suggest that how risky outcomes are framed affects both the structure and content of peoples’ internal deliberation. This result has numerous real-world and managerial implications. For instance, in a negotiations context, it suggests that how a choice between agreeing to an offered contract (a certain option) vs. going out on strike (a risky option) is framed (as options in the gain domain or options in the loss domain) may lead to negotiators assigning different emotional valences to the options and altering the structure of their internal deliberations. Our results suggest that debiasing of risky choice framing effects requires additional understanding how frames affect the content and structure of internal arguments.