Hostname: page-component-586b7cd67f-dsjbd Total loading time: 0 Render date: 2024-11-23T23:24:55.679Z Has data issue: false hasContentIssue false

Measurement is the core disgust problem: Response to Inbar and Scott (2018)

Published online by Cambridge University Press:  01 January 2023

Corey Cusimano*
Affiliation:
Department of Psychology, University of Pennsylvania, 3720 Walnut Street, Solomon Lab Building, Philadelphia, PA, 19104
Edward B. Royzman
Affiliation:
Department of Psychology, University of Pennsylvania
Robert F. Leeman
Affiliation:
Department of Health Education and Behavior, University of Florida; Department of Psychiatry, Yale School of Medicine
Stephen Metas
Affiliation:
Department of Psychology, Widener University
Rights & Permissions [Opens in a new window]

Abstract

Scott, Inbar and Rozin (2016) presented evidence that trait disgust predicts opposition to genetically modified food (GMF). Royzman, Cusimano, and Leeman (2017) argued that these authors did not appropriately measure trait disgust (disgust qua oral inhibition or OI) and that, once appropriately measured, the hypothesized association between disgust and GMF attitudes was not present. In their commentary, Inbar and Scott (2018) challenge our conclusions in several ways. In this response, we defend our conclusions by showing (a) that OI is psychometrically distinct from other affective categories, (b) that OI is widely held to be the criterial feature of disgust and (c) that we were well-justified to pair OI with the pathogen-linked vignettes that we used. Furthermore, we extend our critique to the new findings presented by Inbar and Scott (2018); we show that worry and suspicion (not disgust) are the dominant affective states one is likely to experience while thinking about GMF and that the true prevalence of disgust is about zero. We conclude by underscoring that the present argument and findings are a part of a larger body of evidence challenging any causal effect of disgust on morality.

Type
Research Article
Creative Commons
Creative Common License - CCCreative Common License - BY
The authors license this article under the terms of the Creative Commons Attribution 3.0 License.
Copyright
Copyright © The Authors [2018] This is an Open Access article, distributed under the terms of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.

1 Introduction

Reference Ekman, Matsumoto, Friesen, Ekman and RosenbergScott, Inbar and Rozin (2016) (hereafter, SIR) presented evidence that trait disgust underlies opposition to genetically modified foods (GMFs). We (Reference Costello and OsborneRoyzman, Cusimano & Leeman, 2017) disputed their claim on the basis that SIR did not appropriately measure trait disgust (disgust qua oral inhibition or OI)Footnote 1 and that, once measured appropriately, it appears to play no role in people’s opposition to GMF. Instead we found that sensitivity to a potential/uncertain threat (as gauged by feelings of creepiness in response to a variety of “unpleasant” scenarios comprising the 7-item pathogen sub-scale [TDDS-P] [Reference Fessler, Arguello, Mekdara and MaciasTybur, Lieberman & Griskevicius, 2009]) was the best affective predictor of judgments regarding GMF. Reference Inbar and ScottInbar and Scott (2018) (hereafter, IS) have taken issue with our paper on many grounds. In most cases we believe their criticism misses the mark. In this paper, we lay out our rationale why and present additional empirical evidence that bears on the new findings that IS report. Though this response is a critique, we believe that Inbar, Rozin and Scott have raised some thought-provoking points and we would welcome the opportunity to explore them collaboratively in the future. We begin by reviewing our key objections to SIR and what we have done to address them.

2 Objections to SIR and an overview of Reference Royzman, Cusimano and LeemanRoyzman et al. (2017)

SIR reported an association between subjects’ responses to the Disgust-Sensitivity Scale – Revised (DS-R, Reference Olatunji, Williams, Tolin, Abramowitz, Sawchuk, Lohr and ElwoodOlatunji et al., 2007), a refined variant of Reference Haidt, McCauley and RozinHaidt et al.’s (1994) trait disgust sensitivity scale, and their opposition toward GMF. This scale has been criticized for its content as well as its response options (e.g., Landy & Piazza, 2017; Reference Fessler, Arguello, Mekdara and MaciasTybur et al., 2009). Regarding the former, the majority of vignettes describe situations that could be interpreted as physically disgusting, but also creepy, irritating, frightening, surprising, or upsetting. Regarding the latter, only a subset of items (vignettes) ask participants to report how “disgusting” they find a situation or event, with the remaining items’ response options covering a wide range of negative states, including “bothered”, “upset”, and other non-specific expressions of avoidance. Exacerbating these issues, the meaning of “disgusting” itself has been has been shown to be broad enough to encompass a gamut of affective reactions (e.g., Landy & Piazza, 2017; Reference NabiNabi, 2002; Reference Costello and OsborneRoyzman et al., 2017), making it the denotative equivalent of “bothered”. Given the multi-faceted nature of the DS-R, one would be hard pressed to see it as anything more specific than a measure of negative affect (with emphasis on creepiness, horror, disapproval, and disgust).

Our study was, in part, designed to ask if disgust sensitivity would still predict attitudes toward GMF when we could be more certain that we were measuring actual feelings of disgust. To address the limitations of the DS-R, we used a set of vignettes that are more pathogen-specific (TDDS-P, Reference Tybur, Lieberman and GriskeviciusTybur et al., 2009) as well as a 3-item measure of disgust as oral inhibition (OI), a measure derived from the theoretical meaning of disgust (Reference NabiNabi, 2002; Reference Royzman, Leeman and SabiniRoyzman, Leeman & Sabini, 2008; see below). Prior studies have shown OI (unlike “disgust”Footnote 2) to be specific to pathogen-linked content (Reference Royzman, Atanasov, Landy, Parks and GeptyRoyzman, Atanasov, Landy, Parks & Gepty, 2014; Reference Blake, Yih, Zhao, Sung and Harmon-JonesBlake, Yih, Zhao, Sung & Harmon-Jones, 2016), unrelated to social desirability concerns (Reference Royzman, Kim and LeemanRoyzman, Kim & Leeman, 2015), and resistant to figurative use (Reference Royzman, Leeman and SabiniRoyzman et al., 2008). We also included a set of concretely phrased non-disgust options that made it possible to isolate OI from some pertinent alternative reactions (e.g., social disapproval or unpleasant non-gustatory sensations). The inclusion of specific non-disgust options is a significant improvement for two reasons. First, including distinct alternative response options undercuts any motivation for a subject to use OI to express some other feeling that they may have or misreport what they feel in the interest of signaling that they have had some feelings at all. Second, it allows one to test whether any of these other affective reactions may have driven the original reported association between “disgust” and GMF attitudes. Our study was also an improvement in terms of scale administration (see Reference Royzman, Cusimano and LeemanRoyzman et al., 2017, p. 467, for a summary).

With these refinements in place, the correlation between oral inhibition and people’s attitudes to GM-food was close to zero. A subsequent analysis showed that OI was not predictive of GMF attitudes either as a composite or at the level of its constituent responses (all ps> 0.4). Thus, contrary to the claims made by SIR, disgust-the-feeling (OI) appears to be unrelated to a desire to restrict GM-food (harms and benefits not withstanding). Notably, disgust-the-word (“disgusting”) still did rather well, serving as a significant predictor of opposition to GM-food (r = .25, p = .004), just as it did in SIR’s original study.

In spite of these deflationary results, Inbar and Scott make no attempt to defend their approach. Thus, it appears that what ultimately divides us is not the question of whether DS-R can be retained or salvaged as a measure of trait disgust but whether our approach offered enough of an improvement over SIR’s to make our findings more trustworthy.

IS advance four major reasons why they do not view our study as an improvement. First, they express a concern about our lack of proper psychometric analyses, thereby calling into question whether our novel high-granularity response options did in fact carve out distinct affective categories. Second, and most fundamentally, they assert that OI is incidental to the modern construal of disgust and therefore that we were unjustified using OI to operationalize disgust. Third, they argue that it is particularly inappropriate to pair OI with TDDS-P since it is characterized by lack of “oral” items. And fourth, they question the lay meaning of “creeped out” in the context of the TDDS-P vignettes. Finally, in their response, IS offer new evidence purporting to bolster their claim that disgust is the predominant feeling in reaction to thinking about GMF. We address each of these issues in turn.

3 Have we measured distinct affective categories?

One of Inbar and Scott’s central critiques is that we are not justified to assume that our scale of concrete, high-granularity options (e.g., “feel like gagging”) in fact measured separate affective categories (e.g., distinguishing oral inhibition from non-gustatory aversive sensations or moral disapproval).

In developing this critique, Inbar and Scott state early on that confirmatory factor analysis (CFA) is not appropriate for the three single-item factors in the measure because CFA requires a minimum of two (but preferably three) indicators for each latent factor (Reference BentlerBrown, 2006). This is a valid point, but the issue is broader than that. Factor analysis in general (not only CFA but including exploratory factor analysis), is considered inappropriate for factors with less than two items (Reference Costello and OsborneCostello and Osborne, 2005). As a result, the full 11-item measure utilized by Reference Royzman, Cusimano and LeemanRoyzman et al. (2017) is not appropriate for inclusion in any type of factor analysis since it was designed to include multiple single-item factors. Thus, the exploratory factor analysis conducted and reported by Inbar and Scott was inappropriate on statistical grounds.

CFA is relevant to our measure because it was based on established scales, and items were grouped a priori according to presupposed factors based on theory, prior findings, and face validity (Reference BentlerBrown 2006). Thus, for each of the seven vignettes we conducted CFA for the three putative factors that included at least two indicator variables. This yielded a total of 8 items per vignette: the three oral inhibition items, the three disapproval items, and the two epidermal discomfort items.

The model fit indices included (among others): chi-square (Reference HatcherHatcher, 1994), the SRMR (Reference Hu and BentlerHu & Bentler, 1999), the Comparative Fit Index (CFI; Reference BentlerBentler, 1990), and the Root Mean Square Error of Approximation (RMSEA; Reference Brown, Cudeck, Bollen and LongBrown & Cudeck, 1993).Footnote 3 These indices indicated excellent fit of the data to the proposed model structure. For example, chi-square was statistically significant for 6 of the 7 vignettes with the “poop” vignette being the only one with a significant chi-square value. (Notably, chi-square is often significant even in cases when a model provides an acceptable fit to the data [Reference Floyd and WidamanFloyd & Widaman, 1995; Reference HatcherHatcher, 1994]). CFI values were greater than .95 for all seven vignettes and SRMR values were less than .08 for all seven vignettes. RMSEA was less than .08 for six vignettes with the other coming in under the .10 value that is considered acceptable. Additional indices were equally consistent with the model (contact authors for further details).

In summary, the single item factors included in the measure are not appropriate for any form of factor analysis. This is indeed an inherent limitation to the use of single item factors, however these items included in Reference Royzman, Cusimano and LeemanRoyzman et al. (2017) were highly face valid. CFAs conducted with the remaining response options indicated excellent model fit for each of the seven vignettes measured. These results support a conclusion that the measure utilized in Reference Royzman, Cusimano and LeemanRoyzman et al. (2017) was not psychometrically unsound as argued.

4 Is oral inhibition an appropriate measure of disgust?

IS’s most pivotal critique is that there is a clear scientific consensus that oral inhibition is not a theoretically appropriate conceptualization of disgust. If they are correct, then our failure to observe an association between oral inhibition and GM attitudes has no significant bearing on their claim that feelings of disgust affect GM attitudes. However, it appears that IS are mistaken about the current consensus among disgust scholars.

From Darwin on, scholars viewed disgust as a category of food rejection (e.g., Reference Rozin, Haidt, McCauley, Lewis and HavilandRozin, Haidt & McCauley, 1993; Reference Olatunji and SawchukOluatunji & Sawchuck, 2005), whose origin is the “mammalian bitter taste rejection system” (Reference Rozin, Haidt and FincherRozin, Haidt & Fincher, 2009, p. 1180), and whose facial emblem is the “gape” (Reference Rozin, Lowery and EbertRozin, Lowery & Ebert, 1994) or the “sick face” (Reference Widen, Pochedly, Pieloch and RussellWiden, Pochedly, Pieloch & Russell, 2013). Rozin views disgust as “a food-related emotion” defined as “revulsion at the prospect of oral incorporation of offensive objects” (Reference Rozin and FallonRozin & Fallon, 1987, p. 23; Reference Rozin, Haidt, McCauley, Levinson, Ponzetti and JorgensonRozin, Haidt & McCauley, 2000) and affirms that all the classic disgust “scholars, as well as most others, posit a special relation between disgust and the mouth, and disgust and food rejection” (Reference Rozin, Haidt, McCauley, Levinson, Ponzetti and JorgensonRozin et al., 2000, p. 188). Reference Haidt, McCauley and RozinHaidt et al. (1994), whose trait disgust scale is the precursor to the DS-R (Reference Olatunji, Williams, Tolin, Abramowitz, Sawchuk, Lohr and ElwoodOlatunji et al., 2007), stipulate that disgust is “at its core, an oral defense” (p. 702) while Olatunji & Sawchuck describe it as a manner of food rejection and a “guardian of the mouth” (Reference Olatunji and SawchukOluatunji & Sawchuck, 2005, p. 935). Reference EkmanEkman (2003) explicitly linked disgust with gagging (see also Reference Ekman, Matsumoto, Friesen, Ekman and RosenbergEkman, Matsumoto & Friesen, 2005). Thus, in an influential research report designed to unknot the more pointed theoretical meaning of disgust from the rougher lay meaning of the term, Reference NabiNabi (2002) affirms that, though disgust’s broad “common usage … appears to reflect that which is not only repellent but also irritating or annoying” (Reference NabiNabi, 2002, p. 695), “theoretically, disgust refers to the offence taken to noxious objects or ideas that evoke…nausea” (Reference NabiNabi, 2002, p. 695) (see also Reference Kayyal, Pochedly, McCarthy and RussellKayyal, Pochedly, McCarthy & Russell, 2015; Reference Royzman and SabiniRoyzman & Sabini, 2001).

Most importantly, oral inhibition emerges as the criterial feature of disgust that contemporary scholars reach for when in need of a clear touchstone to separate disgust from non-disgust. Reference NabiNabi (2002) used a desire to vomit (“feel like throwing up”) as a key criterion for establishing if the lay terms “disgusted/“disgust” brought to mind what was mainly disgust or some mix of annoyance and anger. In an analysis seeking to differentiate disgust from a related parasite avoidance adaptation, Reference Kupfer and FesslerKupfer and Fessler (2018) delineate the experience of being disgusted as involving “feelings of revulsion, nausea, gagging, the urge to vomit”. In arguing that Nazis are genuinely disgusting, Reference Sherman, Haidt and CoanSherman, Haidt and Coan (2007) aimed to show that people watching a video about the neo-Nazi culture reported the tightening and clenching of their throats. In claiming that fraud and dishonesty provoke actual disgust, Reference Chan, Boven, Andrade and ArielyChan, Boven, Andrade and Ariely (2014) sought to demonstrate that being exposed to instances of deceit reduces oral consumption of water or milk. And, as part of the project aiming to produce a highly discriminant facial display of disgust, one that would clearly differentiate it from anger, sadness, or fear, Widen et al. (2014) instructed an actress to imagine that she was feeling sick and was about to vomit (with the resultant “sick face” being highly similar to the “gape” face used by SIR).

Some scholars have also forewarned of the dangers of linking disgust with something as non-criterial as avoidance since, unlike OI, it would result in difficulties demarcating it from states such as “fear” (e.g., at spiders and gore; Reference Cisler, Olatunji and LohrCisler, Olatunji & Lohr, 2009; Reference Woody and TeachmanWoody & Teachman, 2000), which people also experience when exposed to pathogen vectors. In the case of a self-report measure such as the TDDS-P, operationalizing disgust as avoidance (e.g. “rate the degree to which this [bleeding cut] makes you want to move away”) would make it nearly impossible to separate it from a medley of other affective states (fear, creepiness, epidermal discomfort, disapproval) which TDDS-P is well capable of invoking (Landy & Piazza, 2017; Reference Costello and OsborneRoyzman et al., 2017).

The current scientific consensus is hard to miss. Oral inhibition is what scholars uniformly appeal to when the goal is isolate the criterial feature(s) of disgust, be it for the purpose of segregating the specific theoretical meaning from a broader lay one, differentiating disgust from a closely related non-disgust response, establishing that a non-prototypical disgust stimulus (e.g., Nazis, fraudsters) is genuinely disgusting, developing a discriminant non-verbal operationalization of disgust, or ascertaining cases of disgust-specific avoidance in a clinical setting. Any non-OI delineation of disgust is thus likely to be a minority position.

What about the authors that IS cite? Could they be the dissenting minority in question? The answer appears to be no. To the degree that these authors take a stab at delineating the disgust experience at all, all appear to endorse OI as the key component of that experience. Reference Tybur, Lieberman and GriskeviciusTybur et al. (2009) list “sensation of disgust” and “disgust facial expression” as the two primary features of the output of the “disgust system” (p. 69) and suggest that disgust is characteristically or “often accompanied by nausea, a desire to vomit, and a loss of appetite” (p. 70). Reference Cisler, Olatunji and LohrMurray and Schaller (2016) limit their comment on the “emotional experience” of disgust to an observation that said experience has “evolved from a more ancient and functionally specific distaste response to oral stimuli (Reference Rozin, Haidt, McCauley, Levinson, Ponzetti and JorgensonRozin et. al, 2000), and many contemporary conceptual accounts identify infectious diseases as the primary selective pressure underlying its evolution” (p. 83). Finally, according to Reference Curtis, de Barra and AungerCurtis, de Barra, and Aunger (2011):

Contact with disgust elicitors, real or imagined, is associated with (i) a characteristic facial expression that is recognizable across cultures, (ii) behaviour patterns that include withdrawal, distancing, stopping or dropping the object of disgust and shuddering, (iii) physiological changes including lowered blood pressure and galvanic skin response, recruitment of serotonin pathways, increased immune strength, and (iv) reports of negative affect including nausea. (p. 390; italicized text is what Inbar and Scott quote as support for their view; bold emphasis is ours).

The quote merits two observations. First, Curtis et al.’s goal is clearly not to define or specify the criterial feature(s) of disgust, but rather to describe some of its observable identifiers or “associates”, none of which are at odds with the conception of disgust as OI. But more importantly, there is little doubt that, in the context of self-report measures, the identifier or associate of greatest relevance is (iv), which specifies what disgust is as an affective state and features nausea front and center.

None of this is obviously to dispute that disgust is linked to pathogen avoidance. Reference Royzman and SabiniRoyzman and Sabini (2001) were among the earlier authors to characterize disgust as a system for minimizing toxin or pathogen exposure at the level of adaptive task description. However, they also noted that the aversive experience of OI is a good candidate response for motivating physical withdrawal. All in all, there is little doubt that the prevailing scientific consensus is squarely on the side of OI being the criterial feature of disgust. The authors that IS have appealed to are either a part of (or, at the very least, do not detract from) this broad consensus.

5 Is oral inhibition ill-suited for TDDS-P?

Even if OI comprised a defensible measure of disgust in general, it may not have been defensible in the context of TDDS-P. In IS’s view, OI is “a priori … a poor fit for these [TDDS-P] non-oral items” (Reference Inbar and ScottInbar & Scott, 2018, p. 5). They are not explicit about why this is an issue, but we have a hunch. When it comes to items featuring red sores, bleeding cuts, or scurrying insects, the key adaptive task seems to be that of avoiding a potential pathogen exposure by eschewing physical contact with its source. One could argue that gagging and pursing one’s lips would have little adaptive value; what one wants first and foremost is to put some distance between oneself and a potentially diseased entity. Thus, the argument goes, people may not have felt true oral inhibition in response to these items at all.

But if the major concern is a lack of variance due to a floor effect, this is a worry easy enough to dispel by looking at our data. We found that oral inhibition (tied with creeped out) was the strongest high-granularity response reported in reaction to the TDDS-P items (with OI’s coefficient of variance [0.42] being 20% higher than that of “disgusted” [0.23], which did predict resistance to GMF, in line with Reference Scott, Inbar and RozinScott et al. [2016]). We see no reason why this should be otherwise. Other than the “moldy leftovers” item and the scurrying cockroach item (see Reference van Huisvan Huis, 2017 on the evidence of insect consumption among early humans and through evolutionary history), at least three items in the TDDS-P feature explicit (body odor) or implied (sweaty palms, dog poop) malodorousness, one of the key inputs to the food rejection system (Reference Rozin, Lowery and EbertRozin et al., 1994)Footnote 4. This issue to the side, it is unclear why the “orality” of an item should matter in the first place. Congruent with the theory of preadaptation (i.e., the co-option of existing structures/systems for new use; Reference Rozin and FallonRozin & Fallon, 1987), there is plentiful evidence that oral inhibition is elicited by a range of “non-oral” items ranging from bodily fluids (Reference Royzman, Leeman and SabiniRoyzman et al., 2008, Pilot) and injuries (Reference Chapman, Kim, Susskind and AndersonChapman, Kim, Susskind & Anderson, 2009; Reference Royzman, Leeman and SabiniRoyzman et al., 2008, Pilot) to anomalous sex (Reference Royzman, Leeman and SabiniRoyzman et al., 2008; Studies 1 and 2; Reference Royzman, Atanasov, Landy, Parks and GeptyRoyzman et al., 2014). In sum, our use of the TDDS-P scale for eliciting oral inhibition is as well justified as the use of OI itself.

6 Taking stock

Let’s collect our results. IS critiqued OI as a theoretically appropriate way to conceptualize and operationalize disgust while also contending that it was a particularly poor match for the TDDS-P vignettes. Having reviewed IS’s arguments, we have concluded that (1) according to the prevailing scientific consensus, oral inhibition is the criterial feature of disgust, (2) has no clear alternative, and (3) is the only reliable way to demarcate disgust from other pathogen avoidance mechanisms (creepiness, epidermal discomfort, fear), at least in the context of a self-report measure à la TDDS-P. Furthermore, (4) the combined weight of theory and evidence strongly indicates that our use of the TDDS-P scale was appropriate for eliciting oral inhibition. Along with the psychometric evidence in the section above, our decision to use a wide range of additional affective response options (including some face valid single-factor response options) further engenders confidence that subjects used our measure as intended.

What can be said about the overall validity of our measure? In modern validity theory (Reference Newton and ShawNewton & Shaw, 2013), validity is not considered a property of the test, but of a particular interpretation of that test adopted for a given purpose and whether this interpretation is justified in light of the totality of evidence before us. The key interpretative move that undergirds all conclusions in Reference Royzman, Cusimano and LeemanRoyzman et al. (2017) is that, imperfect as our assessment might have been, it is a considerably better proxy for the feelings of disgust (as understood by the scientific community at large) and their relationship to GMF attitudes than SIR’s. Everything that has been said above — OI’s hard-to-argue status as the criterial feature of disgust, its apparent specificity, its psychometric differentiation from Disapproval and Epidermal Discomfort, its selective tracking of pathogen-linked content — makes the move well justified.

7 The meaning of “creeped out”

If disgust sensitivity, properly measured, does not predict GMF attitudes, why might disgust sensitivity, improperly measured, do so? We hypothesized that sensitivity to potential threats might have, among other things, confounded the relationship between improperly-measured disgust sensitivity and GMF attitudes. This hypothesis was supported by a correlation between GMF attitudes and feeling “creeped out” in response to the TDDS-P.

IS claim that, while “creeped out” may be associated with unease about a potential threat or riskFootnote 5, it is an open question if this is the meaning that subjects had in mind when rating the items of TDDS-P. Implied in this worry is the claim that (a) the canonical elicitors of creepiness (those that one would expect to loom large in a subject’s minds when queried about the meaning of “creepiness” as such) are categorically different from (b) the content of the TDDS-P vignettes. But is this borne out by the evidence? Reference McAndrew and KoehnkeMcAndrew and Koehnke (2016) conclude from their study that non-normative health and hygiene (e.g., being extremely thin or obese, having greasy hair or pale skin) are typical triggers of creepiness (see also Reference Watt, Maitland and GallagherWatt, Maitland, & Gallagher, 2017, for a convergent analysis). In light of this, it appears that the characteristic triggers of feeling creeped out are well represented in TDDS-P. Indeed, half of the TDDS-P items portray individuals with non-normative, unhealthy physical characteristics or poor hygiene, including “Sitting next to someone who has red sores on their arm.”, “Standing close to a person who has body odor.” “Shaking hands with a stranger with sweaty palms”, and “Accidentally touching a person’s bloody cut”.

Whatever “creeped out” may or may not mean in the context of TDDS-P, it clearly not a stand-in for oral inhibition, the construct that comprises the theoretical meaning of disgust. It is this construct that SIR and IS presumably would want to implicate, not merely as a trait predictor, but also as a state antecedent, of people’s avowed opposition to genetically engineering animals and plants. Thus, the strong correlation between “creeped out” and GMF attitudes should not be taken as evidence in favor of IS’s thesis.

8 State disgust

In the penultimate section of their commentary, Inbar and Scott present an empirical critique of what they perceive to be our theoretical counterpoint to their “disgust fuels the moral opposition” proposal, i.e., that “fear, not disgust, is the primary emotion associated with GM food opposition” (Reference Inbar and ScottInbar & Scott, 2018, p. 3). They offer no textual evidence for this claim. This is not surprising since we never made such a claim. Our paper was concerned with the experience of “creepiness” (which we explicitly and repeatedly set apart from that of “fear” both in the principal study and in the supplementary survey) as a proxy for trait sensitivity to a potential hazard or risk. Our claim that sensitivity or vigilance towards risks may underlie GMF opposition is not equivalent to the claim that fear is the dominant reaction to thinking about GMFs, nor that fear (or a related feeling) is a causal antecedent of GMF oppositionFootnote 6. Therefore, IS are largely correct that we do not show evidence in favor of the claim that state fear is the dominant reaction to GMFs, but this is because no such evidence has ever been searched for in the first place.

This misunderstanding to the side, we have no principled opposition to the view that disgust is the dominant affective reaction towards GMF. That said, we were skeptical of the evidence that IS and SIR presented defending this claim, and conducted a study to re-assess the prevalence of disgust as a reaction to GMF.

9 Study re-assessing the prevalence of state-disgust

We conducted a study (described in detail in the Appendix) geared to re-reassess the absolute and relative prevalence of state disgust when thinking about GMF. Both SIR and IS presented some studies to this effect. Our study departed from theirs in several ways. First, rather than presenting subjects with GMF-related vignettes, (e.g., unintentionally eating genetically modified salmon), subjects were asked simply to report what they felt while thinking about GMF. This affords us a more ecologically valid measure of the types of stimuli/vignettes that might naturally come to mind when thinking about whether GMF should be opposed or prohibited.

We also modified the way we probed subjects’ affective states. SIR asked their subjects to report which of the two response options (disgust vs. anger) best reflects their reaction to different GM scenarios. IS used a similar method with three response options (disgust, anger, and fear) instead. However, this method comes with some strong assumptions. First, it forces people to report one feeling or another (leaving no room for the possibility that no emotion is being felt). Second, it operationalizes disgust as the word “disgust” (assuming, once again, a close correspondence between the theoretical and lay meanings of the term, an assumption disconfirmed some 15 years ago; see also our arguments above). Third, it limits the range of non-disgust options to one (“anger” in SIR) or two (“fear” and “anger” in IS), assuming that these states would adequately and fairly cover anything and everything else one could feel in this case. These assumptions are difficult to defend, and therefore call into question the validity of subjects responsesFootnote 7. To address this, in addition to running a conceptual replication of SIR’s study, we designed two additional conditions that do not make these assumptions and so furnish a truer test of what feelings, if any, people experience.

At the beginning of the study, all subjects (N = 1048 recruited, N = 922 retained following attention check) were queried about whether they opposed or did not oppose “genetically engineering plants and animals for food production” (thus separating opponents from non-opponents) and whether they thought it “should be prohibited no matter how great the benefits and minor the risks of allowing it” (with an affirmative answer leading to the subject being classified as an opposing absolutist). Then, after completing a brief attention check, subjects were randomly assigned to one of three affective report conditions: (1) a replication of IS’s study in which subjects were forced to choose between two feeling states, Angry or Disgusted (2-item FC), (2) a 12-item forced choice condition (12-item FC) in which subjects were forced to choose between feeling Afraid, Angry, Bothered, Confused, Creeped out, Excited, Frustrated, Grossed out [a slang term shown to correspond more precisely than “disgusted” to the theoretical meaning of disgust, Nabi, 2002], Neutral, Sad, Suspicious, or Worried, or (3) an open-ended response mode condition (OERM).

The OERM condition was unique in that it left open the possibility that people do not feel or perceive themselves as having affective reactions at all. Rather, in this condition, subjects were prompted to “use the space below to describe (as you would in a diary) what is going though your mind and what you are experiencing as you are thinking about the practice of genetically engineering plants and animals for food production”. Subjects were further enjoined to note that there are “no right or wrong answers” and not to “hold back or self-censor in any way”.

To diagnose the prevalence of affective language in OERM, we carried out a series of content analyses using a total of 81 affective word stems corresponding to the feeling concepts tapped in the 12-item FC condition of the study. 80 of these were organized into 8-item clusters detailed in Table 2 (Appendix). Due to its precise classification being the present point of contention, “creeped out” was retained as a “free-standing” term, though, in our view, the evidence from prior research (Reference Costello and OsborneRoyzman et al., 2017; Reference McAndrew and KoehnkeMcAndrew & Koehnke, 2016), clearly substantiates its inclusion in a Fear (Afraid + Worry + Suspicion) super-cluster.

As we predicted, SIR’s procedure for measuring state disgust overestimates the absolute prevalence of disgust in relation to other, more assumption-free methods (see Table 1 above). When subjects’ choices were limited to disgust and anger, disgust substantially and significantly dominated non-disgust for absolute opponents (73.3 percent) as well as for the sample as a whole (62.2 percent). However, only 2.8% of the sample chose “grossed out” in the 12-item FC condition (Table 3, Appendix) and only 1.7% of the sample mentioned one of the target disgust-related words in the OERM condition (Table 4, Appendix), both significant drops from the 62.2 percent, χ2(1, 633) = 256.55, p< .0001 and χ2(1, 598) = 246.5, p< .0001, respectively (based on the “N-1” Chi-squared test as recommended by Reference CampbellCampbell, 2007, and Reference RichardsonRichardson, 2011), with no significant difference between the 12-item condition and OERM (χ2[1, 633] = 0.82, p = 0.36).

Table 1: Proportion of subjects choosing/using disgust-related terms as a function of study condition and evaluative orientation (absolute opposition, non-absolute opposition, non-opposition).

Note. FC = forced choice; OERM = open-ended response mode. Superscripts indicate whether disgust is more dominant than

a is dominated by

b or is equivalent to

c non-disgust by Chi-square goodness-of-fit at α = 0.05.

The open-ended response format (Table 4, Appendix), which provides the most assumption-free approach to the problem, suggests that very few people spontaneously perceived themselves as experiencing any emotion at all (41 out of 288, 14.2%). For those apt to experience any feeling at all, feeling worried appears to be three times more common than feeling disgusted (see Appendix for details). Finally, it is worth noting that 1.7 percent (the absolute prevalence of disgust in the open-ended condition) is likely to be an overestimate and that a truly conservative estimate of spontaneously experienced and causally relevant state disgust is likely to be closer to zero (see Appendix for discussion).

These results strongly disqualify disgust as the driver of absolutist opposition in the face of consequentialist reasons. If Reference Scott, Inbar and RozinScott et al.’s (2016) sentimentalist viewpoint was correct, and absolute opposition to GM food was in fact “disgust-based” (p. 316) or “disgust-backed” (p. 322), the last thing one would expect to find is that the majority of GMF’s absolutist opponents feel no disgust at all.

10 Conclusion

In this reply, we defended our critique of the relationship between trait disgust and GMF and extended it to the issue of state disgust. In both cases, SIR and IS appear to overestimate the role that disgust may play in opposing GMF due to imprecisely measuring disgust or by providing limited response options that artificially constrain a subject’s capacity to state their feelings (or lack thereof). When these limitations are addressed, we find a different suite of feelings to come forth, by and large, related to the perceptions of danger, misfortune, or risk.

When considered in the broader context of disgust and moral judgment research, our findings are not at all surprising. A meta-analysis (Reference Landy and GoodwinLandy & Goodwin, 2015), and large-scale replication (Reference Johnson, Cheung and DonnellanJohnson, Cheung & Donnellan, 2014) both conclude that feelings of incidental disgust have practically no influence on moral judgment. Similar in spirit are several past studies (e.g., Reference Fessler, Arguello, Mekdara and MaciasFessler, Arguello, Mekdara & Macias, 2003; Reference Royzman, Leeman and BaronRoyzman, Leeman & Baron, 2009; see also Reference Royzman, Kim and LeemanRoyzman, Kim & Leeman, 2015) that found no significant association between disgust sensitivity and moral evaluations when special design precautions (time delay, misdirection, appropriate controls) were in place. Our own (Reference Costello and OsborneRoyzman et al., 2017) study found no relationship between the fine-tuned measure of disgust as OI and either GMF or moral purity concerns (as measured by MFQ-P, Reference Graham, Nosek, Haidt, Iyer, Spassena and DittoGraham et al., 2011]; r = .14, p = .09).Footnote 8 The findings assembled in recent years have lead another research group to conclude that “there is little evidence for a unique predictive effect of physical, pathogen-based disgust experiences on moral judgments” (Reference Piazza, Landy, Chakroff, Young, Wasserman, Strohminger and KumarPiazza, Landy, Chakroff, Young & Wasserman, 2018, p. 75). From the Bayesian point of view, one’s a priori expectation that disgust predicts GMF moralization should be fairly low.

By contrast, our expectation that people will be principally worried or suspicious about the consequences of GM food should be high. Many studies have provided evidence that people critique GM food based on their unease about the possible harm that genetically modified food may have on the populace, and express a suspicion that scientists have not studied GM sufficiently or long enough to rule out negative effects (e.g., Reference Chen and LiChen & Li, 2007; Reference Connor and SiegristConnor & Siegrist, 2010; Reference Onyango and NaygaOnyango & Nayga, 2004; Reference Prati, Pietrantoni and ZaniPrati, Pietrantoni & Zani, 2012; Reference Rzymski and KrólczykRzymski & Krolczyk, 2016). Indeed, a review of subjects’ verbatim statements in the OERM condition of our study, indicates that 60 percent of the absolutist opponents and 65 percent of all opponents were thinking uniquely in terms of harm and risk.

All in all, our study (Reference Costello and OsborneRoyzman, et al., 2017) was methodologically and conceptually sound. Our critique, as well as our novel findings, fit neatly with the recent pattern of findings showing that disgust plays little role in moral judgment. The burden of proof is on those who wish to claim otherwise.

Appendix: Study re-assessing the prevalence of state-disgust

Method

Subjects.

We recruited 1048 individuals from Amazon’s Mechanical Turk (561 reported Female, AgeM = 35.7 AgeSD = 12.23).

Procedure and Design.

At the beginning of the study, all subjects were queried about whether they opposed or did not oppose “genetically engineering plants and animals for food production” (thus separating opponents from non-opponents) and whether they thought it “should be prohibited no matter how great the benefits and minor the risks of allowing it” (with the affirmative answer leading to the subject being classified as an opposing absolutist). This allowed us to sort each subject in one of the three categories: absolutist opponent, non-absolutist opponent, non-opponent. Then, after completing a brief attention check, subjects were randomly assigned to one of the three affect report conditions described below: (1) a two-item forced choice condition (2-item FC) serving as a replication of the study reported in SIR, (2) a 12-item forced choice condition (12-item FC), or (3) an open-ended response condition (OERM).

In the 2-item FC (modeled after Scott et al., 2016) and 12-item FC conditions, subjects were asked to indicate which of the available affect terms best described what they felt while thinking about genetically engineering plants and animals for food production. In 2-item FC, the options were limited to Angry and Disgusted. In the 12-item condition, the options were considerably broader: Afraid, Angry, Bothered, Confused, Creeped out, Excited, Frustrated, Grossed outFootnote 9, Neutral, Sad, Suspicious, and Worried.

We reasoned that “worry”, “suspicion”, and other aforementioned terms would constitute better alternatives to disgust than “anger” (or “fear”). The bulk of the published data on GMF attitudes shows that people are concerned about the possible harmful consequences that these products may have on their well-being or health, and are suspicious that scientists have studied GM sufficiently or long enough to rule out key negative effects (e.g., Reference Chen and LiChen & Li, 2007; Reference Connor and SiegristConnor & Siegrist, 2010; Reference Onyango and NaygaOnyango & Nayga, 2004; Reference Prati, Pietrantoni and ZaniPrati, Pietrantoni & Zani, 2012; Reference Rzymski and KrólczykRzymski & Krolczyk, 2016). And, as pointed out by Jonathan Baron (personal communication, June 15, 2018), “worry” is also the affective category most commonly cited in Slovic’s “psychometric approach to risk” studies of lay responses to emerging technologies and medicine (e.g., Reference Peters, Slovic, Hibbard and TuslerPeters, Slovic, Hibbard & Tussler, 2006; Reference SlovicSlovic, 2000). Consistent with Slovic, we see worry largely as the ruminative cousin of fear, arising as part of the affective sequelae of prospecting (Reference Seligman, Railton, Baumeister and SripadaSeligman, Railton, Baumeister & Sripada, 2013) future threats over which one can exert some (but only partial) control.

The two forced choice conditions were alike in that all the subjects had to feel something, and it was up to us, the researchers, what feelings they could express. By contrast, in the third open-ended response mode (OERM) condition, we left open the possibility that people do not feel anything at all. Rather, in this condition, subjects were prompted to “… use the space below to describe (as you would in a diary) what is going though your mind and what you are experiencing as you are thinking about the practice of genetically engineering plants and animals for food production.”

The subjects were further enjoined to note that there are “no right or wrong answers” and not to “hold back or self-censor in any way”Footnote 10. This circumlocution was motivated by several prior reports (reviewed by Feldman Barrett, 2017) suggesting that framing an affect-related question in terms of specific emotion concepts or feelings may result in an overestimate of a particular affective state due to subjects’ constructing their internal experiences around the concepts in question. Thus, the OERM condition was the most conservative (assumption-free) of the three experimental conditions in that it made no direct reference to affect.

Emotion coding.

To diagnose the prevalence of affective language in OERM, we carried out a series of content analyses using a total of 81 word stems corresponding to the feeling concepts tapped in the 12-item FC condition of the study. 80 of these were organized into 8 item-clusters detailed in Table 2.Footnote 11 (No affective cluster was formed around “bothered” due the extremely low granularity of the term.) Due to its precise classification being the present point of contention, “creeped out” was retained as a “free-standing” term, though, in our view, the evidence from prior research (Reference Costello and OsborneRoyzman et al., 2017; Reference McAndrew and KoehnkeMcAndrew & Koehnke, 2016), clearly substantiates its inclusion in the Fear (Afraid + Worry + Suspicion) super-cluster.

Table 2: 81-word affective word stems used in the OERM condition of the study.

Note. In all cases, subjects either used a single affective term or two affective terms from the same cluster, making all responses assignable to a single cluster.

Results and discussion

124 subjects (12% of our entire sample) failed a brief attention check and were removed. Including these individuals has no noticeable effect on the results. All analyses reported below were performed on the remaining 922 subjects. Of these subjects, 424 (46%) indicated that they opposed GMF. Of people who indicated that they currently oppose GMF, 312 (74%) agreed with the statement that GMFs should be prohibited “no matter what”, and thus would be considered as “absolutist” opposers by SIR’s criterion.

As we predicted, SIR’s procedure for measuring state disgust overestimates the absolute prevalence of disgust. When subjects’ choices were limited to disgust and anger, disgust substantially and significantly dominated non-disgust for absolute opponents (73.3 percent) as well as for the sample as a whole (62.2 percent; see Table 1, main text). By contrast, only 2.8% of the sample chose “grossed out” in the 12-item FC condition (Table 3), and only 1.7% of the sample mentioned one of the target disgust-related words in the OERM condition (Table 4), both significant drops from 62.2 percent, χ2(1, 633) = 256.55, p< .0001 and χ2(1, 598) = 246.5, p< .0001, respectively (based on the “N-1” Chi-squared test as recommended by Campbell, 2007, and Richardson, 2011]), with no significant difference between the 12-item condition and OERM (χ2[1, 633] = 0.82, p = 0.36).

Table 3: Frequency of feeling states selected in the 12-emotion forced choice condition.

Note. Rows ordered by rank frequency in total sample. “Grossed out” highlighted to indicate rank.

Table 4: Frequency of feeling states selected in the open response choice format condition.

Note. Rows ordered by rank frequency in total sample. Disgust highlighted to indicate rank.

Disgust does not appear to be an especially prominent feeling for people absolutely opposed to GMF, either. Only 6.7% of these subjects selected “grossed out” and only 4.3% of these subjects provided one of the disgust-cluster words in OERM (with 4 out of 5 cases using the non-specific word “disgust”, which commonly means merely “disapproval”). These both represent substantial decreases from SIR’s 2-condition method,χ2(1, 220) = 99.51, p< .0001 and χ2(1, 208) = 99.31, p< .0001, respectively (with no significant difference between 12-item FC and OERM, χ2[1, 196] = 0.53, p = 0.46). In contrast to disgust, other feelings, e.g., worry, suspicion, and apathy were far more common in people’s reactions. The open-ended response format, which provides the most assumption-free approach to the problem, suggests that very few people spontaneously perceived themselves as experiencing any emotion at all (41 out of 288, 14.2%), though opponents appeared somewhat more likely to do so (24/140, 17%) than non-opponents (14/148, 9.5%), χ2(1, 288) = 3.07, p = .08.

For those apt to experience any feeling at all, feeling worried appears to be three times more common than feeling disgusted (Table 4), though even worry comprises a relatively small proportion of the overall sample (5.9% for worry vs 1.7% for disgust). The sheer paucity of affect under OERM is in striking contrast to the conclusions one might have drawn from the 2-item FC. Finally, it is worth noting that 1.7 percent (the absolute prevalence of disgust in the open-ended condition) is likely to be an overestimate. As indicated earlier, in everyday speech, “disgust” or “disgusting” are often used to signify “immoral”, “bothered”, or “disliked”. This also likely overestimates the extent to which these feelings would be causally implicated in (or be even temporally antecedent to) the rise of absolutist disapproval. Thus, a truly conservative estimate of spontaneously experienced and causally relevant state disgust (disgust quo OI) is likely to be closer to zero.

Disgust words vs. disgust faces

There is little doubt that IS/SIR would challenge these results. They could, for example, argue that their 2016 assessment was not predicated solely on emotion words. In their original study (Reference Ekman, Matsumoto, Friesen, Ekman and RosenbergScott et al., 2016), some subjects indicated their preference by being asked to select from a set of facial displays: two anger faces (the “maximal” anger face in the open mouth form with bilateral lip raise and the “minimal” anger face in the lip press form) and two disgust faces (the “maximal” gape/tongue protrusion disgust face and the “minimal” bilateral upper lip raise disgust face).

There are a number of problems with this method – key among them is the inclusion of the minimal disgust face as one of the bases for demarcating disgust from anger. As Reference Rozin, Lowery and EbertRozin et al. (1994) themselves have observed,

“Among disgust situations, upper lip raise [AU 10] was the significant (p < .01) predominant response in two cases: ‘seeing a deformed person’ and ‘looking at pictures of the slaughter at a WWII concentration camp’…Upper lip raise was also the predominant response for anger, contempt, and rights violation (stealing), confirming its role in the anger expression. Thus…the major results of the choice test… affirm a link between upper lip raise and other moral emotions.” (p. 878).

According to Rozin et al.’s own data (see also Reference Widen, Pochedly, Pieloch and RussellWiden et al., 2013), the upper lip raise AU was not selectively related to the experience of disgust. Thus, it appears that, as with “disgust” itself, one of the constituent disgust displays lacks discriminant validity as a gauge of disgust vs. anger, contempt, and disapproval as well as any capacity to differentiate between physical and socio-moral domains.

To address this concern, we re-analyzed Scott et al.’s study comparing the maximal (gape) disgust face with its maximal anger counterpart.Footnote 12 With the minimal disgust and anger faces removed, we no longer observed significant anger/disgust differences for any of the four scenarios. This was the case irrespective of whether we looked at the sample as a whole or limited our analyses to GMF opponents only.

One could also argue that forcing subjects to choose between two types of facial configurations that visually stereotype two folk emotion concepts inherits all or most of the problems we discussed above in the context of emotion words. These include the key criticism that applied to the forced-choice approach: people are forced to “feel” something (leaving no room for the hypothesis that they process the information in an emotion-free way). And the most likely emotions, including suspicion, worry, or apathy, are clearly not on the list. The face of fear is fundamentally a face of terror and there is simply no established facial configuration for the lower-activation states of “worry”, “suspicion”, or “concern”.

Finally, because Inbar and Scott’s presentation of their newer findings emphasize faces over words, it is worth noting that many of the theoretical and methodological assumptions underlying this method have been strongly and repeatedly questioned (see Reference RussellRussell, 1994, 2003; Feldman Barrett, 2017 for discussion). In our view, the use of facial displays is still of historical interest, but its current scientific status is so unsettled that its use as a means to resolve yet another scientific debate is contentious at best.

Footnotes

Note. In all cases, subjects either used a single affective term or two affective terms from the same cluster, making all responses assignable to a single cluster.

Note. Rows ordered by rank frequency in total sample. “Grossed out” highlighted to indicate rank.

Note. Rows ordered by rank frequency in total sample. Disgust highlighted to indicate rank.

1 To be precise, the claim is that disgust is oral inhibition that is about the world. OI with no external referent would not be disgust but simply dyspepsia or “feeling sick.”

2 For instance, Reference Tybur, Lieberman and GriskeviciusTybur et al. (2009) observed that subjects reported equally high levels of “disgust” in response to pathogen-linked elicitors (M = 3.87) and so-called moral disgust elicitors with no apparent pathogen-linked content (M = 3.70). By contrast, in a different study, such “moral disgust elicitors” elicited no OI unless they also featured pathogen-linked content (Reference Royzman, Atanasov, Landy, Parks and GeptyRoyzman et al., 2014). Reference Royzman, Leeman and SabiniRoyzman et al. (2008) found that, though people used the word “disgust” in response to pathogen-linked elicitors (e.g., representations of waste, gore, and sex) as well as the stimuli with no apparent pathogen-linked content (e.g., a photo of Hitler), they reported OI only towards the pathogen-linked content.

3 To show a good fit for the model, the chi-square statistic should be non-significant (Reference Floyd and WidamanFloyd & Widaman, 1995). The CFI should range between 0 and 1, with values .95 or higher representing the best model fit (Reference Hu and BentlerHu & Bentler, 1999; Reference McDonald and HoMcDonald & Ho, 2002). To indicate good fit, the SRMR should be less than .08 (Reference Hu and BentlerHu & Bentler, 1999). For the RMSEA, Reference Brown, Cudeck, Bollen and LongBrown and Cudeck (1993), a figure less than or equal to .08, indicates reasonable fit while RMSEA should never be greater than .10 (Reference Brown, Cudeck, Bollen and LongBrown & Cudeck, 1993).

4 Consistent with this, Reference Blake, Yih, Zhao, Sung and Harmon-JonesBlake et al. (2016) report oral inhibition reactions to items closely matching those featured in TDDS-P.

5 This was supported by prior work, as well as as a supplementary “word meaning check” (Reference Costello and OsborneRoyzman et al., 2017) in which roughly one-third of the sample filled out a brief questionnaire about the meaning of “creeped out”. An overwhelming majority (83%) agreed that “creeped out” meant “feeling nervous/uneasy” and another 81% associated it with “feeling like there is a possible threat”.

6 For instance, we find it plausible that people’s trait risk-sensitivity influence GMF opposition in affectless ways: for example, people with higher trait disgust-sensitivity may make different lifestyle choices than someone with low trait disgust-sensitivity, including associating with different people, that results in different moral attitudes down the line.

7 Reference Scott, Inbar and RozinScott et al. (2016) make a similar set of assumptions in their experiments using facial displays (see Appendix for discussion).

8 By contrast, disgust-the-word did rather well, serving as a significant predictor of both opposition to GM-food (r = .25, p = .004) and MFQ-P (r = .21, p = .02).

9 While “grossed out” is undoubtedly more specific than “disgust”, it is still not a viable alternative to OI. Reference NabiNabi’s (2002) Table 3 indicates that, though the affective experience tapped by “grossed out” is more strongly related to gustatory discomfort that “disgusted”, it is also related to general avoidance, and some desire to retaliate. Moreover, since “grossness” is an attribute of textures and tastes alike, “grossed out” does not appear to be fit to differentiate between OI and epidermal discomfort. Thus, even narrowing the linguistic scope of disgust to “grossed out” is less than ideal for those wishing to capture the experience in its most pure form.

10 A preparatory focus group discussion indicated that, though both locutions — “going through [their] mind” vs. what one is “experiencing” — were readily understood as evincing an interest in the sum total of one’s mental states, the former was perceived as conveying a stronger interest in one’s “thoughts”, with the latter conveying a stronger interest on one’s “feelings.” Consistent with this analysis, the two locutions were counterbalanced for order within the prompt.

11 Following the initial search, one of the authors conducted a thorough overview of each individual statement, noting any additional affective terms not acquired by the initial search. This follow-up search identified two potential references to an affective state (“hopeful” and “squeamish”, respectively). Neither term was included in the final analyses.

12 We are grateful to Sydney Scott for providing access to their original facial display data and for conducting the preliminary analyses based on that data.

References

Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238246.CrossRefGoogle ScholarPubMed
Blake, K. R., Yih, J., Zhao, K., Sung, B., & Harmon-Jones, C. (2016). Skin-transmitted pathogens and the heebie jeebies: evidence for a subclass of disgust stimuli that evoke a qualitatively unique emotional response. Cognition and Emotion, 31(6), 11531168.CrossRefGoogle ScholarPubMed
Brown, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In Bollen, K. A., & Long, J. S. (Eds.), Testing Structural Equation Models (pp. 136162). Newbury Park, CA: Sage.Google Scholar
Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, NY, US: Guilford Press.Google Scholar
Campbell, I. (2007) Chi-squared and fisher-irwin tests of two-by-two tables with small sample recommendations. Statistics in Medicine, 26(19), 36613675.CrossRefGoogle ScholarPubMed
Chan, C., Boven, L. V., Andrade, E. B., & Ariely, D. (2014). Moral violations reduce oral consumption. Journal of Consumer Psychology, 24(3). 381386.CrossRefGoogle ScholarPubMed
Chapman, H. A., Kim, D. A., Susskind, J. M., & Anderson, A. K. (2009). In bad taste: evidence for the oral origins of moral disgust. Science, 323(5918), 12221226.CrossRefGoogle ScholarPubMed
Chen, M., & Li, H. (2007). The consumer’s attitude toward genetically modified foods in taiwan. Food Quality and Preference, 18(4), 662674.CrossRefGoogle Scholar
Cisler, J. M., Olatunji, B. O., & Lohr, J. M. (2009). Disgust sensitivity and emotion regulation potentiate the effect of disgust propensity on spider fear, blood-injection-injury fear, and contamination fear. Journal of Behavior Therapy and Experimental Psychiatry, 40(2), 219229.CrossRefGoogle ScholarPubMed
Connor, M., & Siegrist, M. (2010), Factors influencing people’s acceptance of gene technology: the role of knowledge, health expectations, naturalness, and social trust. Science Communication, 32(4), 514538.CrossRefGoogle Scholar
Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10(7), 173178.Google Scholar
Curtis, V., de Barra, M., & Aunger, R. (2011). Disgust as an adaptive system for disease avoidance behaviour. Philosophical Transactions of the Royal Society B: Biological Sciences, 366(1563), 389401.CrossRefGoogle ScholarPubMed
Ekman, P. (2003). Emotions revealed: Recognizing faces and feelings to improve communication and emotional life. New York, NY: Henry Holt.Google Scholar
Ekman, P., Matsumoto, D., & Friesen, W. V. (2005). Facial expression in affective disorders. In Ekman, P. & Rosenberg, E. L. (Eds.), Series in affective science. What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (FACS) (2nd ed.). (pp. 429440). New York, NY, US: Oxford University Press.Google Scholar
Feldman Barrett, L. How emotions are made: The secret life of the brain. (2017). New York, New York: Houghon Mifflin Harcourt Publishing Company.Google Scholar
Fessler, D. M., Arguello, A. P., Mekdara, J. M., & Macias, R. (2003). Disgust sensitivity and meat consumption: A test of an emotivist account of moral vegetarianism. Appetite, 41(1), 3141.CrossRefGoogle Scholar
Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7(3), 286299.CrossRefGoogle Scholar
Graham, J., Nosek, B. A., Haidt, J. Iyer, R., Spassena, K., Ditto, P. H. (2011). Mapping the moral domain. Journal of Personality and Social Psychology, 101(2), 366385.CrossRefGoogle ScholarPubMed
Haidt, J., McCauley, C., & Rozin, P. (1994). Individual differences in sensitivity to disgust: A scale sampling seven domains of disgust elicitors. Personality and Individual Differences, 16(5), 701713.CrossRefGoogle Scholar
Hatcher, L. (1994). A step-by-step approach to using the SAS system for factor analysis and structural equation modeling. Cary, NC: The SAS Institute.Google Scholar
Hu, L. & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: a Multidisciplinary Journal, 6(1), 155.CrossRefGoogle Scholar
Inbar, Y. & Scott, S. E. (2018). (in press). People respond to gm food with disgust more than fear: Comment on Royzman, Cusimano, and Leeman (2017). Judgment and Decision Making, 13.CrossRefGoogle Scholar
Johnson, D. J., Cheung, F., & Donnellan, M. B. (2014). Does cleanliness influence moral judgments? A direct replication of Schnall, Benton, and Harvey (2008). Social Psychology, 45(3), 209215.CrossRefGoogle Scholar
Kayyal, M. H., Pochedly, J., McCarthy, A., & Russell, J. A. (2015). On the limits of the relation of disgust to judgments of immorality. Frontiers in Psychology, 6, 951.CrossRefGoogle ScholarPubMed
Kupfer, T. & Fessler, D. (2018). Ectoparasite defence in humans: Relationships to pathogen avoidance, and clinical implications. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1751).CrossRefGoogle ScholarPubMed
Landy, J. F., & Goodwin, G. P. (2015). Does incidental disgust amplify moral judgment? A meta- analytic review of experimental evidence. Perspectives on Psychological Science, 10(4), 518536.CrossRefGoogle ScholarPubMed
Landy, J. F., & Piazza, J. (in press). Reevaluating moral disgust: sensitivity to many affective states predicts extremity in many evaluative judgments. Social Psychological and Personality Science.Google Scholar
McAndrew, F. T., & Koehnke, S. S. (2016). On the nature of creepiness. New ideas in psychology, 43, 1015CrossRefGoogle Scholar
McDonald, R. P., & Ho, M. H. (2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7(1), 6482.CrossRefGoogle ScholarPubMed
Muth, E. R., Stern, R. M., & Koch, K. L. (1998). A psychophysiological paradigm for the study of nausea: A rotating optokinectic drum, the electrogastrogram, and the nausea profile. Journal of Psychophysiology, 12, 5663.Google Scholar
Murray, D. R., & Schaller, M. (2016). The behavioral immune system: Implications for social cognition, social interaction, and social influence. In S, Mahfould (Eds.), Advances in Experimental Social Psychology, 53, 75129.CrossRefGoogle Scholar
Nabi, R. L. (2002). The theoretical versus the lay meaning of disgust: implications for emotion research. Cognition and Emotion, 16(5), 695703.CrossRefGoogle Scholar
Newton, P. E., & Shaw, S. D. (2013). Standards for talking and thinking about validity. Psychological Methods, 18(3), 301319.CrossRefGoogle ScholarPubMed
Olatunji, B. O., Sawchuk, C. N. (2005). Disgust: Characteristic features, social manifestations, and clinical implications. Journal of Social and Clinical Psychology, 24(7), 932962.CrossRefGoogle Scholar
Olatunji, B. O., Williams, N. L., Tolin, D. F., Abramowitz, J. S., Sawchuk, C. N., Lohr, J. M., & Elwood, L. S. (2007). The disgust scale: item analysis, factor structure, and suggestions for refinement. Psychological Assessment, 19(3), 281297.CrossRefGoogle Scholar
Onyango, B., & Nayga, R. (2004). Consumer acceptance of nutritionally enhanced genetically modified food: relevance of gene transfer technology. Journal of Agricultural and Resource Economics, 29(3), 567583.Google Scholar
Peters, E., Slovic, P., Hibbard, J. H., & Tusler, M. (2006). Why worry? Worry, risk perceptions, and willingness to act to reduce medical errors. Health Psychology, 25(2), 144152.CrossRefGoogle ScholarPubMed
Piazza, J., Landy, J. F., Chakroff, A., Young, L., & Wasserman, E. (2018). What disgust does and does not do for moral cognition. In Strohminger, N. & Kumar, V. (Eds.), The Moral Psychology of Disgust (pp. 5381). New York: Rowman & Littlefield.Google Scholar
Prati, G., Pietrantoni, L., & Zani, B. (2012). Influenza vaccination: The persuasiveness of messages among people aged 65 years and older. Health Communication, 27(5), 413420.CrossRefGoogle ScholarPubMed
Richardson, J. T. (2011). The analysis of 2 X 2 contingency tables - yet again. Statistics in Medicine, 30(8), 890.CrossRefGoogle Scholar
Royzman, E. B., Atanasov, P., Landy, J. F., Parks, A., & Gepty, A. (2014). CAD or MAD? Anger (not disgust) as the predominant response to pathogen-free violations of the divinity code. Emotion, 14(5), 892907.CrossRefGoogle ScholarPubMed
Royzman, E. B., Cusimano, C., & Leeman, R. F. (2017). What lies beneath? Fear vs. disgust as affective predictors of absolutist opposition to genetically modified food and other new technologies. Judgment and Decision Making, 12(5), 466480.CrossRefGoogle Scholar
Royzman, E. B., Kim, K., & Leeman, R. F. (2015). The curious tale of Julie and Mark: Unraveling the moral dumbfounding effect. Judgment and Decision Making, 10(4), 296313.CrossRefGoogle Scholar
Royzman, E. B., Leeman, R. F., & Baron, J. (2009). Unsentimental ethics: Towards a content- specific account of the moral-conventional distinction. Cognition, 112(1), 159174.CrossRefGoogle ScholarPubMed
Royzman, E. B., Leeman, R. F., & Sabini, J. (2008). “You make me sick”: Moral dyspepsia as a reaction to third-party sibling incest. Motivation and Emotion, 32(2), 100108.CrossRefGoogle Scholar
Royzman, E. B., & Sabini, J. (2001). Something it takes to be an emotion: The interesting case of disgust. Journal for the Theory of Social Behavior, 31(1), 2959.CrossRefGoogle Scholar
Rozin, P., & Fallon, A. E. (1987). A perspective on disgust. Psychological Review, 94(1), 2341.CrossRefGoogle ScholarPubMed
Rozin, P., Haidt, J., & Fincher, K. (2009). From oral to moral: Is moral disgust an elaboration of a food rejection system. Science, 323(5918), 11791180CrossRefGoogle Scholar
Rozin, P., Haidt, J., & McCauley, C. R. (1993). Disgust. In Lewis, M. & Haviland, J. (Eds.). Handbook of emotions (pp. 575594). New York: Guilford.Google Scholar
Rozin, P., Haidt, J., & McCauley, C. R. (2000). Disgust. Entry in: Levinson, D., Ponzetti, J., & Jorgenson, P. (eds.) Encyclopedia of human emotions. Volume 1 (second edition) (pp. 188193). New York: Macmillan.Google Scholar
Rozin, P., Lowery, L., & Ebert, R. (1994). Varieties of disgust faces and the structure of disgust. Journal of Personality and Social Psychology, 66(5),870888.CrossRefGoogle ScholarPubMed
Russell, J. A. (1994). Is there universal recognition of emotion from facial expression? A review of cross-cultural studies. Psychological Bulletin, 115, 102141.CrossRefGoogle ScholarPubMed
Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145172.CrossRefGoogle ScholarPubMed
Rzymski, P., & Królczyk, A. (2016). Attitudes toward genetically modified organisms in poland: to gmo or not to gmo? Food Security, 8(3), 689697.CrossRefGoogle Scholar
Scott, S. E., Inbar, Y., & Rozin, P. (2016). Evidence for moral opposition to genetically modified food in the united states. Perspectives on Psychological Science, 11(3), 315324.CrossRefGoogle ScholarPubMed
Seligman, M. E.P., Railton, P., Baumeister, R. F., & Sripada, C. (2013). Navigating into the future or driven by the past. Perspectives on Psychological Science, 8(2), 119141.CrossRefGoogle ScholarPubMed
Sherman, G., Haidt, J., & Coan, J. ( (2007). Nazis really are disgusting: Psychophysiological evidence for sociomoral disgust. Unpublished manuscript. University of Virginia.Google Scholar
Slovic, P. (Ed.). (2000). Risk, society, and policy series. The perception of risk. London, England: Earthscan Publications.Google Scholar
Tybur, J. M., Lieberman, D., Griskevicius, V. (2009). Microbes, mating, and morality: individual differences in three functional domains of disgust. Journal of Personality and Social Psychology, 97(1), 103122.CrossRefGoogle ScholarPubMed
van Huis, A. (2017). Did early humans consume insects? Journal of Insects as Food and Feed, 3(3), 161163.CrossRefGoogle Scholar
Watt, M. C., Maitland, R. A., & Gallagher, C. E. (2017). A case of the “heeby jeebies”: An examination of intuitive judgements of “creepiness”. Canadian Journal of Behavioural Science / Revue canadienne des sciences du comportement, 49(1), 5869.CrossRefGoogle Scholar
Widen, S. C., Pochedly, J. T., Pieloch, K., Russell, J. A. (2013). Introducing the sick face. Motivation and Emotion, 37(3), 550557.CrossRefGoogle Scholar
Woody, S. R., & Teachman, B. A. (2000). Intersection of disgust and fear: Normative and pathological views. Clinical Psychology: Science and Practice, 7(3), 291311.Google Scholar
Figure 0

Table 1: Proportion of subjects choosing/using disgust-related terms as a function of study condition and evaluative orientation (absolute opposition, non-absolute opposition, non-opposition).

Figure 1

Table 2: 81-word affective word stems used in the OERM condition of the study.

Figure 2

Table 3: Frequency of feeling states selected in the 12-emotion forced choice condition.

Figure 3

Table 4: Frequency of feeling states selected in the open response choice format condition.

Supplementary material: File

Cusimano et al. supplementary material

Cusimano et al. supplementary material 1
Download Cusimano et al. supplementary material(File)
File 90.9 KB
Supplementary material: File

Cusimano et al. supplementary material

Cusimano et al. supplementary material 2
Download Cusimano et al. supplementary material(File)
File 101.7 KB