Semantic differences in visually similar face emojis

Abstract

The literature on face emojis raises the central question whether they should be treated as pictures or conventionalized signals. Our experiment addresses this question by investigating semantic differences in visually similar face emojis. We test a prediction following from a pictorial approach: small visual features of emojis that do not correspond to human facial features should be semantically less relevant than features that represent aspects of facial expressions. We compare emoji pairs with a visual difference that either does or does not correspond to a difference in a human facial expression according to an adaptation of the Facial Action Coding System. We created two contexts per pair, each fitted to correspond to a prominent meaning of one or the other emoji. Participants had to choose a suitable emoji for each context. The rate at which the context-matching emoji was chosen was significantly above chance for both types of emoji pairs and it did not differ significantly between them. Our results show that the small differences are meaningful in all pairs whether or not they correspond to human facial differences. This supports a lexicalist approach to emoji semantics, which treats face emojis as conventionalized signals rather than mere pictures of faces.

1. Introduction

Face emojis are stylized representations of human facial expressions, which have become an integral part of written digital communication. Written communication lacks signals of face-to-face communication, such as facial expressions and gestures. This is one important reason why emojis, which can take over many of the functions of such body movements, have become so popular. They have been argued to fulfill the role of gestures (McCulloch & Gawne, 2018; Pasternak & Tieu, 2022; Pierini, 2021) facial expressions (e.g., Weiß et al., 2020), particles (Beisswenger & Pappert, 2019), or discourse connectives (Beisswenger & Pappert, 2019) and to facilitate the understanding of indirect meaning (Holtgraves & Robinson, 2020).

While initial research on emojis was of a predominantly descriptive or applied nature (e.g., Kralj Novak et al., 2015), typically focusing on emoji use, recent years have seen the emergence of a more theoretically oriented literature. For example, Grosz et al. (2023) and Kaiser and Grosz (2021) present an analysis of action emojis as event descriptions, which are similar to free adjuncts and are connected to the surrounding text via discourse relations. Grosz et al. (2023) show how face emojis comment on a target proposition by expressing how it relates to a discourse value held by the author. Cohn et al. (2019) studied the grammatical structure of emojis. They found that word-replacing emojis mostly replace nouns and adjectives in texts and that emoji sequences only show basic linear grammar patterns.

Another line of research focuses on the processing of emojis. Weissman and Tanner (2018) found in an ERP experiment that statements marked as ironic with a winking face emoji are processed similarly to statements with verbal irony. Scheffler et al. (2022) present a self-paced reading study, which shows that emojis replacing nouns in a sentence can represent full lexical entries, including both meaning and phonological representations. They do, however, give rise to increased reading times compared to words. Tang et al. (2020) compared the processing of incongruous emoji-text combinations to that of incongruous words in sentences in an ERP study. Their results suggest that emoji meaning is more difficult to process than word meaning. Weissman et al. (2023) found evidence for a high degree of lexical entrenchment for emojis with conventionalized meanings. According to experiments presented by Weissman (2022) and Weissman (2024), such emojis commit the author of a message to their meaning, and they can be used to tell a lie.

Moreover, a number of studies investigated the potential of face emojis to convey emotions as well as their link to human facial expressions. Gantiva et al. (2020) studied the cortical processing of face emojis and human faces in an ERP study and found that face emojis elicit neural responses that are similar to those caused by human faces. Boutet et al. (2023) compared existing emojis, photos of human faces and novel emojis, which included additional visual features of human facial expressions with regard to their potential to convey six basic emotions. Their results suggest that adding visual features from facial expressions increases the accuracy with which an emotion can be recognized. However, Kendall et al. (2016) compared schematic cartoon faces with photos of human faces in an experimental study and found that at short exposure times, emotions conveyed by the former were identified with a higher accuracy than emotions conveyed by the latter. The authors conclude that simplified iconic representation can convey emotional information more efficiently than realistic pictures. Jaeger et al. (2019) asked participants to rate the emotional valence and arousal associated with emojis. They conclude that facial emojis encompass a broad spectrum of emotional valence and arousal, and that they can communicate a diverse array of emotions. However, a study by Franco and Fugate (2020) identified differences between different platform renderings of the same emoji in terms of which emotions they are interpreted to convey.

A crucial question that concerns the very nature of emoji semantics has, however, not been directly addressed in the experimental research on emojis so far. It is the question whether the meaning of face emojis without symbolic components (such as dollar signs) is conventionalized like the meaning of linguistic signals, or whether it is purely iconic in that emojis should be understood as text-accompanying pictures. The study we present in this paper addresses this question by investigating semantic differences in visually similar face emojis, e.g., and (neutral face, expressionless face). Note that while the emojis displayed in the text of this paper are the emojis from the LaTeX emoji package, we tested WhatsApp emojis in our experiment.

According to the pictorial approach (Maier, 2023), which proposes purely iconic semantics for emojis, face emojis are pictures of faces, which are interpreted to represent the face of the author of the message. For example, Maier (2023, p. 31) argues that is a stylized picture of a face and that pragmatic enrichment gives rise to the understanding that this is a rendering of what the author’s face looks like when typing the emoji. Maier (2023, p. 10) further states that there is ‘a “natural” correspondence between shapes in the sign (specifically the shapes of mouth and eyes) and properties of the speaker’s face and/or emotional state.’ Such a view makes predictions for whether two face emojis map onto the same facial expression or different ones. Consider, for example, the two emojis and , which both represent smiling, happy faces. Note that the former emoji features eyebrows in a neutral position while the latter does not. As human faces usually have eyebrows, we can assume that the two emojis map onto the same facial expression in a human face. In contrast, the two emojis and clearly map to different facial expressions as the former emoji represents open and the latter closed eyes. Thus, we draw the following inference from the pictorial account: small visual differences between two emojis should have a privileged status in terms of semantic relevance if they correspond to differences in human faces; by contrast, differences with no correspondence to human facial features should have less semantic relevance.

By contrast, a lexicalist approach emphasizes that emojis are associated with conventionalized meanings. Cohn and Ehly (2016) describe what they call the Japanese Visual Language, which is used in mangas and whose components also occur in emojis (see also Cohn, 2013). They use the label ‘morpheme’ for ‘units of meaningful visual representation’, that is, eye and mouth shapes as well as teardrops or snot bubbles (Cohn & Ehly, 2016, p. 20). For example, a snot bubble on the nose symbolizes sleep. Relatedly, Grosz et al. (2023) treat emojis as a variant of emotive expressions and thus as loosely equivalent to conventionalized linguistic signals (see also Grosz et al., 2023 and Kaiser & Grosz, 2021 who argue that face emojis are similar to expressives). They propose that the meaning of can be rendered in a way similar to the meaning of the interjections yay or whee; the contribution of can be compared with the contribution of a swear word like damn. While it is beyond dispute that face emojis exhibit stylized features of a human face, the lexicalist approach allows for the possibility that the minimal units an emoji consists of (e.g., eye or mouth shapes) carry partially conventionalized meaning regardless of whether they are plausible depictions of (components of) human faces. Accordingly, small differences in emojis can be meaningful, even if they do not represent differences in human faces. While we focus on face emojis in this paper, the pictorial and the lexicalist approach also cover non-face emojis. According to Maier (2023, p. 22), object emojis convey truth-conditional content and normally ‘depict what the evaluation world looks like’. However, Maier acknowledges that object emojis can also convey metaphorical and metonymic meanings. Grosz et al. (2023) present an analysis of object emojis in sentence-final position as event descriptions with an open argument slot by which they are anaphoric to an implicit agent. Thus, also with regard to non-face emojis, the two accounts focus either on their pictorial or linguistic quality.

Whether the difference between visually similar face emojis maps to a difference in corresponding human facial expressions can be captured via the Facial Action Coding System (FACS; Ekman & Friesen, 1978), which decomposes facial expressions into minimal Action Units (AUs) and has been adapted for the annotation of face emojis by Fugate and Franco (2021). Against the background of the FACS, we can distinguish two types of pairs of visually similar emojis: Emoji pairs with a visual difference that corresponds to an AU difference ([AU+]), like the pair and (grinning face with big eyes, grinning squinting face) and emoji pairs with a visual difference that does not correspond to an AU difference ([AU−]), like the pair and (grinning face with smiling eyes, beaming face with smiling eyes).

The latter pair shares the AUs 12 + 25 + 26 + 63, representing an arched-up mouth with an open mouth space and eyes looking up, but does not differ in any other Action Units. We followed Fugate and Franco (2021) in coding the eyes as AU 63 ‘eyes looking up’. Alternatively, this eye shape could be interpreted as AU 6 ‘cheek raiser’ in both emojis. The two emojis’ visual difference lies in the presence of lower teeth in beaming face with smiling eyes. Whether lower teeth show in a smile with an open mouth space is not something a person can manipulate, which is why this is not captured in the FACS coding. The pair and shares the same AUs with regard to the mouth, that is, AU 12 + 25 + 26, which represent an arched-up mouth with an open mouth space, but they have different AUs for their eyes. The eye shape of is coded as AU 5 ‘eyes wide’, whereas the eye shape of is coded as AU 43 ‘eyes closed’.

In our experiment, we investigate whether the two types of visual differences ([AU+] and [AU $ - $ ] differences) lead to differences in meaning between the two emojis of a pair. Thereby, we assess the explanatory power of the pictorial and the lexicalist approach. More precisely, our experiment tests a prediction that follows from the pictorial approach, namely that small visual differences that do not correspond to AU differences should be semantically less relevant than visual differences that do correspond to AU differences, as the former do not have any clear real-life counterparts. Our results show that actually both types of visual differences are meaningful. This is compatible with a lexicalist approach to emoji semantics, which treats face emojis as conventionalized signals rather than mere pictures of faces.

2. Method

2.1. Design and materials

We tested three [AU+] emoji pairs and three [AU−] emoji pairs, shown in Figure 1. The emojis were rendered in the WhatsApp version as we aimed to test an emoji version that is commonly known in our sample population, which consists of emoji users in Germany. A survey with 18.243 participants, conducted between July 2020 and August 2021, yielded that 89% of the participants who were born between 1981 and 1995 and 93% of the participants who were born between 1996 and 2009 have WhatsApp (Initiative D21, 2022). Thus, WhatsApp is widely used in Germany, and the dominant smartphone operating system is Android (68% market share in the first quarter of 2023, according to https://www.kantarworldpanel.com/global/smartphone-os-market-share/), which unlike iOS does not overwrite WhatsApp emojis with its own emoji versions. The sample size of the emoji pairs was determined based on the availability of suitable emoji pairs without an AU difference. Constraints were that each emoji should occur in only one pair and that we aimed to avoid emojis with symbolic components like or , which cannot reflect actual human faces.

Figure 1. Emoji pairs tested.

The annotation of the emojis’ AUs, given in Figure 1, is based on the coding rubric by Fugate and Franco (2021). With the exception of the emojis grinning squinting face, astonished face, slightly frowning face and frowning face, which are not annotated in their paper, the AU codings were identical to Fugate and Franco’s (2021) existing AU codings for the emojis. Note, however, that Fugate and Franco do not provide AU codings for WhatsApp emojis. Instead, we used their codings of iOS emojis for guidance, which are very similar in design to the WhatsApp emojis. In the case of the emoji pairs with an AU difference, the AU that is exclusive to one of the two emojis is marked in boldface in the figure. Note further that we follow Fugate and Franco in not coding intensities of AUs. Intensity coding of AUs in face emojis comes with the challenges that it is hard to define the neutral expression of an emoji face and that a full spectrum of AU intensities cannot be observed in face emojis.

For each emoji pair, we created two contexts, each fitted to correspond to a prominent usage or meaning of one emoji or the other. The prominent usage or meaning of the emojis was determined based on their description on Emojipedia.org and on the results of a norming study with German speakers (Scheffler & Nenchev, submitted). In the norming study, participants were shown each emoji individually, without any surrounding linguistic context, and they were asked to name up to three meanings of each emoji. The wording of the prompt was Was bedeutet das Emoji? Nennen Sie bis zu drei verschiedene Bedeutungen. (‘What does the emoji mean? List up to three different meanings.’) The retrieved meanings of the two emojis in a given pair overlap to some extent, so we aimed to single out a meaning aspect that is part of only one of the emojis of a pair. The meaning aspect that was selected for each emoji and the verb phrase by which it was expressed in the experiment are listed in Figure 1 in the third line and fourth line, respectively.

We presented each context to the participants and asked them to choose between the two emojis of a pair in a forced-choice task. They were instructed that a person named Alex or Anna wants to write WhatsApp messages to their friend, but is not yet familiar with the use of emojis. The latter information was included to motivate the task the participant had to fulfill, namely to help Alex/Anna choose the emoji that best matched the text. For example, for grinning face with smiling eyes (), we had identified ‘amusement’ as a prominent meaning, while beaming face with smiling eyes () was found to often express ‘intense joy’. Figure 2 shows the presentation of two experimental items for this emoji pair. Red circles mark the context-matching emoji of the respective item. An English rendering of the item’s text is given in (1) (for the left side of the figure) and (2) (for its right side).

1. (1) Alex writes to his best friend Stefan:

   I just learned that my cousin’s dog has his own advent calendar.

   

   Figure 2. Presentation of test items in the experiment.

   Alex is amused.

   Which of the two emojis matches the message better?

   

1. (2) Alex writes to his best friend Stefan:

   I just learned that I won 500€ in the lottery.

   Alex is overjoyed.

   Which of the two emojis matches the message better?

   .

Our experimental design had 1 factor (AU difference) with 2 levels [AU+/AU−]. This factor was tested within participants and between items. The dependent variable was the measured rate (per condition) with which a context-matching emoji was chosen.

We tested 12 contexts corresponding to the total number of emojis we tested. For each context, we created four lexicalizations. For example, for the ‘amusement’ context, there were three other scenarios in which Alex/Anna had just received amusing news besides the scenario presented in Figure 2. They were: their cousin named her baby Ikea, their niece wants a Playmobil graveyard for her birthday, and their nephew’s dream job is Easter bunny. Thus, we had 48 test items in total (see Supplementary Appendix). The items were distributed over four experimental lists, so that each context was only seen once by each participant. In addition to the 12 test items, each list contained 12 filler/control items. They were of the same overall form as the test items, but the two emojis of a filler item were very distinct from one another. Therefore, it was unambiguously clear which emoji should be chosen in a given context. For example, in one context, the protagonist writes to her friend that she has to cancel their date, and the text below the screenshot states that she is expecting a (female) mechanic. The two emojis to choose from in this context were mermaid and woman mechanic.

There were two versions of the experiment. In one version, the protagonist of the stimuli and his friend (Alex and Stefan) were male, and in the other version, both the protagonist and her friend (Anna and Stefanie) were female. As the literature has reported gender differences regarding the use of emojis and attitude towards them (e.g., Chen et al., 2018; Herring & Dainas, 2020), we aimed to control for an influence of the protagonist’s gender.

The experiment was preregistered here: https://osf.io/hcma8/

Data and R-script are available at: https://osf.io/k2t9p/

2.2. Predictions

The iconic semantics of the pictorial approach to face emojis implies that [AU+] differences have a privileged status over [AU−] differences because they reflect differences in human facial features. Thus, assuming the pictorial approach, there should be a clear preference for the context-matching emoji for emoji pairs with an [AU+] difference. However, for [AU−] emojis, the context-matching emojis should be chosen significantly less often than for [AU+] pairs because the two emojis of an [AU−] pair can be expected to make an equivalent semantic contribution, which would cause them to be chosen more randomly.

In contrast, according to the lexicalist approach, both types of difference can create a difference in meaning between the two emojis of a pair of visually similar emojis. Under this approach, the context-matching emoji can be expected to always be the preferred choice irrespective of whether the emoji pair features an [AU+] or an [AU−] difference.

In this experiment, we assume the pictorial approach and attempt to provide evidence against the lexicalist approach: If in pairs with an AU difference, context-matching emojis are chosen significantly more often than at chance level and significantly more often than is the case with emojis of pairs without an AU difference, this would be in line with the pictorial approach, which implies a privileged role of realistic properties of human faces (such as AUs) and such a result would provide evidence against the lexicalist approach. If the rate of context matching emojis is above the chance level for at least one type of emoji and not significantly higher for emojis of pairs with an AU difference than is the case with emojis of pairs without AU difference, this result is compatible with the lexicalist approach, as it suggests that all manipulations matter, not just the realistic depictive ones.

2.3. Participants

In order to determine the size of the sample of participants, we conducted a power analysis based on a pilot version of the experiment using the software R (R Core Team, 2022), RStudio (RStudio Team, 2022) and the packages lme4 (Bates et al., 2015) and simr (Green & MacLeod, 2016). We fitted a generalized linear mixed model (logistic regression) for the pilot data and modified the slope to have a significant effect on the factor AU difference. The analysis yielded that more than 500 participants would be required to achieve an optimal statistical power of 0.8 for a difference between the conditions of 14 percentage points. Due to monetary constraints, we chose a sample size of 160 participants, for which a statistical power of approximately 0.73 was reached in the simulation.

Thus, 160 participants were recruited via Prolific and received 1.40€ as compensation for their participation. They were native speakers of German, residing in Germany. To avoid that different levels of emoji literacy or different degrees of familiarity with emojis affect the results, we aimed for a homogeneous group of participants. By using Prolific’s pre-screening filters, we targeted participants between 18 and 35 years of age who use WhatsApp on an Android phone. This age group can be assumed to have an adequate level of emoji literacy (Herring & Dainas, 2020) and can be expected to be familiar with the WhatsApp emojis used in the experiment.

One participant who exceeded the specified maximum age of 35 years (based on self-reported age) was excluded. The remaining 159 participants were aged between 18 and 35 years old ( $ M=26.61 $ , $ SD=4.32 $ ), 109 specified male, 47 female and 3 non-binary genders.

There were no exclusions due to replies to filler/control items, for which we had determined a threshold of maximally three mistakes. Performance on filler/control items was good: There were two participants who made one mistake each and one participant who made three mistakes.

Although we had filtered for Android users on Prolific, there were 6 (4%) participants who reported that their smartphone operating system was iOS and 3 (2%) participants who chose the option ‘other’ for this question. We did not exclude those participants as the smartphone operating system was not specified as an exclusion criterion in our preregistration. Due to the general similarity in design for the 12 emojis in our study across operating systems and devices, we do not expect a difference in behavior from these participants.

The majority of our participants use WhatsApp frequently according to their answers: 68% (108) use it multiple times a day, 18% (28) use it daily, 11% (18) every few days and 3% (5) only once in a while. The majority also reported to use emojis frequently: 30% (47) in almost every message, 44% (70) often, 19% (31) occasionally, 5% (8) rarely, 2% (3) never. As the literature has reported differences between men and women with regard to their use of and attitude toward emojis, we analyzed our data in this respect, too. As there were only three participants with non-binary gender, we did not include these participants in the gender-related analyses. Figure 3 shows how frequently the male and female participants of our experiment use emojis. In line with previous reports in the literature, we observe that the female participants in our study use emojis more frequently than the male participants.

Figure 3. Emoji use by gender.

The attitude towards emojis was overall positive among the participants. 22% (35) of the participants have a very positive attitude, 43% (68) a positive, 23% (36) a rather positive, 11% (17) a neutral and 2% (3) a rather negative attitude towards emojis. The answer options for this question ranged from very positive to very negative. (The options ‘negative’ and ‘very negative’ were not chosen by any participants.) Figure 4 shows the attitude towards emojis by gender. Again, corresponding to what has been reported in the literature, the female participants have a more positive attitude towards emojis than the male participants.

Figure 4. Attitude towards emojis by gender.

Finally, the participants reported a good understanding of emojis. 40% (63) stated their understanding is very good, 35% (56) stated it is good, 24% (38) replied that it is rather good, and 1% (2) reported moderate understanding. The answer options ranged from very good to very bad (with ‘moderate’ in the middle of the scale).

To sum up, these data indicate that the participants of our experiment are quite familiar with emojis, have a positive attitude towards them and use emojis frequently.

2.4. Procedure

The experiment was conducted as a web-based experiment created with the software _magpie (Franke et al., 2018). Thus, participants completed the experiment by means of their own devices. They were assigned to one of the two versions of the experiment (male or female protagonist) and to one of the experimental lists.

Each participant was tested on 6 emoji pairs in 12 contexts, that is, they saw each emoji pair twice, in one context favoring one member of the pair, and in one context favoring the other member. To avoid that the two contexts of an emoji pair occur in immediate succession in the experiment, the set of stimuli was divided into two blocks. One context of a pair occurred in block one and the other in block two. Each block contained 6 test items and 5 filler/control items, presented in pseudo-randomized order. The order of the blocks varied between lists. In half of the 48 stimuli, the (by our assumption) matching emoji was presented on the left side, and in the other half, it was presented on the right side. For test items, the order of matching/non-matching emoji was varied between lists. Each block was preceded by a filler/control item.

After completing the experimental task, participants were asked optional questions about their demographics and about the frequency with which they use emojis, their attitude towards and understanding of emojis, their use of WhatsApp and about their smartphone operating system with which they use emojis (see Section 2.3).

The completion of the experiment took between 1.7 and 17.1 min ( $ M=3.8 $ , $ SD=2 $ ).

3. Results

For data analysis, we used the software R (version 4.2.2, R Core Team, 2022) and RStudio (version 12.0, RStudio Team, 2022). The packages tidyverse (Wickham et al., 2019) and gridExtra (version 2.3, Auguie, 2017) were used for data processing and analysis, and the packages lme4 (Bates et al., 2015) and afex (version 1.2–1, Singmann et al., 2023) were used for the inferential statistical analysis.

The rate of context-matching emojis was 68% (650) in the [AU+] condition and 71% (675) in the [AU−] condition. There were no notable differences in the results between the version with the male ([AU+] = 68% (325), [AU−] = 70% (338)) and the female ([AU+] = 69% (325), [AU−] = 71% (337)) protagonist nor were there notable differences between male ([AU+] = 69% (450), [AU−] = 71% (462)) and female ([AU+] = 66% (186), [AU−] = 71% (202)) participants.

Figure 5 shows the rate at which a context-matching emoji was chosen for the individual emoji pairs. Each of the six bar charts is accompanied by the relevant emoji pair, such that the left emoji in the pair is the context-matching emoji for the left bar in the chart, whereas the right emoji is the context-matching emoji for the right bar. The green bars show the rate with which the context-matching emoji is chosen in the respective context. For almost all contexts, the matching emoji is generally preferred with matching rates above chance level. In the case of the pair neutral face and expressionless face ( and ), however, the latter emoji was preferred both in the ‘annoyance’ and in the ‘mild irritation’ context.

Figure 5. Results for individual emoji pairs.

For the inferential statistical analysis, we built a generalized linear mixed model (logistic regression) with context match (yes/no) as the dependent variable. The model included AU difference as a fixed factor, random slopes for participants and random intercepts for contexts and lexicalizations. The predicted probability of the choice of a context-matching emoji is 73% in the [AU−] condition and 72% in the [AU+] condition ( $ \mathrm{logit}\ \mathrm{difference}=-0.06, SE=0.44,z=-0.15,p=0.88 $ ). To confirm that the factor AU difference had no significant effect, as indicated by the model output, we performed a likelihood ratio test. It yielded that the minimal difference between the two conditions is not significant ( $ X(1)=0.0218,p=0.88 $ ). (Note that if the small difference were significant, this result would not be compatible with the pictorial hypothesis as the higher rate of matching emojis is in the [AU−] condition.) The intercept of the model ( $ \mathrm{logit}\ \mathrm{value}=0.99, SE=0.21,z=3.18,p=0.002 $ ) shows that the rate of context-matching emojis is above the chance level in both conditions.

4. Discussion

As evidenced by the clearly above-chance rate of matching emojis in both types of emoji pairs (i.e., emoji pairs with and without a difference in facial action units), each emoji is preferred over a visually similar emoji in a context that represents a prominent aspect of its meaning, as determined on the basis of the emoji’s Emojipedia.org entry and a norming experiment. Thus, subtle differences in emojis that otherwise look alike lead to differences in meaning.

An exception was the pair and . In the case of this pair, there was a general preference for in both contexts. A possible explanation is that participants found the mild irritation contexts (being placed on hold for 10 min in a service hotline, strike of the garbage collection, malfunctioning wifi router, pouring rain when wanting to go jogging) strongly annoying and not mildly irritating, even though it was stated that the protagonist is only ‘a bit irritated’ (etwas genervt).

Our experiment indicates that the small visual difference between the two emojis of a pair does, in fact, create a difference in meaning. This is equally the case for [AU+] differences that map to different facial actions in human faces and for [AU−] differences that represent the same facial actions in a human face. This result is compatible with a theory of face emojis, which assumes a symbolic semantics and allows for the possibility that the minimal components an emoji consists of carry conventionalized meaning similar to how linguistic signals are conventionalized.

Our results carry implications for the division of labor between semantics and pragmatics. The pictorial approach assumes a minimalist semantics, under which face emojis are essentially stylized pictures (of the author’s face) and nothing more than that. This is coupled with an independently motivated pragmatic machinery, which derives the more fine-grained and rich patterns that have been observed for the interpretation of emojis. Such a model would have difficulties explaining the differences in meaning between emojis without facial action unit differences we observed in our experiment, such as and . In principle, a different pragmatic process would need to be defined for each emoji to derive its specific contextual meaning.

By contrast, the lexicalist approach allows for a richer compositional semantics of face emojis by which individual parts of an emoji carry conventionalized meaning partially independent from their iconicity. As mentioned in the introduction, Cohn and Ehly (2016) analyze facial shapes such as mouths and eyes as morphemes and thus connect certain forms with conventionalized meanings. The lexicalist approach can, moreover, handle cases of emojis that contain symbolic components like , or more easily.

We wish to stress, however, that the experimental results presented in this paper do not constitute direct evidence for the lexicalist approach. They are merely compatible with it. Our approach was to assume the pictorial approach and try to provide evidence against the lexicalist approach; we showed that the experimental results did not provide such evidence, while showing that context-matching emojis are chosen significantly above chance level in all conditions. Future work should devise ways to probe the lexicalist hypothesis more directly, for example, by investigating the compositionality of the minimal units that emojis consist of. We currently plan to investigate to which extent single features such as eye or mouth shapes are associated with certain meanings, by testing them independently from the combinations with other shapes they co-occur with in existing emojis.

Face emojis are clearly iconic in that they represent human faces. However, due to our experimental results and due to the fact that there are a number of emojis that include symbolic features, we anticipate that a hybrid theoretical semantic approach is eventually required. Such a hybrid approach would incorporate both iconic and symbolic meaning components. This would make it possible to account for the whole spectrum from highly iconic emojis to emojis that incorporate symbolic components.