Social hierarchical information impacts language comprehension. Nevertheless, the specific process underlying the integration of linguistic and extralinguistic sources of social hierarchical information has not been identified. For example, the Chinese social hierarchical verb 赡养, /shan4yang3/, ‘support: provide for the needs and comfort of one’s elders’, only allows its Agent to have a lower social status than the Patient. Using eye-tracking, we examined the precise time course of the integration of these semantic selectional restrictions of Chinese social hierarchical verbs and extralinguistic social hierarchical information during natural reading. A 2 (Verb Type: hierarchical vs. non-hierarchical) × 2 (Social Hierarchy Sequence: match vs. mismatch) design was constructed to investigate the effect of the interaction on early and late eye-tracking measures. Thirty-two participants (15 males; age range: 18–24 years) read sentences and judged the plausibility of each sentence. The results showed that violations of semantic selectional restrictions of Chinese social hierarchical verbs induced shorter first fixation duration but longer regression path duration and longer total reading time on sentence-final nouns (NP2). These differences were absent under non-hierarchical conditions. The results suggest that a mismatch between linguistic and extralinguistic social hierarchical information is immediately detected and processed.

The literature on face emojis raises the central question whether they should be treated as pictures or conventionalized signals. Our experiment addresses this question by investigating semantic differences in visually similar face emojis. We test a prediction following from a pictorial approach: small visual features of emojis that do not correspond to human facial features should be semantically less relevant than features that represent aspects of facial expressions. We compare emoji pairs with a visual difference that either does or does not correspond to a difference in a human facial expression according to an adaptation of the Facial Action Coding System. We created two contexts per pair, each fitted to correspond to a prominent meaning of one or the other emoji. Participants had to choose a suitable emoji for each context. The rate at which the context-matching emoji was chosen was significantly above chance for both types of emoji pairs and it did not differ significantly between them. Our results show that the small differences are meaningful in all pairs whether or not they correspond to human facial differences. This supports a lexicalist approach to emoji semantics, which treats face emojis as conventionalized signals rather than mere pictures of faces.

In this study, our objective was to explore the impact of hearing loss on the conceptual system underlying word meaning. We collected perceptual strength norms for 200 Italian words from early deaf individuals with limited or no access to auditory information and compared them to existing norms from hearing individuals. For each word, participants provided perceptual strength ratings for each perceptual modality. Our results revealed a significant reduction of the auditory modality in the norms provided by deaf individuals compared to the hearing population. However, we did not observe an overall decrease in reported perceptual strength. Interestingly, we found a heightened involvement of other sensory modalities accompanied by reduced modality exclusivity in the conceptualization of words, indicating that deaf individuals heavily rely on information coming from the other perceptual modalities to form concepts. These findings suggest that hearing loss leads to a reorganisation of word conceptualization, characterised by increased multisensoriality. Importantly, although diminished, the auditory modality remains present, suggesting that deaf individuals can still infer auditory-associated knowledge about words to some extent.

Reading official letters and being able to react to them appropriately is part of the daily life of every adult in many countries. Although the history of the plain language movement dates to the past century, it is only now that efforts are being made in Poland to adapt official documents to their audience. In this paper, we describe the results of a study (N = 685) in which we examined how particular linguistic forms and text structure affect the reception of an official-looking text – comprehension as well as emotions experienced, trust in the office, feelings of helplessness in the situation of reading the writing, and many other variables. Based on Imbir’s (2016, New Ideas in Psychology, 43, 39–49), two-system theory of emotions, we aim to examine how emotions – triggered by these complex stimuli – affect their processing. We hope that the results of our interdisciplinary research will answer the questions and hypotheses posed by researchers and find application in the reforms currently underway. We believe that by linking findings from cognitive psychology, the psychology of emotions and linguistics, we will also expand knowledge in the broad discussion on overcoming barriers between administrative offices and target audiences.

This essay is the introduction to the Special Issue ‘Events in language and mind: Theoretical and empirical advances in the event integration theory’. We first review Leonard Talmy’s event integration theory in addition to some critiques of this framework. Following this, we point to some empirical research inspired by this framework, which explores the interaction between language and cognition. We then briefly introduce the papers in this volume and discuss their contributions to the event integration framework. We conclude with some limitations, questions and future directions.

This paper presents an audiovisual perceptual analysis of the wh-question and wh-exclamation intonation in Brazilian Portuguese using auditory–visual congruent and incongruent stimuli, to investigate the relative importance of each modality in signaling pragmatic meanings. Ten Brazilian Portuguese speakers (five female) were filmed while producing both speech acts 10 times. Next, artificial stimuli were created: audio and visual cues were either matched (audio and video from the same speech act) or mismatched (audio and video from the different speech acts), resulting in 10 congruent and 10 incongruent stimuli of the wh-questions and the wh-exclamations. The perceptual experiment was taken by 36 Brazilians who identified the stimulus as a question or an exclamation. Results from the logistic regression showed that the factor ‘congruence’ was significant and had a significant interaction with ‘speakers’, which means that the congruent stimuli increased the comprehension of the Brazilian Portuguese wh-questions and wh-exclamations. In contrast, the incongruent stimuli tended to lower listeners’ identification, but to a degree depending on individual speakers’ strategies. Although variation in the accuracy of expressing both speech acts was also found across speakers, this study corroborates that the visual channel impacts the perceptual identification of the pragmatic intonation function of distinguishing sentence mode.

In this work, we use a mathematical model of the property listing task dynamics and test its ability to predict processing time in semantic and lexical decision tasks. The study aims at exploring the temporal dynamics of semantic access in these tasks and showing that the mathematical model captures essential aspects of semantic access, beyond the original task for which it was developed. In two studies using the semantic and lexical decision tasks, we used the mathematical model’s coefficients to predict reaction times. Results showed that the model was able to predict processing time in both tasks, accounting for an independent portion of the total variance, relative to variance predicted by traditional psycholinguistic variables (i.e., frequency, familiarity, concreteness imageability). Overall, this study provides evidence of the mathematical model’s validity and generality, and offers insights regarding the characterization of concrete and abstract words.

Embodied imagery hypothesis proposes the activation of perceptual-motor systems during language processing. Previous studies primarily used concrete visual stimuli to investigate mental imagery in language processing by native speakers (NSs) and second language (L2) learners, but few studies employed schematic diagrams. The study aims to investigate mental imagery in processing prepositional phrases by English NSs and L2 learners. Using image-schematic diagrams as primes, we examine whether any mental imagery effect is modulated by target preposition (over, in), the abstractness of meaning (spatial, extended), and stimulus onset asynchrony (SOA; 1,040 ms, 2,040 ms). A total of 79 adult L2 learners and 100 NSs of English completed diagram–picture matching and semantic priming phrasal decision tasks. Results revealed interference effects on L2 processing of over phrases and under 2,040 ms SOA, but no such effects were observed in the NS group. The selective interference effects in L2 suggest different mental imagery patterns between L1 and L2 processing, and processing schematic diagram primes requires high cognitive demands, potentially leading to difficulties in integrating visual and linguistic information and making grammaticality judgments. The findings partially validate schematic diagrams as visual representations of concepts and suggest the need for further examination of schematic diagrams with varying degrees of complexity.

Distributional semantic representations were used to investigate crossmodal correspondences within language, offering a comprehensive analysis of how sensory experiences interconnect in linguistic constructs. By computing semantic proximity between words from different sensory modalities, a crossmodal semantic network was constructed, providing a general view of crossmodal correspondences in the English language. Community detection techniques were applied to unveil domains of experience where crossmodal correspondences were likely to manifest, while also considering the role of affective dimensions in shaping these domains. The study revealed the existence of an architecture of structured domains of experience in language, whereby crossmodal correspondences are deeply embedded. The present research highlights the roles of emotion and statistical associations in the organization of sensory concepts across modalities in language. The domains identified, including food, the body, the physical world and emotions/values, underscored the intricate interplay between the senses, emotion and semantic patterns. These findings align with the embodied lexicon hypothesis and the semantic coding hypothesis, emphasizing the capacity of language to capture and reflect crossmodal correspondences’ emotional and perceptual subtleties in the form of networks, while also revealing opportunities for further perceptual research on crossmodal correspondences and multisensory integration.

Research using masked priming and parafoveal preview techniques has shown that visual letter similarity has an impact on word processing during the initial stages in Latin-derived scripts. However, these effects appear to be absent in Arabic. One reason for this discrepancy could be attributed to the distinctive features of the Arabic script, which includes numerous letters sharing a basic form while varying in the location or number of diacritics. To shed light on this issue, the present study employed Arabic letters rather than words in two masked priming experiments: an alphabetic decision task and a letter-matching task. Both experiments showed that visually similar letters were more effective as primes than visually dissimilar letters. These findings suggest that the processes of letter identification in Arabic and Latin scripts may be roughly alike, implying that differences in visual letter similarity across scripts may arise at later stages of processing.

This study addresses the importance of orofacial gestures and acoustic cues to execute prosodic patterns under different communicative settings in Farsi. Given that Farsi lacks morpho-syntactic markers for polar questions, we aim to determine whether specific facial movements accompany the prosodic correlates of questionhood in Farsi under conditions of degraded information, that is, whispering and wearing face masks. We hypothesise speakers will employ the most pronounced facial expressions when whispering questions with a face mask to compensate for the absence of F0, reduced intensity and lower face invisibility. To this end, we conducted an experiment with 10 Persian speakers producing 10 pairs of statements and questions in normal and whispered speech modes with and without face masks. Our results provide support to our hypotheses that speakers will intensify their orofacial expressions when confronted with marked conditions. We interpreted our results in terms of the ‘hand in hand’ and ‘trade-off’ hypothesis. In whispered speech, the parallel realisation of longer word duration and orofacial expressions may be a compensatory mechanism for the limited options to convey intonation. Also, the lower face coverage is mutually compensated for by word duration and intensified upper facial expressions, all of which in turn support the trade-off hypothesis.

A well-known method of studying iconic words is through the collection of subjective ratings. We collected such ratings regarding familiarity, iconicity, imagery/imageability, concreteness, sensory experience rating (SER), valence and arousal for Mandarin ABB words. This is a type of phrasal compound consisting of a prosaic syllable A and a reduplicated BB part, resulting in a vivid phrasal compound, for example, wù-mángmáng 雾茫茫 ‘completely foggy’. The correlations between the newly collected ABB ratings are contrasted with two other sets of prosaic word ratings, demonstrating that variables that characterize ABB words in an absolute sense may not play a distinctive role when contrasted with other types of words. Next, we provide another angle for looking at ABB words, by investigating to what degree rating data converges with corpus data. By far, the variable that characterizes ABB items consistently throughout these case studies is their high score for imageability, showing that they are indeed rightfully characterized as vivid. Methodologically, we show that it pays off to not take rating data at face value but to contrast it with other comparable datasets of a different phenomenon or data about the same phenomenon compiled in an ontologically different manner.

This study investigated whether speakers use multimodal information (speech and gesture) to differentiate the physical and emotional meanings of the polysemous verb touch. We analyzed 302 hand gestures that co-occurred with this perception verb. For each case, we annotated (1) the meaning of touch (physical vs. emotional), (2) the gesture referent speakers physically touched (other-touch vs. self-touch), (3) the personal pronoun following the verb and (4) if they used intensifiers and negation. There were three main findings. First, we have seen that when speakers express the physical meaning, they are likely to reach an external referent (other-touch), but when they imply the emotional meaning, they tend to touch their own body (self-touch). Second, the most frequent co-speech gesture (chest-touching gesture) was associated with the emotional meaning, uncovering the metaphor the heart is container for emotions. Third, this study showed that the physical meaning of touch usually coexists with a wide variety of personal pronouns and negation words; in contrast, the emotional meaning of touch occurs primarily with the pronoun me and it is usually modified by intensifiers. Thus, speakers use both speech and gesture to differentiate the meanings of the polysemous verb touch.

Japanese pitch accent is phonemic, making it crucial for second-language learners to acquire. Building on theories of multimodal learning, the present study explored how auditory, visual and gestural training of Japanese pitch accent affected behavioral, neural and meta-cognitive aspects of pitch perception across two experiments. Experiment 1 used a between-subjects pre/posttest design to train native English speakers to perceive Japanese pitch accents in one of the following three conditions: (1) baseline (audio + flat notation), (2) pitch height notation (audio + notation mimicking pitch height) and (3) pitch height notation + a left-hand gesture (L-gesture) (to engage the contralateral right hemisphere specialized for suprasegmental pitch processing). Our results indicated that (2) pitch height notation training was most robust in its benefits, as participants in this condition improved on trained and novel words alike. Experiment 2 used a within-subjects design to extend Experiment 1 in three ways: adding a right-hand gesture (R-gesture) condition (to engage more segmental language areas in the left hemisphere), introducing a neural correlate of cognitive load (measured by EEG alpha and theta power) and performing a metacognitive subjective assessment of learning (e.g., ‘Which training did you find the most helpful?’). The results showed that although there were no differences among our four training conditions on learning outcomes or EEG power, participants made the most positive subjective evaluations about pitch height notation and R-gesture training. Together, the results suggest that there may be a ‘just right’ amount of multimodal instruction to boost learning and increase engagement during foreign language pitch instruction.

With a focus on politicians’ and medical experts’ gratitude expressions in UK government COVID-19 briefings, this research describes how perspective and intensity were modulated in expressing gratitude to realise different pragmatic intentions. This corpus-assisted analysis finds that retrospective or prospective gratitude expression was adopted by the two British elite groups to build solidarity (encouraging) and/or make requests (directing) for protecting public health. Gratitude of varying intensities was expressed (e.g. by highlighting metaphorical dimensions such as WIDTH and DEPTH) to correspond to the importance of a benefit (judged by how much the given benefit matches the receiver’s needs and preferences) and/or to implicitly display the evaluation of the benefactor’s responsibility and efforts. We tentatively formulate a dynamic model of gratitude expression in public discourse and shed light on the metaphorical conceptualisation of English gratitude expression and the power of gratitude expression in boosting social cohesion and directing social actions in a discourse of crisis.

Movement scientists have proposed to ground the relation between prosody and gesture in ‘vocal-entangled gestures’, defined as biomechanical linkages between upper limb movement and the respiratory–vocal system. Focusing on spoken language negation, this article identifies an acoustic profile with which gesture is plausibly entangled, specifically linking the articulatory behaviour of onset consonant lengthening with forelimb gesture preparation and facial deformation. This phenomenon was discovered in a video corpus of accented negative utterances from English-language televised dialogues. Eight target examples were selected and examined using visualization software to analyse the correspondence of gesture phase structures (preparation, stroke, holds) with the negation word’s acoustic signal (duration, pitch and intensity). The results show that as syllable–onset consonant lengthens (voiced alveolar /n/ = 300 ms on average) with pitch and intensity increasing (e.g. ‘NNNNNNEVER’), the speaker’s humerus is rotating with palm pronating/adducing while his or her face is distorting. Different facial distortions, furthermore, were found to be entangled with different post-onset phonetic profiles (e.g. vowel rounding). These findings illustrate whole-bodily dynamics and multiscalarity as key theoretical proposals within ecological and enactive approaches to language. Bringing multimodal and entangled treatments of utterances into conversation has important implications for gesture studies.

The production of speech and gesture is exquisitely temporally coordinated. In autistic individuals, speech-gesture synchrony during spontaneous discourse is disrupted. To evaluate whether this asynchrony reflects motor coordination versus language production processes, the current study examined deliberately performed hand movements during speech in youth with autism spectrum disorder (ASD) compared to neurotypical youth. Neurotypical adult performance provided a mature baseline. Participants read aloud rhythmic nursery rhymes, while producing a beat-like hand movement. An automated pixel-change video measure identified kinematic peaks; using smoothed acoustic envelope analyses, we identified peaks in speech. Results indicated few diagnostic group differences in explicit speech-movement coordination, although adolescent performance differed from adults. Adults demonstrated higher tempo and greater rhythmicity in their coordination; this group difference suggests that the method is sufficiently subtle to reveal individual differences and that this form of complex coordination undergoes ongoing maturation beyond adolescence. The sample is small, and thus results are necessarily preliminary. In the context of prior speech-gesture coordination studies, these findings of intact synchrony are consistent with the hypothesis that it is the demands of discourse planning, rather than motor coordination, that have led to prior findings of asynchrony during spontaneous speech; this possibility awaits future research.

An important quality to assess in others is their cooperativeness. We hypothesized that people use linguistic markers in their partners’ speech as a proxy of their cooperativeness in other tasks: specifically, we predicted that participants would prefer syntactically similar conversation partners as cooperation partners in a monetary game. We found that, indeed, participants preferably selected syntactically similar conversation partners as cooperation partners, but only when the participants could communicate using their naturally preferred constructions. In contrast, when participants were forced to communicate using dispreferred constructions, they rather cooperated with those partners that matched their natural preference than with those that matched their overt linguistic use. This pattern of results was likely driven by participants valuing representational alignment (i.e., being aligned on both linguistic features and their mental representations) more than incidental behavioral alignment (i.e., superficial convergence on similar linguistic features during interaction). This is because representational alignment is a potential indicator of group membership and may be associated with in-group benefits such as reputation, reciprocity and normative behavior. Those benefits may outweigh the benefits of simple behavioral alignment, which could be a potential indicator of others’ willingness to cooperate. This has important implications for communication in intercultural settings where members of diverse linguistic groups negotiate cooperative actions.

Monomorphemic words have been found to influence category formation, as they encode one general category and thus activate it more than other related categories in the same lexical network. On the other hand, multimorphemic words can encode multiple categories from the same network by the multiple forms they combine. Superordinate categories are encoded by sub-lexical forms (e.g., affixes), while the entire words encode lower categories in the hierarchical structure. In the present study, we asked whether sub-lexical forms influence the learning of the meaning encoded by the entire word they underlie. We used Semitic-like words where sub-lexical forms (syllabic patterns) encode superordinate categories of manner-of-motion, and the entire words encode lower-level categories (moving characters). In our main experiment, a word-learning test showed that a shared syllabic pattern had a negative effect on the learning of the moving characters encoded by the entire word. This effect was revealed mainly in dimensions related to the superordinate category encoded by the pattern. The effect and its direction are beyond the expectations of enhanced category representations suggested in previous literature. We conclude that the effect of word-form is beyond the specific category they encode and can have different directions at different hierarchical levels.