Are schematic diagrams valid visual representations of concepts? Evidence from mental imagery in online processing of English prepositions

Abstract

Embodied imagery hypothesis proposes the activation of perceptual-motor systems during language processing. Previous studies primarily used concrete visual stimuli to investigate mental imagery in language processing by native speakers (NSs) and second language (L2) learners, but few studies employed schematic diagrams. The study aims to investigate mental imagery in processing prepositional phrases by English NSs and L2 learners. Using image-schematic diagrams as primes, we examine whether any mental imagery effect is modulated by target preposition (over, in), the abstractness of meaning (spatial, extended), and stimulus onset asynchrony (SOA; 1,040 ms, 2,040 ms). A total of 79 adult L2 learners and 100 NSs of English completed diagram–picture matching and semantic priming phrasal decision tasks. Results revealed interference effects on L2 processing of over phrases and under 2,040 ms SOA, but no such effects were observed in the NS group. The selective interference effects in L2 suggest different mental imagery patterns between L1 and L2 processing, and processing schematic diagram primes requires high cognitive demands, potentially leading to difficulties in integrating visual and linguistic information and making grammaticality judgments. The findings partially validate schematic diagrams as visual representations of concepts and suggest the need for further examination of schematic diagrams with varying degrees of complexity.

1. Introduction

As a language-induced cognitive phenomenon, mental imagery is defined as the ability to mentally resemble and internally recreate perceptual-motor experiences in the absence of external physical stimuli (Barsalou, 1999, 2008). It posits that comprehending linguistic input about a perceived event involves a mental enactment of corresponding perceptual-motor components in the brain (Bergen, 2007, 2019). Research on mental imagery has predominantly used concrete pictures to explore its effects on the first language (L1) processing in adult native speakers (NSs; de Koning et al., 2017a; Schütt et al., 2023; Stanfield & Zwaan, 2001; Zwaan et al., 2002). A few L1 mental imagery studies have used visual schematic diagrams to examine whether they can activate offline and online language processing (Richardson et al., 2001, 2003). Compared to the abundant research on L1-based embodied mental imagery, there are far fewer studies on mental imagery in second language (L2) processing (Ahn & Jiang, 2018; Norman & Peleg, 2022; Wheeler & Stojanovic, 2006). No prior studies have adopted schematic diagrams to examine L2 mental imagery.

Intertwining with human cognition and arising from sensorimotor experiences, image schemas are recurring and dynamic patterns of human perception (Lakoff & Johnson, 1980; Mandler, 2005). These schemas, such as UP-DOWN, CONTAINMENT and FORCE, are multimodal analog representations that provide a holistic conceptual summary of perceived spatial relations and movements (Mandler, 2004). A schematic diagram is what cognitive linguists often use to visually illustrate the underlying spatiotemporal relationships of these schemas and externalized linguistic representations. Recently, schematic diagrams have been increasingly used in cognitive linguistics (CL)-inspired L2 pedagogy, but have produced very mixed findings (Boers et al., 2008, 2009; Takimoto, 2021; Wong et al., 2018; Zhao et al., 2020). Despite these growing applications of schematic diagrams, whether diagrams are appropriate visual representations of linguistic concepts and whether the schematic meanings conveyed by diagrams are comprehensible to L2 learners are often not empirically validated.

Currently, there is a great dearth of empirical studies employing reaction-time (RT)-based behavioral experiments to evaluate the validity of schematic diagrams in CL analysis, as well as how L1 and L2 speakers mentally represent the meanings conveyed by diagrams. These investigations are pivotal for establishing the psychological basis of diagrams as pedagogical aids in L2 instruction. To address these research gaps, the present study aims to examine whether and how mental imagery is triggered by diagrams during the online processing of English prepositional phrases among both L1 and L2 speakers.

We adopted a priming paradigm and utilized schematic diagrams illustrating the spatial configurations of prepositional senses derived from published CL diagrams (Tyler & Evans, 2003) as visual primes. Behavioral patterns of both L1 and L2 speakers were measured in response to prepositional phrases that shared similar spatial configurations as the schematic diagrams. By employing a priming paradigm, we can capture the real-time perception and cognition of diagrams. Additionally, this paradigm allows us to explore the extent to which the encoding of a diagram facilitates or interferes with the online processing of spatial configurations associated with prepositions. Ultimately, these findings will provide empirical validation for the integration of CL diagrams in language instruction.

1.1 Mental imagery in language processing and influential factors

Previous research on L1-based mental imagery has observed both compatibility and interference effects (Bergen et al., 2007, 2010; Kaschak et al., 2005; Liu & Bergen, 2016; Stanfield & Zwaan, 2001). The compatibility effect suggests a shared neural structure for language comprehension and spatial perception (Bergen, 2007). For instance, processing a sentence like A boy climbs a mountain activates the UP-DOWN schema, leading to faster responses to a compatible vertical spatial configuration than to an incompatible horizontal one. In contrast, the interference effect implies mutual inhibition between language comprehension and spatial perceptions, resulting in slower responses when both processes recruit the same perceptual neurons concurrently (Bergen, 2007). The precise mechanisms underlying these effects of mental imagery are not fully specified, but facilitation of imagery in tasks like object discrimination and image–word matching occurs when there are shared properties of identity, shape or location between the real and imaginary objects (Craver-Lemley & Arterberry, 2001). Interference effects are also observed when there is temporal overlap and weak integrability between matching tasks (Bergen, 2007).

A widely used task in mental imagery studies is the priming-based sentence–picture verification task. Participants read or hear a prime sentence or a word and then verify whether a subsequent picture matches the meaning of the prime. Some studies reverse the presentation sequence, where participants saw the picture before the verbal stimuli (Bergen et al., 2003; Lindsay, 2003). These task paradigms yielded mental imagery effects on processing language encoding orientation (Engelen et al., 2011; Richardson et al., 2003; Stanfield & Zwaan, 2001), shape (de Koning et al., 2017a; Rommers et al., 2013; Schütt et al., 2023; Zwaan et al., 2002) and size (Chen et al., 2020; de Koning et al., 2017b).

Bergen et al. (2003) found interference in an image–verb matching task, where English NSs primed with a verb image (e.g. run) took longer to reject effector-matching verbs (e.g. kick) than effector-mismatching verbs (e.g. drink). The interference was attributed to competition and mutual inhibition in brain circuits when words and pictorial representations for similar actions are presented (Bergen et al., 2010; Kaschak et al., 2005). Narayan et al. (2004) using a lexical matching task with English NSs also observed a similar interference effect.

Many factors influence the outcome of mental imagery, one of which is the abstractness of meanings. In a study by Liu and Bergen (2016), English NSs judged sentence sensibility by pressing a button at different distances from their body. Their RT results showed that processing concrete language (e.g. Jeffery has thrown the pills onto the floor) led to compatibility effects, whereas processing abstract language (e.g. Dan is confessing his secret to the courtroom) resulted in interference effects. The authors argued that the spatial opaqueness of abstract language made it harder to process and mentally simulate simultaneously. In contrast, Richardson et al. (2003) reported no NS mental imagery difference based on semantic abstractness. They used concrete and abstract verbs normed for vertical (bomb, respect) or horizontal (pull, argue) image schemas and observed interference in a visual discrimination task but facilitation in a visual memory task, regardless of the encoded literal or metaphorical spatial meanings. The finding was attributed to the cognitive process underlying literal and metaphorical connections (Lakoff, 1987).

Stimulus onset asynchrony (SOA), the time between prime and target onset, is another experimental influential factor that affects mental imagery outcomes (Bergen et al., 2007; Masson et al., 2008). Bergen et al. (2007) noted interference with a shorter SOA (50–200 ms) suggesting resource conflict, while compatibility was more likely with sufficient priming time. However, interference effects were observed in longer SOA conditions (e.g. 1,500 ms in Bergen et al., 2003; Wheeler & Stojanovic, 2006), highlighting the complexity of SOA impact. Considering the diverse primes used in mental imagery experiments, ranging from words to images, concrete to abstract, an oversimplified conclusion about SOA is less informative. It is more meaningful to treat SOA as a variable and explore its effects on mental imagery outcomes in perceptual and language processing.

1.2 Mental imagery in L2 processing

Compared to the abundant research on L1-based embodied mental imagery, there are significantly fewer studies on L2 mental imagery (Feng & Zhou, 2021; Vukovic & Williams, 2014; Wheeler & Stojanovic, 2006). Similar to the patterns of L1 mental imagery, both compatibility (Ahn & Jiang, 2018; Dudschig et al., 2014; Koster et al., 2018; Tomczak & Ewert, 2015) and interference effects (Bergen et al., 2010; Buccino et al., 2017; Vukovic & Williams, 2014; Wheeler & Stojanovic, 2006) have been found in L2 mental imagery research. However, some studies have reported no effects or attenuated effects in L2 mental imagery relative to L1 mental imagery (Chen et al., 2019; Norman & Peleg, 2022; Wu, 2016) due to a reduced degree of embodiment in an L2 compared to L1 (Foroni, 2015), or the formation of different mental representations in L1 and L2 (Dudschig et al., 2014; Monaco et al., 2019). Most existing studies using linguistic and visual stimuli support the perceptual-motor activation in L2 processing, covering various aspects like size and orientation (Koster et al., 2018), shape (Ahn & Jiang, 2018), motion (Tomczak & Ewert, 2015) and emotion (Dudschig et al., 2014; Foroni, 2015). However, no studies have utilized diagrams to examine the extent to which perceptual-motor and linguistic information can be integrated into L2 processing and the potential factors that might influence this integration.

L2 proficiency has been identified as a key factor influencing the outcomes of L2 mental imagery. Wheeler and Stojanovic (2006), for instance, employed the same image–verb matching task and stimuli as Bergen et al. (2003) and found similar interference effects on L2 learners’ processing of English verbs. They also observed that the size of mental imagery effects increased as L2 proficiency developed. In other words, as L2 learners became more familiar with the target verbs, they were better able to fully identify and understand them, resulting in stronger interference effects. Ahn and Jiang (2018) reported compatibility effects, extending the scope to orientation and shape in a sentence–picture matching task. The authors argued that their L2 participants had developed native-like semantic integration abilities for processing linguistic and real-world knowledge and forming semantic representations of sentences. Consequently, sufficient L2 semantic integration competence contributed to the enactment of mental imagery.

In contrast, Chen et al. (2019) and Norman and Peleg (2022) conducted studies that did not find evidence of L2 (and L3) mental imagery, and their findings led to the argument that L2 proficiency may not significantly impact mental imagery in non-native language processing. Chen et al. (2019) focused on the shape feature in multilingual speakers’ comprehension of L1 Cantonese, L2 Mandarin and L3 English. They discovered a compatibility effect in L1 processing but observed no such effect in L2 or L3, despite participants having higher proficiency in L2 than L3. The authors suggested that different conceptual systems were at play in L1, L2 and L3, with strong evidence of L1 mental imagery but a lack of embodied imagery in non-native language processing irrespective of proficiency levels. Norman and Peleg (2022) also explored mental imagery with regard to shape. They detected compatibility effects in L1 Hebrew sentence processing but found no effects in L2 English. Their conclusions supported the idea of distinct L1 versus L2 conceptual systems and suggested that L2 comprehension might not be grounded in sensorimotor knowledge as L1 comprehension. The authors attributed these differences to the language acquisition settings, as their participants were late Hebrew–English bilinguals who acquired Hebrew in naturalistic settings but primarily received English instructions in formal settings. Consequently, they argued that the manner of language acquisition could modulate the presence of mental imagery effects, resulting in a reduced degree of embodiment of conceptual representations in L2 compared to L1 (Dudschig et al., 2014; Foroni, 2015). The debate over whether L2 learners’ processing patterns and behaviors align with those of NSs remains a topic of controversy in the field.

1.3 The application of schematic diagrams

Diagrams are graphic representations composed of simple symbols, such as lines, dots, arrows and boxes, that are used to convey meanings and represent spatial relations (Bryant & Tversky, 1999; Tversky et al., 2000). These symbols, while basic, have the versatility to represent a wide range of concepts. For example, circles may denote plates and nodes, while lines can signify boundaries, limits and divisions. Serving as external graphic aids, diagrams visualize our internal mental representations of both concrete (e.g. people, space) and abstract concepts (e.g. time, quantity). By mapping diagram elements onto real-world referents, they facilitate inferences about spatial relationships (Tversky, 2015; Tversky et al., 2016). Additionally, diagrams bridge the gap between abstract information and the tangible world (Karaca, 2012), making abstract properties and relations more accessible (Hutchins, 2005). However, as abstractions of concepts, diagrams may invite multiple interpretations due to their one-to-many mappings, leading to ambiguity in diagram interpretations (Tversky, 2015). For example, an over diagram that represents the concept above could be interpreted as spatial representations of other prepositions, such as up or on, resulting in reduced reliability of the over diagram and corresponding spatial meanings.

Diagrams play a crucial role in supporting cognitive processes involved in expressing and understanding complex linguistic meanings (Tylén et al., 2013). Unlike written language, diagrams are a unique type of pictorial language capable of elucidating phenomena by encapsulating specific instances within a prototype or schemata of a category (Churchland, 1992). Tyler and Evans (2003) expanded on this concept by introducing diagrams to represent the proto-scene of English prepositions. They defined proto-scene as ‘an idealized mental representation across the recurring spatial scenes associated with a particular spatial particle’ (pp. 52) and as the ‘abstract mental representation of the primary sense’ of a preposition (pp. 65). Central to a proto-scene are the trajector (TR) and landmark (LM), which represent the fundamental spatial relationships evoked by the preposition. This proto-scene also provides a conceptual structure for understanding the spatial meaning conveyed by the preposition across different contexts. Understanding a diagram involves a prototype activation process, extracting general information from its well-constructed schematic representation with added unique features to aid interpretation (Craver, 2007). Diagrams also exemplify the least ‘embedded’ level of mental representation, being the least situated and context-specific, most symbolic, abstract and imaginative (Zwaan, 2014).

In theoretical CL literature, diagrams are viewed as imagistic tools extensively used to illustrate complex linguistic systems by providing heuristic representations of their meanings (Lakoff, 1987; Langacker, 2013; Lindstromberg, 2010; Tyler & Evans, 2003). Schematic diagrams, a CL creation, represent conceptual spatiotemporal configurations of abstract schematic meanings, such as CONTAINMENT, PATH, FORCE and so forth. Despite doubts about their ability to completely and realistically envision semantic concepts (Marshall & Freitas, 2021), diagrams are supported by Langacker (2013), noting that they excel in ‘elaborating technical displays of great complexity’ and ‘provide a level of precision and explicitness sufficient for most purposes, together with a kind of usability that facilitates discovery’ (pp. 10).

Diagrams have been employed in empirical research as visual stimuli to demonstrate the relationship between abstract schemas and linguistic instantiations (Richardson et al., 2003; Spivey et al., 2005). Richardson et al. (2001) conducted a norming study and found English NSs exhibited highly consistent diagrammatic representations of concrete and abstract verbs, with greater consistency in verbs associated with vertical schemas (bomb, respect) compared to verbs linked with predicted horizontal schemas (push, offend), providing empirical support for using schematic diagrams to represent both concrete and abstract meanings. Nevertheless, the diagrams generated by cognitive linguists are lack of enough empirical validation for their conceptual meanings (Tyler et al., 2011).

Recently, diagrams have become increasingly prevalent in CL-inspired L2 pedagogy for teaching abstract concepts, grammar and vocabulary (Wong et al., 2018; Zhao et al., 2020). It was believed that diagrams assist learners in establishing conceptual mappings between physical and non-physical domains of human experience, encouraging deep processing and dual coding of language and images to enhance the mnemonic benefits for learning outcomes (Boers et al., 2008). These diagrams vary in composition, some being purely graphic, while others incorporated word annotations alongside graphic symbols. These highly symbolic diagrams may possess limited visual saliency for prototype activation, and their less intuitive intended meaning may invite more interpretations. However, the accessibility of meaning to L2 learners remains an empirical question to be investigated.

1.4 The present study

The current study is the first investigation into mental imagery in the online processing of English prepositions, specifically primed by schematic diagrams. Our research aims to address several critical research gaps. Firstly, previous research on mental imagery in spatial language processing has primarily focused on verb or noun comprehension (Bergen, 2005; Bergen et al., 2003; Richardson et al., 2001, 2003). Despite the crucial role prepositions play in expressing spatiality in English, previous studies have scarcely explored embodied mental imagery effects on preposition processing. Secondly, diagrams, as a widely used form of visualizing conceptual and schematic representations, have not been sufficiently investigated in mental imagery studies (Richardson et al., 2003), and none in L2-based studies. However, diagram-based research will provide valuable insights into the psychological reality of abstract image schemas in language processing (Spivey et al., 2005). Investigation into diagrams in L2 processing will offer plausible explanations for the effectiveness (or lack thereof) of diagram-based CL pedagogy. Thirdly, while factors such as semantic abstractness and SOA have been recognized as important determinants of L1 mental imagery effects, these factors have not been thoroughly examined in previous L2 mental imagery studies.

Therefore, our study aims to address these gaps and examine mental imagery effects in the context of online processing of English prepositions. English prepositions are known to be challenging for many L2 learners (Tyler et al., 2011). They have also garnered significant attention from cognitive linguists, who have created rich materials, including schematic diagrams and explanations that clarify the relationship between the spatial and extended (metaphorical) senses (Tyler & Evans, 2003). Despite the abundance of diagrams in the CL literature, the present study focused on over and in as target prepositions. These diagrams have previously demonstrated efficacy as L2 instructional aids in classroom settings (Wong et al., 2018; Zhao et al., 2020), making them intriguing candidates for further exploration of their psychological underpinnings. To investigate the extent to which L2 learners’ mental imagery patterns resemble those of NSs, we have collected data from adult English NSs to establish a baseline. Specifically, the study investigates the following research questions:

1. a) Do schematic diagrams yield a mental imagery effect in adult NS speakers’ and L2 learners’ online processing of English prepositional phrases (PPs)?

2. b) To what extent is any observed mental imagery effect modulated by target prepositions, abstractness of senses and SOAs?

2. Method

2.1 Participants

A total of 79 Chinese adult L2 learners of English (4 males and 75 females, mean age = 24.7, SD = 2.1) participated in the study. All were postgraduate students pursuing master’s degrees at a large Australian public university. Their first language was Mandarin Chinese and their average self-rated Chinese proficiency was 93.4 out of 100 (SD = 10.25). These participants achieved an English proficiency level of IELTS overall 6.0 (M = 6.9, SD = 0.4), corresponding to the B2 to C2 levels of the Common European Framework of Reference (Council of Europe, 2020). On average, they had 15.4 years of English learning experience (SD = 3.8) and an average of 12.9 months of residence in English-speaking countries (SD = 15.8). They reported an average of 10 hours per week of English communication (SD = 11.8). The baseline group consisted of 100 adult NSs of English (47 males and 53 females) with an average age of 44.1 (SD = 14.4). These NSs were recruited from Amazon Mechanical Turk and reported residing in one of the four English-speaking countries (87 in the USA, 10 in the UK, 2 in Canada and 1 in Australia). Their average self-rated English proficiency was 97.5 (SD = 7.5).

All participants were randomly allocated to one of two SOA conditions (1,040 ms vs. 2,040 ms). L2 learner participants in the two SOA conditions demonstrated comparable English proficiency, as indicated by their IELTS overall scores (t = −1.230, p = 0.223). A total of 21 participants (18 NSs and 3 NNSs) with accuracy rates below 70% in the priming task were excluded from the analysis due to data reliability concerns. After this exclusion, the final dataset included 76 L2 learners and 82 NSs of English (with 38 NNSs and 41 NSs in each SOA condition). All participants provided written informed consent and completed the same tasks, receiving monetary compensation for their participation.

2.2 Design and materials

We chose to focus on the prepositions over and in as they represent distinct spatial schemas in mental imagery: UP-DOWN and CONTAINMENT, respectively. Over exhibits a more complex prepositional polysemy, encompassing both a prominent location sense (e.g. The lamp is over the table) and an equally notable motion sense (e.g. He walks over the hill; Herskovits, 1997). Despite these variations, both senses imply that the TR is positioned higher than the LM, leading Tyler and Evans (2003) to argue that the locational HIGHER THAN sense is the primary sense of over ¹. This primary sense is abstracted from specific spatial scenes and is referred to as the proto-scene of over. On the other hand, in, along with on and at, constitutes a fundamental set of prepositions in English (Kelleher et al., 2009). The prototypical sense of in (CONTAINMENT) is topological in nature (Kelleher et al., 2009), with minimal variation on its LM dimensionality. The LM can represent a canonical three-dimensional space (e.g. in the box) or a non-canonical two-dimensional planar surface (e.g. in the street; Tyler & Evans, 2003). In the polysemy systems of these prepositions, at least one extended sense is chained to its prototypical spatial sense, a connection motivated by conceptual metaphors (MORE IS UP and STATE IS A CONTAINER for over and in, respectively; Lakoff, 1987; Lakoff & Johnson, 1980).

The schematic diagrams depicting the prototypical spatial configurations of the two prepositions were adopted from Tyler and Evans (2003) (Table 1). Each diagram contains a TR and an LM, with the TR being conceptually movable and the LM stationary (Herskovits, 1997). The original schematic diagrams of over and in in Tyler and Evans (2003) were presented as black-and-white images. In our version, we enhanced the visual contrast saliency between the TR and LM by highlighting the TR with a red dot and thickening the black lines to represent the LM. In the over diagram, the area below the dashed line represents potential contact between the TR and LM (Tyler & Evans, 2003). In the in diagram, the space below the dashed line represents where the movable TR may be contained by the LM (Tyler & Evans, 2003).

Table 1. Target prepositions, schematic diagrams, and senses (Tyler & Evans, 2003)

Note: LM, landmark (ground); TR, trajector (figure).

We opted for the two diagrams provided by Tyler and Evans (2003) for several important reasons. Firstly, these diagrams align with our focus on the two target prepositions and encapsulate their central senses as identified by Tyler and Evans (2003). Their CL analysis of prepositions is argued to be grounded in speaker usage and holds potential relevance for L2 speakers (Tyler, 2012). Furthermore, we avoided diagrams containing graphic elements such as axes and arrows, as they do not pertain directly to the spatial relations inherent in our target prepositional meanings. We also avoided diagrams with explanatory text to prevent the introduction of additional cues that could interfere with the processing of the target prepositional phrase. Additionally, we eschewed diagrams featuring realistic picture elements, as our investigation encompassed both the spatial and metaphorically extended non-spatial sense of each preposition. Opting for abstract diagrams without realistic pictures allows for the same graphic representation of both concrete and abstract senses of the prepositions. Lastly, the distinct visual presentation of the over and in diagrams reduces the likelihood of confusion during rapid processing.

In the current study, participants first undertook a diagram–picture matching task, designed as training to ensure a consistent understanding of target schematic diagrams before the main semantic priming task. Both tasks were programmed and administered using PsyToolkit (version 3.4.2; Stoet, 2010, 2017).

2.2.1 The diagram–picture matching task

Given that schematic diagrams are abstract images with potentially ambiguous meanings that are not inherently self-explanatory, the diagram–picture matching task was formulated to aid participants in grasping the TR–LM spatial configurations depicted in the target diagrams. This was achieved by presenting participants with concrete pictures featuring drawings of real-world objects, which served as visual representations of sentences illustrating the spatial senses of over and in. As participants associated these scenes with schematic diagrams, it was anticipated that their relevant sensorimotor experiences would be activated and retrieved from long-term memory, enhancing their understanding of the spatial elements in the diagrams.

Task materials were two target diagrams and 30 pictures. Initially, 20 target sentences (10 per target preposition) and 10 filler sentences (involving non-target prepositions behind, between, out, at and towards) were created, all representing proto-spatial senses (Supplementary Material 1). Sentences were sourced from CL literature (Lindstromberg, 2010; Tyler & Evans, 2003) and the Corpus of Contemporary American English (COCA; Davies, 2008). A diagram–picture pair was deemed matched if the picture depicted a scene that illustrated the proto-scene of the target preposition, which is what the diagram represents. Figure 1 presents examples of matched pairs for over (Figure 1a: The bird flies over the city) and in (Figure 1b: A mole lives in a burrow). Conversely, a diagram–picture pair was considered mismatched if the picture portrayed a scene that did not illustrate the proto-scene of the target preposition (e.g. over diagram – in picture: A mole lives in a burrow). All pictures were black-and-white and created by a contracted professional graphic designer.

Figure 1. Sample stimuli of the diagram–picture matching task for over (a) and in (b).

Participants received written descriptions in their L1 before diagram–picture matching, explaining key features of the diagrams (e.g. the denotations of red dots, solid black lines and dashed lines). During the task, they assessed diagram–picture pairs (20 matched and 20 mismatched) with the diagram and picture randomly placed on either side of the screen. Participants determined if the pairs matched and received immediate corrective feedback. If a participant made a ‘matched’ judgment for a matched diagram–picture matching pair, the trial was recorded as accurate; otherwise, it was recorded as incorrect. Correct judgments indicated participants’ ability to comprehend the spatial configuration of diagrams by correctly associating the TR and LM in diagrams with real-world objects in realistic pictures. Participants who achieved accuracy above 70% after the 40 trials proceeded to the priming task. If not, they continued until reaching the required accuracy.

2.2.2 The semantic priming task

The task adopted a 2 relatedness (related, unrelated) × 2 target preposition (over, in) × 2 sense (spatial, extended) × 2 SOA (1,040 ms, 2,040 ms) factorial design, structured as a binary acceptability judgment task. Participants were presented with either a related or unrelated diagrammatic prime and then instructed to judge the acceptability of target prepositional phrases (PPs). Target PPs (e.g. over the horizon) were selected from COCA (Davies, 2008) and consisted of three words (Preposition + Determiner/Adjective + Noun), where the noun served as the LM in the configuration. Each prime was displayed as a schematic diagram with an embedded word. The embedded word representing the TR was annotated in the red dot. The related and unrelated primes shared the same TR word but were paired with different diagrams. For instance, the TR word sun annotated in the over diagram is a related prime for the target PP over the horizon (Figure 2), whereas the same TR word annotated in the in diagram served as an unrelated prime for the same target PP (Figure 3).

Figure 2. A related trial of the semantic priming task (over diagram – over phrase).

Figure 3. An unrelated trial of the semantic priming task (in diagram – over phrase).

Two task stimuli versions were created using a Latin square design, ensuring each prime word was presented to each participant only once, without overlap across prepositions and senses. Each version consisted of 152 stimuli, including 88 target PPs (22 target phrases × 2 prepositions × 2 senses) and 64 fillers (Supplementary Material 1). Fillers were intentionally constructed as ungrammatical phrases (e.g. the to city, in room our), encompassing both target prepositions (over, in) and other fillers (on, to, from, under, across, through, below, before, after, between). The task stimuli were presented in a random order.

We maintained matching prime word frequency, target PP frequency and event plausibility of the prime–PP combinations (e.g. sun | over the horizon) across different stimulus conditions (see Supplementary Material 1). These frequencies were verified using COCA (Davies, 2008). No significant differences were found in prime word frequency between prepositions (t = 1.251, p = 0.215) or between senses (t = 1.452, p = 0.151). Similarly, no significant differences were observed in target PP frequency between prepositions (t = −0.150, p = 0.881) or between senses (t = 0.037, p = 0.971).

An event plausibility task was conducted online using Qualtrics (Provo, UT, USA, version 12.2020). Additional 58 adult NSs of English were recruited from Amazon Mechanical Turk to participate in this norming task. Data from 4 participants were excluded due to unreliability. Participants were asked to rate the event plausibility of prime–PP combinations (e.g. helicopter | over the city) on a 7-point Likert scale, ranging from 1 (absolutely impossible) to 7 (definitely possible). No significant differences were observed between prepositions (t = −0.012, p = 0.990) or between senses (t = 1.050, p = 0.297).

The semantic priming task procedure (Figures 2 and 3) followed de Wit and Kinoshita (2015), with fonts being set as Arial 30. The task initiated with a 250-ms spot fixation (+) at the screen center, followed by a prime displayed for 1,000 or 2,000 ms, separated by a 40 ms interstimulus interval. Subsequently, a target phrase (e.g. over the horizon) appeared, and participants judged its acceptability by pressing the ‘F’ or ‘J’ as quickly as possible. The target phrase remained until participants responded or a maximum of 5,000 ms had elapsed. Each trial concluded with a 500-ms blank page before transitioning to the spot fixation for the next trial. Five training trials were set before the formal trials, with data excluded from the analysis. Only the RTs of correct responses were included in the analysis.

The two SOA conditions were designed following the SOA of 1,500 ms in Lindsay (2003) and Wheeler and Stojanovic (2006). The design was also based on feedback from a pilot study involving three advanced L2 learners who completed the two tasks. The pilot participants found a prime presentation time of less than 1 s to be too short for meaningful processing of the prime and expressed greater comfort and better experiences with an extended 2-s presentation time. Pilot study data were excluded from the final analysis.

2.3 Statistical analysis

The R statistical platform (version 4.0.3; R Core Team, 2020) was used for data analysis. Before analysis, data trimming removed responses shorter than 200 ms and longer than 3,000 ms (2.3% of the data points). The lme4 package (version 1.1–29; Bates et al., 2022) and lmerTest package (version 3.1–3; Kuznetsova et al., 2017) were used to construct mixed-effects models to examine fixed effects and random effects on RTs (Linck & Cunnings, 2015), with an alpha value set at 0.05 for all models. Separate models were built to analyze the RTs of NS and L2 learner groups respectively.

A simple contrast-coding scheme (−0.5, 0.5) was applied to categorical variables, including preposition, sense, SOA and relatedness. Dummy coding (1, 0) was applied to accuracy status in the priming task (correct coded as 1; incorrect as 0). Numerical variables including RTs and covariates (i.e. prime word frequency, target PP string frequency, syllable length of target PPs, prime–PP event plausibility, number of matching task trials, matching task accuracy) were log-transformed (natural log). All models were fitted with participants and items as random effects, with random intercepts for participants and items, a by-item random slope for SOA and by-participant random slopes for prepositions and senses. Model convergence was confirmed, and statistical assumptions were checked. The emmeans package (version 1.8.4; Lenth et al., 2023) applied Tukey correction for pairwise comparisons. Cohen’s d (1977) was reported as the effect size for RTs, interpreted based on Plonsky and Oswald’s (2014) recommendations: 0.60, 1.00, 1.40 corresponding to small, medium and large effect sizes for within-subject contrasts, and 0.40, 0.70 and 1.00 as small, medium and large effect sizes for between-group contrasts. Graphics were generated using the ggplot2 package (version 3.4.0; Wickham, 2016).

3. Results

3.1 The diagram–picture matching task

Descriptive statistics of the number of trials, RTs and accuracy rates (ARs) of the diagram–picture matching task are presented in Supplementary Material 2. For the L2 learner group, results of paired-sample t-tests showed the RTs for judging over diagrams were significantly longer than in diagrams (MD = 1,024.54, t = 0.32, p < 0.001, 95% CI [0.58, 1.10], Cohen’s d = 0.84). It was also found the acceptance ARs of the over diagram were significantly higher than the in diagram (MD = 0.05, t = 2.52, p = 0.01, 95% CI [0.06, 0.52], Cohen’s d = 0.29) but the rejection ARs of judging the over diagram were significantly lower than the in diagram (MD = 0.04, t = 2.03, p = 0.046, 95% CI [0.00, 0.46], Cohen’s d = 0.23). For the NS group, results of paired-sampled t-tests showed no difference between the RTs (t = 1.91, p = 0.06), acceptance ARs (t = 1.31, p = 0.20) or rejection ARs (t = −0.70, p = 0.49) of over and in diagrams.

3.2 The semantic priming task

Descriptive statistics of the RTs and ARs of the semantic priming task performance of the L2 learner and NS groups are presented in Supplementary Material 2.

3.2.1 The L2 learner group: Response times results

A full linear mixed-effects model on L2 RTs was built by including all the fixed effects (with interactions) and covariates (Supplementary Material 3). The proficiency (log(IELTS)) × relatedness interaction was built into the full model but turned out to be not significant. We then removed this interaction and only kept proficiency as a covariate. All the non-significant covariates were removed to construct the second model⁴, which proved to be a better one than the full model according to both Akaike information criterion (AIC) and Bayesian information criterion (BIC) values. Table 2 presents the model 2 output.

Table 2. L2 learners’ response time results of linear mixed effects modeling

Note: ***p < 0.001, **0.001 < p < 0.01, *0.01 < p < 0.05.

Preposition and sense had significant fixed effects on RTs, while relatedness and SOA did not create significant fixed effects. The post hoc analyses showed the mean RTs for over phrases (M = 1,061, SE = 34.0, 95% CI [987, 1,139]) were estimated to be 81 ms longer than those of the in phrases (M = 980, SE = 28.7, 95% CI [918, 1,046]; Cohen’s d = 0.28, SE = 0.10, 95% CI [0.09, 0.47], corresponding to a small effect). The mean RTs for extended senses (M = 1,072, SE = 32.9, 95% CI [1,001, 1,148]) were estimated to be 103 ms longer than spatial senses (M = 969, SE = 29.0, 95% CI [906, 1,036]; Cohen’s d = 0.36, SE = 0.09, 95% CI [0.19, 0.54], corresponding to a small effect). English proficiency indicated by the IELTS overall score was a significant covariate, indicating participants with a higher L2 proficiency spent less time processing English PPs than lower-proficiency participants.

The model also yielded significant two-way interactions. A preposition × relatedness interaction (Figure 4) revealed that the mean RTs of judging over phrases in related trials (M = 1,077, SE = 34.9, 95% CI [1,002, 1,158]) were estimated to be 33 ms slower than over unrelated trials (M = 1,044, SE = 33.9, 95% CI [971, 1,123]; t = 3.07, p = 0.002, Cohen’s d = 0.11, SE = 0.04, 95% CI [0.04, 0.18], corresponding to a small effect) but there was no difference between related and unrelated trials in the RTs of judging in phrases (t = −1.21, p = 0.23). A SOA × relatedness interaction (Figure 5) suggested that the mean RTs in related trials (M = 1,067, SE = 39.3, 95% CI [983, 1,159]) were estimated to be 25 ms longer than in unrelated trials (M = 1,042, SE = 38.4, 95% CI [960, 1,132]) in the 2,040 ms SOA condition (t = 2.36, p = 0.02, Cohen’s d = 0.08, SE = 0.04, 95% CI [0.01, 0.16], corresponding to a small effect), but not such a difference was observed in the 1,040 ms SOA (t = −0.50, p = 0.62).

Figure 4. Response times of related and unrelated trials by preposition. Black bars represent medians, and green dots represent the individual response times per participant per trial.

Figure 5. Response times of related and unrelated trials by stimulus onset asynchrony. Black bars represent medians, and blue dots represent the individual response times per participant per trial.

3.2.2 The native speaker group: Response times results

A full linear mixed-effects model on English NSs’ RTs was built by including all the fixed effects (with interactions) and covariates. All the non-significant covariates were removed to construct the model, which proved to be better than the full model according to both AIC and BIC values (Supplementary Material 3).

Among the fixed effects, target preposition (b = −0.04, SE = 0.01, 95% CI [−0.06, −0.01], t = −2.53, p = 0.012) was found to be a significant predictor of the RTs of NSs’ PP judgment. NSs spent significantly longer time (35 ms) judging over phrases (M = 993, SE = 30.4, 95% CI [927, 1,065]) than judging in phrases (M = 958, SE = 27.9, 95% CI [897, 1,024]; Cohen’s d = 0.15, SE = 0.06, 95% CI [0.03, 0.26], corresponding to a small effect). The event plausibility of prime–PP combinations was the only significant covariate (b = −0.51, SE = 0.16, 95% CI [−0.82, −0.19], t = −3.16, p = 0.002), implying the prime–PP combinations with a higher event plausibility were processed faster than those with a lower event plausibility.

4. Discussion

The study investigates Chinese L2 English learners’ mental imagery in processing English PPs in comparison with English NSs’ mental imagery in L1 processing via visual abstraction of schematic diagrams. RT results revealed interference effects on processing of over phrases and PPs under 2,040 ms SOA in the L2 learner group but not in the English NS group. Several factors influenced RTs of phrasal judgment. Preposition had a main effect on RTs in both L1 and L2 processing, resulting in longer RTs for processing over phrases. Sense influenced RTs in L2 PP processing, with extended senses leading to longer RTs compared to spatial senses. Besides, L2 learners’ RTs decreased with higher L2 proficiency, while NSs’ RTs decreased with greater event plausibility.

4.1 Selective interference effects of diagrams on L2 online processing

The interference in the L2 learner group but not in the NS group aligns partially with previous findings in L2 mental imagery (Bergen et al., 2010; Wheeler & Stojanovic, 2006). The discrepancy in mental imagery effects between the two participant groups could be attributed to the varying cognitive demands involved in processing schematic diagrams and linguistic stimuli. Many L1 mental imagery studies reported compatibility effects using concrete pictures (de Koning et al., 2017a; Engelen et al., 2011; Schütt et al., 2023) with a higher level of embeddedness and proximity to contextualized situations (Zwaan, 2014), thus resulting in the least cognitive demands and less difficulty integrating the pictures with linguistic contexts, whereas the current study used schematic diagrams, representing symbolic mental representations with lower embeddedness and higher abstraction, making the processing more cognitively demanding than pictures, which offsets the compatibility effects in the NS group.

The interference effect in the L2 learner group may be due to mutual inhibition and information integration. Firstly, the interference effects might stem from mutual inhibition arising from the competition between simultaneously processing visual and linguistic input through the same perceptual-motor neural circuit (Bergen et al., 2010; Kaschak et al., 2005). According to the theory of mirror neurons (Perry et al., 2018), neural structures can be activated when closely related words and pictorial representations of similar actions are presented (Bergen et al., 2003). In the current design, the TR word was embedded in the diagram and presented simultaneously as the prime. This activation process may lead to mutual inhibition between two mirror neural representations, one for the TR word (linguistic representation) and the other for the diagrammatic element representing the TR word (visual representation), as they compete with one another.

Additionally, L2 learners might be more affected in integrating information from various multimodal sources (Martin et al., 2016; Romero-Rivas et al., 2017), given their reduced degree of embodiment in L2 compared to L1 (Dudschig et al., 2014; Foroni, 2015). For late bilinguals who acquired their L2 in a formal school setting, L2 lexical concepts were initially stored as mental representations in their L1 (Silverberg & Samuel, 2004), resulting in weak associations between L2 forms and mental representations (Dudschig et al., 2014; Monaco et al., 2019). Another contributing factor is that L2 learners tend to have a relatively limited working memory capacity for L2 processing compared to that of NSs for L1 processing, and these discrepancies are particularly noticeable in late bilinguals (Michael & Gollan, 2005; Zhou et al., 2017). The current priming design involves the separate presentation of the TR and LM words in the diagrammatic prime and target phrase. This separation may hinder participants’ mental construction of the complete TR–LM spatial configuration when they solely view the diagrammatic prime. Consequently, this complexity places greater demands on cognitive resources for processing the initial stimuli, which limits the cognitive resources to process the subsequent stimuli.

The study identifies selective interference effects in L2 learners’ processing of over phrases, when presented with a related prime of the over diagram embedded with a TR word, but relatedness was not significant for the in-phrase processing. This preposition-specific interference effect can be attributed to both the lexical properties of over and the over diagram. As a preposition, over is less frequent but more semantically complex than in. The frequency difference (1,222,405 instances for over versus 16,541,037 instances for in according to COCA) implies that speakers have fewer encounters with the various uses of over, potentially leading to a lower quality of conceptual understanding (Bybee, 2010).

Additionally, the preposition over exhibits greater semantic complexity and polysemy compared to in. In terms of dynamicity, the prototypical sense of over (i.e. HIGHER THAN) is abstracted from both static location (e.g. The lamp is over the table) and dynamic motion senses (e.g. He walks over the hill; Herskovits, 1997). Conversely, the prototypical sense of in (CONTAINMENT) is abstracted solely from static senses (e.g. in the car, in danger), as dynamic containment is typically expressed by the preposition into rather than in (Tyler & Evans, 2003). Therefore, over, incorporating both static and dynamic senses in its prototypical sense, is semantically more intricate than in, whose extended senses remain static. Additionally, Tyler and Evans’ (2003) principled polysemy model indicates that over (with 14 extended senses in a five-cluster semantic network) demonstrates a higher degree of semantic polysemy than in (with 7 extended senses in four clusters), suggesting a more complex semantic structure for over. Given its greater polysemy and less direct chaining between its proto-scene and extended senses, comprehending the semantics of over can pose a challenge, even for NSs (Coventry & Mather, 2002). This complexity is evidenced by the longer RTs observed for judging over phrases in both NSs and L2 learners in our study.

As a diagram, over is more ambiguous and less reliable than in. The fuzziness in the vertical separation of TR and LM complicates precise localization in a vertically oriented image (Hampe & Grady, 2005). As a result, the over diagram may project the central sense of above (Lakoff, 1987). Speakers might also interpret it as a spatial representation of up or on. Thus, the over diagram could lead to ambiguity with multiple interpretations (Tversky, 2015), and pose challenges in its processing. In comparison, the in diagram serves as a more reliable cue for the preposition in and containment interpretation. It cannot be used for other prepositions, like into, out, out of and through, maintaining a clear prototypical meaning and interpretation.

We contend that the potentially ambiguous over diagram is more cognitively demanding than the in diagram, supported by diagram–picture matching results. While NSs showed no differences in RTs, acceptance or rejection ARs between over and in diagrams, L2 learners spent significantly more time processing over diagrams and had lower rejection ARs. The longer RTs suggest deliberation and higher rejection ARs indicate the greater ambiguity of the over diagram. Rejecting diagram–picture mismatching pairs demands a higher level of understanding of the diagram, requiring participants to grasp both the TR and LM while distinguishing the depicted spatial configuration of the over diagram from others. Therefore, the diagram–picture matching results underscore that L2 learners invest more cognitive effort in processing the over diagram.

Finally, selective interference effects of diagrams emerged in L2 learners’ PP processing only at a 2,040 ms SOA, differing from previous patterns observed at shorter SOA (e.g. Bergen et al., 2007). This finding contrasts with prior studies but aligns with Bergen et al. (2003) and Wheeler and Stojanovic (2006), where a longer SOA (up to 1,500 ms) revealed interference effects. The longer SOA enables deeper prime processing (Shaki & Fischer, 2023), allowing L2 learners more time to digest the primes by integrating the TR word with diagrams. This higher processing depth also demands greater cognitive effort in mapping integrated information onto the LM in the target phrase.

4.2 Factors that influence L1 and L2 processing

The study identified that target prepositions influenced the RTs of phrasal judgment in both participant groups, with a greater impact on L2 learners compared to NSs. Despite advanced English proficiency in L2 learners, they exhibited nativelike sensitivity to prepositions, but with a slower processing speed. The faster responses to over phrases were not due to the differences in syllable length between the over versus in phrases, as it was not a significant covariate in both mixed models, thus statistically controlling this potential confounding variable (Sonbul, 2015). Instead, we attribute the phenomenon to the higher complexity and lower frequency of over in corpus, which has been explained in the previous discussion.

In terms of abstractness of sense, L2 learners processed extended senses slower than spatial senses, which was not observed in NSs. This difference could stem from the figurative nature of extended senses, rooted in conventional conceptual metaphors MORE IS UP and STATE IS A CONTAINER (Lakoff, 1987). Understanding figurative language involves cognitive effort in activating conceptual mapping between target and source domains (Lai & Curran, 2013), posing a greater challenge for L2 learners who lack sensitivity and familiarity with the metaphors in the target language (Horvat et al., 2021; Littlemore et al., 2011). It implies that L2 learners may compute the meaning of word components to reach an integral phrasal comprehension, contrasting with NSs who tend to process it as a whole with minimal computation during the phrasal processing (Shi et al., 2023).

The findings on covariates also highlight a significant distinction between L1 and L2 processing. The RTs of NSs were influenced by the event plausibility of prime–PP combinations (e.g. sun | over the horizon), while L2 learners did not display a strong sensitivity to this input factor. This finding aligns with usage-based studies of language acquisition (Ellis et al., 2008; Gries et al., 2005; Zhao et al., 2020), which demonstrate that NSs and L2 learners are sensitive to different input-based statistical information. NSs are primarily influenced by associational information (event plausibility defined by the association between the prime word and the phrasal context), while L2 learners often fail to show the same degree of sensitivity to contextualized probability as NSs.

Finally, the quality of L2 phrasal processing was modulated by the level of L2 proficiency (Wolter & Yamashita, 2018), as evidenced by the current finding that higher L2 proficiency leads to faster RTs in L2 learners’ PP judgment. This suggests that higher L2 proficiency enhances grammatical sensitivity in PP judgment.

4.3 The validity of schematic diagrams as representations of concepts

The discovery of selective interference effects in the L2 learner group provides compelling evidence of the presence of mental imagery in L2 processing and bolsters the validity of schematic diagrams as conceptual representations. Future studies may identify compatibility effects, serving as additional evidence to substantiate the validity of schematic diagrams. We acknowledge the current limitations and provide recommendations for future researchers and implications for future L2 instructors.

Firstly, we suggest future studies to increase the integrability of visual and linguistic information in the prime with the target. Considering that linguistic information could be more readily comprehensible compared to diagrammatic visual information, reversing the current visual–linguistic stimuli sequence by presenting linguistic stimuli (i.e. PP) first as the prime followed by diagrams embedded with TR word (and LM word) as the target might lead to the emergence of compatibility effects. This paradigm finds support in previous studies that presented sentences as primes and pictures as targets in the sentence–picture verification tasks (Ahn & Jiang, 2018; Bergen, 2007; Chen et al., 2020; de Koning et al., 2017a; Wang & Zhao, Under Review).

Secondly, it is advisable for future research to explore diagrams with varying degrees of transparency and graphic diversity. The current study only examined the over and in diagrams provided by Tyler and Evans (2003) as exemplars among the abundant diagrams in CL literature and empirical research. These two diagrams are somehow abstract in nature, employing only fundamental graphic symbols like lines and dots to convey meanings, without incorporating supplementary graphic elements such as explanatory text or realistic imagery. We encourage future research to investigate a broader range of diagrams in subsequent experiments, encompassing other varieties of graphic elements including arrows, axes, text labels, shapes, symbols, colors, realistic pictures and so forth. This investigation can contribute to the field of mental imagery by addressing whether the graphic diversity of diagrams affects cognitive processing demands and their interaction with language processing.

Thirdly, we encourage future research in mental imagery to incorporate diagrammatic and linguistic stimuli that pertain to other abstract domains. Schematic diagrams, composed of abstract visual symbols, may be more suitable than pictures to investigate mental representations associated with abstract grammatical and semantic domains such as time, force dynamics, countability and figurativeness. These abstract domains have their roots in concrete spatial domains (Lakoff, 1987), such as motion paths (Wang & Zhao, Under Review). We argue that diagrams may offer a more appropriate means of representing abstract concepts that are invisible and challenging to describe solely through language, potentially leading to more pronounced mental imagery effects.

Finally, in terms of pedagogical implications, the present study contributes to the existing CL literature and provides a rationale for incorporating schematic diagrams into CL-based L2 instructions. These findings are in line with previous CL-based L2 instructional studies, which have suggested that the use of pictorial elucidations can potentially lead to distraction effects on L2 learners’ acquisition of relatively complex linguistic items (Boers et al., 2008, 2009). Given the observed selective interference effects, caution can be warranted in selecting and using diagrams as instructional tools. For instance, when employing schematic diagrams to teach prepositions like over to L2 learners, providing explicit explanations is advisable to mitigate potential ambiguity or confusion.

In conclusion, the study introduces an innovative experimental paradigm, integrating semantic information of the TR word with image-schematic representation, influencing real-time language processing by encapsulating a figure–ground relation. Selective interference effects were observed in L2 learners’ processing of over phrases and PPs under the longer SOA condition, which suggests that mental imagery effects were modulated by target preposition and SOA but not sense. The findings support the mental imagery process in the L2 processing of English prepositions and the psychological reality of image-based semantic representations. The study provides new empirical evidence on the interaction between visual perception, mental imagery and linguistic processes, and inspires the application of schematic diagrams in future psycholinguistic studies.