Cross-linguistic interactions across modalities: Effects of the oral language on sign production

Abstract

To investigate cross-linguistic interactions in bimodal bilingual production, behavioural and electrophysiological measures (ERPs) were recorded from 24 deaf bimodal bilinguals while naming pictures in Catalan Sign Language (LSC). Two tasks were employed, a picture-word interference and a picture-picture interference task. Cross-linguistic effects were explored via distractors that were either semantically related to the target picture, to the phonology/orthography of the Spanish name of the target picture, or were unrelated. No semantic effects were observed in sign latencies, but ERPs differed between semantically related and unrelated distractors. For the form-related manipulation, a facilitation effect was observed both behaviourally and at the ERP level. Importantly, these effects were not influenced by the type of distractor (word/picture) presented providing the first piece of evidence that deaf bimodal bilinguals are sensitive to oral language in sign production. Implications for models of cross-linguistic interactions in bimodal bilinguals are discussed.

1. Introduction

A well-established phenomenon in the literature is that bilinguals cannot restrict lexicalization to one of their languages. While speaking, listening, or reading, bilinguals’ two languages are simultaneously and automatically activated, revealing that lexical access in bilinguals is largely language non-selective (for discussion see, e.g., Kroll, Bogulski & McClain, 2012). Evidence of cross-linguistic interactions comes from studies on word comprehension (Marian & Spivey, 2003b, 2003a; Wu & Thierry, 2010) and word production (Colomé & Miozzo, 2010; Hermans, Bongaerts, De Bot & Schreuder, 1998), showing that activation of the non-intended language influences processing in bilingual's intended language.

Critically, the non-selective nature of bilingual lexical activation has also been shown in bilinguals with two languages of different modality (oral and sign), termed “bimodal bilinguals”. A number of experiments have showed that deaf and hearing bimodal bilinguals activate sign properties when processing words (Kubus, Villwock, Morford & Rathmann, 2015; Morford, Kroll, Piñar & Wilkinson, 2014; Morford, Occhino-Kehoe, Piñar, Wilkinson & Kroll, 2017; Morford, Wilkinson, Villwock, Piñar & Kroll, 2011; Shook & Marian, 2012; Villameriel, Dias, Costello & Carreiras, 2016). For example, Morford et al. (2011) showed that phonological relationships in American Sign Language¹ (ASL) influenced semantic similarity judgements of written word pairs in English (see also Villameriel et al., 2016, for similar results with hearing bimodal bilinguals and Morford et al., 2014, for a different result with hearing bimodal bilinguals).

Much scarcer is the evidence showing cross-linguistic influences of words on sign processing (Emmorey, Mott, Meade, Holcomb & Midgley, 2020; Giezen & Emmorey, 2016; Hosemann, Mani, Herrmann, Steinbach & Altvater-Mackensen, 2020; Lee, Meade, Midgley, Holcomb & Emmorey, 2019). Using the same paradigm as Morford et al. (2011), Lee et al. (2019) showed that hearing bimodal bilinguals were sensitive to the phonological relationship of the English-translations (i.e., rhymed) while judging the semantic relationship of ASL sign pairs. Relevant here, results were not replicated in the deaf group, unless deaf individuals were aware of the English phonological manipulation.

While those results help demonstrate that cross-linguistic interactions in bilinguals are not modality-specific, they are also suggestive that, as in unimodal bilingualism, cross-linguistic interactions are not a ubiquitous phenomenon. At least two factors should be considered when exploring cross-linguistic interactions in bilinguals. The first relates to language dominance and proficiency. Cross-linguistic influences from L2 to L1 are weaker than the reverse and they only occur when sufficient proficiency in L2 has been attained (Van Hell & Tanner, 2012). Because most deaf signers are more dominant and proficient in sign language than in the oral language, this imbalance between languages could explain the lack of L2 (spoken) influence on L1 sign comprehension for deaf bilinguals in Lee et al. (2019). The second factor, and more specific to bimodal bilingualism, relates to the mechanisms of phonological activation of the oral language, which might be different between hearing and deaf bilinguals. Because deaf bimodal bilinguals acquire the oral (L2) language via the written form (e.g., also referred as sign-print bilinguals; Piñar, Dussias & Morford, 2011), phonological effects could be enlarged when spoken phonology is directly induced by the written language or, as showed in Lee et al. (2019), in those deaf bilinguals with higher phonological awareness of the oral language.

Keeping these factors in mind, in the present study we tested deaf bimodal bilinguals during sign language production to further characterize cross-linguistic effects of the oral L2 on the sign L1 language. Before describing our study, cross-linguistic effects in language production and theoretical models of bilingual lexical selection are described.

1.1. Cross-linguistic effects in language production

In language production, most of the evidence on cross-linguistic interactions comes from studies using interference tasks. In these tasks, both a picture and a distractor are presented; participants are then asked to name the picture while ignoring the distractor. Experimental manipulations of the relationship between the distractor and the picture have been studied to inform models of bilingual language production (for a review see Hall, 2011). In particular, semantic and phonological effects² in picture-interference tasks have produced a fruitful debate concerning the role of competition in bilingual lexical selection.

Considering semantic effects, semantically related distractor words in the non-intended language (e.g., distractor: gato (cat) – target: DOG) elicit semantic interference (slower naming latencies and more errors in the semantically related condition than in the unrelated condition; Costa & Caramazza, 1999; Costa, Colomé, Gómez & Sebastián-Gallés, 2003; Hermans et al., 1998). Models of bilingual language production have explained these semantic interference effects as a result of lexical competition between the two languages (Hermans, 2004; Hermans et al., 1998) or within the intended language (Costa et al., 2003; Roelofs, Piai, Garrido Rodriguez & Chwilla, 2016). Even though both views propose that interference is based on conflict at the lexical level, between-language competition assumes that lexical selection is accomplished through competition of all activated candidates regardless of language, and within-language competition assumes that distractors are automatically translated and then competition only occurs among lexical candidates in the target language. Alternatively, models assuming non-competitive lexical selection (Response Exclusion Hypothesis; Mahon, Costa, Peterson, Vargas & Caramazza, 2007) explain semantic interference effects as arising post-lexically, from control processes operating just prior to articulation.

At the ERP level, most of the picture-word interference studies have reported N400-like modulations of the semantic effect. Less negative ERP for pictures presented with semantically-related words (relative to semantically-unrelated words) have been taken as an index of semantic priming in competitive and non-competitive accounts of speech production alike (Blackford, Holcomb, Grainger & Kuperberg, 2012; Dell'Acqua et al., 2010; Koester & Schiller, 2008; Piai, Roelofs, Jensen, Schoffelen & Bonnefond, 2014; Roelofs et al., 2016; Zhu, Damian & Zhang, 2015). Importantly here, differences between models arise in the predicted timing of the semantic interference effects. While lexical competition models predict a semantic effect during lexical selection, the response exclusion account predicts a later effect, much closer to the speech onset. Considering the time estimates of lexical selection in simple picture naming, which occurs at around 200 ms (e.g., Costa, Strijkers, Martin & Thierry, 2009), ERP modulations occurring at ~200 ms have been taken as competition occurring at the lexical level, therefore supporting lexical-competition models. Conversely, semantic effects starting at ~400 ms may imply a post-lexical locus, supporting non-competitive lexical selection models. However, it is not trivial to map time course estimates from simple picture naming to the picture-word interference task (in which naming latencies are prolonged). In addition, the electrophysiological literature does not show a consistent picture regarding the time course of semantic distractor effects, with studies showing earlier (200–500 ms; Aristei, Melinger & Rahman, 2011; Hoshino & Thierry, 2011) and later effects (325–600 ms; Blackford et al., 2012), making it difficult to localize the origin of semantic effects during speech production.

Relative to unimodal bilingualism, studies with bimodal bilinguals seem to favour predictions of the response exclusion hypothesis. Giezen and Emmorey (2016) and Emmorey et al. (2020) found no behavioural semantic interference effects in the picture-word interference task when hearing or deaf bimodal bilinguals were signing pictures in the presence of auditory or written distractors. According to the response exclusion account, semantic interference effects are not predicted in sign language because there should be no post-lexical conflict between signs and word responses (Emmorey et al., 2020; Giezen & Emmorey, 2016). Conversely, a semantic facilitatory effect is predicted as a result of activation of the semantic properties of the picture caused by a semantically related word prime. Indeed, Emmorey et al. (2020), obtained a semantic facilitation effect supporting this prediction (cf. Giezen & Emmorey, 2016). In addition, consistent with a semantic priming effect, they observed an ERP modulation starting around 300 ms, with pictures paired with semantically related words showing an early N400-like attenuation. Interestingly, semantic effects in Emmorey et al. (2020) matched the timing obtained in Baus and Costa (2015) when lexical variables (i.e., lexical frequency, iconicity) were manipulated in a picture signing task, which reinforces the idea that language interactivity is modality invariant and has a lexical origin (Shook & Marian, 2012).

Here we further explored behavioural and ERP correlates of semantic processing in deaf bimodal bilinguals. If as described by the response exclusion account there is no competition at the articulatory buffer between signs and words (Emmorey et al., 2020; Giezen & Emmorey, 2016), then we should obtain a facilitatory effect of semantic relatedness behaviourally and a semantic priming effect at the ERP level. Note however that even if signs and words do not compete (at lexical or articulatory levels) semantic effects could be also expected as a result of within-(oral) language competition. Unlike unimodal bilinguals, bimodal bilinguals can produce signs and words at the same time (i.e., code-blending) because sign and oral languages use different motor systems. In this context, lexical competition might occur among candidates of the non-intended language (oral modality), which might end up affecting how words and signs are synchronized to produce a code-blend sign (Hosemann et al., 2020; Vinson, Thompson, Skinner, Fox & Vigliocco, 2010). We return to this issue in the discussion.

Experiments with phonological distractor manipulations have broadly shown that distractors in the non-intended language which are phonologically related to the target facilitate picture naming (e.g., distractor: dos (two) – target: DOG; Colomé & Miozzo, 2010; Costa et al., 2003; Costa, Miozzo & Caramazza, 1999; Hermans et al., 1998). Conversely, distractors in the non-intended language which are phonologically related to the target's translation slow down picture naming (e.g., distractor: pera (pear) - target: DOG (perro in Spanish); Boukadi, Davies & Wilson, 2015; Costa et al., 2003; Hermans et al., 1998; Hoshino & Thierry, 2011; Knupsky & Amrhein, 2007; but see Costa, Albareda & Santesteban, 2008). Similar to semantic effects, phonological interference effects have been attributed to competition at the lexical level in models assuming between-language competition (Hermans, 2004) and to competition at the phonological level by models assuming within-language lexical competition (Costa et al., 2003; Roelofs et al., 2016).

At the ERP level, Hoshino and Thierry (2011) showed similar semantic and phono-translation effects in a picture-word interference task. The behavioural interference occurred in the presence of reduced negativities for the semantic and phono-translation conditions relative to the unrelated condition. Both semantic and phono-translation effects elicited ERP modulations in two time windows (at around 200 ms and 350 ms respectively), which were interpreted as evidence of cross-language competition at the lexical level and beyond.

To the best of our knowledge, cross-modal phonological effects through the oral language in picture-word interference have not been tested in bimodal bilinguals. Different directions of the phonological effect could be expected depending on within or between-language competition views. Following within-language competition views (Roelofs et al., 2016) interference could only occur at the phonological level which is shared across oral languages. Because conflict at the phonological level should not exist between sign and oral languages, facilitation should be observed due to priming of the translation-equivalent in the non-intended language. Note that the same result would be predicted in code-blending production, when mouthing is activated and articulated together with the sign. In contrast, if phonological interference effects are observed, as have been found in unimodal picture-word interference studies, this finding would support between-language competition views, where interference occurs at the lexical level (Hall, 2011). It should be noted that the response exclusion account has not been described to account for phonological effects in bilingual production. One tentative prediction for bimodal production could be that, since language membership is a response-relevant feature and no competition needs to be solved at the articulatory level, the phonology of the oral language is irrelevant and easily disregarded. In consequence, there should not be phonological influence from the oral language while signing.

1.2. The present study

In the present study, we explored cross-linguistic interactions in sign production by testing deaf bimodal bilinguals in two tasks, a picture-word and a picture-picture interference task. Comparing performance in two different interference tasks allowed us to examine whether cross-linguistic effects require the oral language to be directly activated by the (written) distractors in the task.

Semantic and phono-translation effects and their locus during sign production were evaluated, allowing us to test behavioural and electrophysiological traces of lexical selection processes in bimodal sign production. For example, the picture of a DOG (perro in Spanish) was presented with the distractor word or the distractor picture “gato” (cat in English; semantic condition), the word/picture “pera” (pear in English; form-related condition), or the word/picture “casa” (house in English; unrelated condition) superimposed on the target picture.

To explore cross-linguistic semantic effects (the contrast between the semantic and the unrelated condition) predictions necessarily must be put forward in the context of the picture-word inference task, given that results in the picture-picture interference task would not be informative regarding the involvement of the oral language in the task. A semantic effect in the picture-word interference task would demonstrate that activation of the oral lexicon (induced by the distractor word) influences sign production. If our results support the non-competitive nature of lexical selection, cross-modal co-activation should result in facilitation due to semantic priming because there should be no post-lexical conflict between sign and oral languages (for discussion see Emmorey et al., 2020; Giezen & Emmorey, 2016; Mahon et al., 2007).

The contrast between the form-related and the unrelated condition has the potential to reveal more about cross-linguistic interactions across modalities. Any differential effect of a distractor that is form-related to the Spanish name of the target picture would imply that Spanish was activated during LSC sign production and that the Spanish lexicon influenced sign production. In addition, if the phonological effects do not differ between tasks, it would suggest that these effects are not driven by the explicit presence of the Spanish language in the task.

2. Methods

2.1. Participants

Twenty-four deaf LSC-Spanish bilinguals (12 females, M age = 34.5 years, SD = 14.2 years) participated in the study. Twenty-two participants had profound hearing loss (91-120 dB), one participant had a severe hearing loss (71-90 dB), and one participant had a moderately-severe hearing loss (56-70 dB). Four participants reported using hearing aids, one reported the use of cochlear implants and nineteen participants did not use any type of hearing device. One additional participant was run, but excluded due to an excessive number of artefacts. All participants reported normal or corrected vision and no history of neurological problems. Self-ratings of LSC and Spanish proficiency were collected through a language background questionnaire (Table 1). All participants completed an informed consent form before the experiment and were paid for their participation.

Table 1. Demographic information of the participants. Mean ratings (M) and standard deviation (SD)

* Self-ratings from a language questionnaire; proficiency was rated on a 10-point scale ranging from ‘almost none’ to ‘very proficient’

2.2. Materials

A set of thirty pictures and a separate set of ninety picturable words were selected as targets and distractors, respectively, from different databases (Bates, D'Amico, Jacobsen, Székely, Andonova, Devescovi, Herron, Lu, Pechmann, Pléh, Wicha, Federmeier, Gerdjikova, Gutierrez, Hung, Hsu, Iyer, Kohnert, Mehotcheva, Orozco-Figueroa, Tzeng & Tzeng, 2003; Snodgrass & Vanderwart, 1980). In the picture-picture interference task, the stimuli consisted of two overlapping pictures, with targets in green and distractors in red. In the picture-word interference task, stimuli consisted of target pictures with a written superimposed word. In both tasks, each picture was paired with three different distractors (see table S1 in online supplementary material) and these distractor-set pairings were the same in both tasks. In the form-related condition, distractors were phonologically and orthographically similar to the Spanish name of the targets (e.g., CEREZA-cerebro; ‘cherry-brain’ in English), with phono-translation distractors and targets overlapping on 3 phonemes/letters on average (SD = 0.86; e.g., PINcel – PINgüino, ‘brush - penguin’ in English) and, in most cases, corresponding in their first syllables. Due to the nature of the Spanish language as a language with transparent orthography, materials selected based on phonological relations in Spanish were also mainly orthographically related. Thus, we refer to this condition as form-related condition. In the semantically related condition, distractors were from the same semantic field but were not form-related (e.g., CEREZA-manzana; ‘cherry-apple’ in English). A set of unrelated distractors were selected as the baseline condition (e.g., CEREZA-llave; ‘cherry-key’ in English). Targets and distractors were always phonologically unrelated in LSC and did not have obligatory mouth patterns as an intrinsic component of the sign. Furthermore, Spanish names for the distractor pictures were matched across conditions in number of phonemes/letters, lexical frequency, concreteness, and familiarity from the Spanish corpus B-Pal (Davis & Perea, 2005) (see table S2 in online Supplementary Material).

2.3. Procedure

Participants were tested individually in an electrically shielded and dimly lit room. Instructions and other communication during the experiment were given in LSC by a hearing proficient signer. The order of the two tasks, picture-picture interference and picture-word interference, was counterbalanced across participants³. In each task, stimuli were presented in two blocks of 45 trials, and each task began with a practice block of three warm-up trials. E-Prime 2.0 ® was used to present the stimuli and record signing latencies. At the beginning of each trial, an instructional message asked participants to press and hold the spacebar to start the trial. Then, a 500 ms black screen was followed by a 500 ms central fixation cross and a 300 ms black screen. Target-distractor pairs were then displayed and maintained until participants released the spacebar in order to sign the name of each target picture. A final 500 ms black screen appeared at the end of each trial. Signing latencies were calculated from the onset of the stimuli display until the key release (see Baus & Costa, 2015; Giezen & Emmorey, 2016; for the same method). Participants’ responses were recorded on video and checked for accuracy after the experiment ended. In addition to the signed responses, possible mouth movements elicited during sign production were checked by a hearing non-signer researcher.

2.4. Behavioural Analysis

Two target-distractor pairs were removed from all the analyses reported. One because participants reported a sign from the same semantic field instead of the desired sign (‘boat’ instead of ‘sailboat’), and the other because participants used the same signs adding mouthing to disambiguate between them instead of different signs (‘hair comb’ and ‘brush’).

We analysed the data by fitting linear mixed models, treating participants and items as crossed random factors (Baayen, Davidson & Bates, 2008). Models were fitted in R (R Core Team, 2019) using the package lme4 (Bates, Mächler, Bolker & Walker, 2015). Signing latencies where fitted with linear mixed models and error rates with generalized mixed models (binomial family). Models included fixed effects for task (sum coded), condition (treatment coded, unrelated condition as baseline), and their interaction. Significance of the fixed effects estimates was determined using the Satterthwaite approximation for degrees of freedom provided by the lmerTest package (Kuznetsova, Brockhoff & Christensen, 2015). Additional analyses on log transformed latencies (to alleviate problems related to non-normality) as well as additional analyses including fixed effects for task sequence, mouthing⁴, and its interactions with the other fixed effects lead to the same conclusions as the latency analyses reported here.

We aimed to fit models with the maximal possible random-effects structure (Barr, Levy, Scheepers & Tily, 2013). We started out with a maximal model containing random slopes for distractor condition, task, and their interaction for both participants and items. In cases of non-convergence, we step-wise simplified the random structure, by dropping random correlations and the interaction terms before dropping main effect slopes from the model. In case of singular model fits, we first dropped the interaction terms before dropping condition or task slopes with an estimated variance (close to) zero.

2.5. EEG recording and analysis

EEG activity was continuously recorded from 30 Ag-AgCl electrodes, mounted on an elastic cap (ActiCap, Munich, Germany) and positioned according to the international 10-20 system. EEG data was recorded online to a common reference located at electrode site FCz. Eye movements and blinks were monitored with two electrodes placed below the right eye and at the outer canthus of the left eye. EEG data was sampled at 500 Hz with a bandpass of the hardware filter of 0.1–125 Hz.

Offline EEG data processing was carried out using the EEGLAB (Delorme & Makeig, 2004) and ERPLAB (Lopez-Calderon & Luck, 2014) MATLAB toolboxes. Signals were filtered offline with a bandpass filter of 0.1-30 Hz and re-referenced to the average activity of the two mastoids. Artefacts were corrected by means of an independent component analysis (Extended RunICA, 30 components). ERPs were computed offline for each participant in each condition, time-locked to the onset of the target stimuli presentation, relative to a 100 ms pre-stimulus baseline and until 750 ms post-stimulus onset. Epochs with amplitudes above or below 100μV or with a difference between the maximum and the minimum amplitude exceeding 75μV were considered artefacts and discarded from the analysis. One participant with an excessive number of artefacts (36% of trials) was discarded from the analysis.

Mean amplitudes for seven post-target onset latency windows were submitted to repeated-measures ANOVAs. ERPs analysis were analysed every 100 ms in order to cover early and late components: 50-150 ms, 150-250 ms, 250-350 ms, 350-450 ms, 450-550 ms, 550-650 ms, and 650-750 ms. The factors included in the analysis were: type of distractor (semantically related, form-related, and unrelated), electrode cluster (Anterior Left: F3, FC1; Anterior Right: F4, FC2, Central Left: FC5, C3, CP5; Central Right. FC6, C4, CP6, Centro-Posterior Left: CP1, P3; Centro-Posterior Right: CP2, P4; and Occipital: O1, Oz, O2; see Figure 1) and task (picture-picture and picture-word). Follow-up analyses were corrected using the Bonferroni correction and adjusted p-values are reported.

Fig. 1. Electrode montage used in the present study. Highlighted ROIs were used in the analysis.

3. Results

3.1. Behavioural results

Signing latencies were significantly slower in the picture-picture task than in the picture-word task, β = -110 ms, SE = 22.5, t(42.8) =4.88, p < .001. Compared to the unrelated condition, responses were faster in the form-related condition, β = -35 ms, SE = 9.9, t(29.9) = 3.48, p = .002. This phonological facilitation effect did not change significantly across tasks, β = -1 ms, SE = 22.7, t(30.1) = 0.06, p =.956. In contrast, there was no significant difference between the unrelated and the semantic condition, β = -4 ms, SE = 10.3, t(29.5) =0.36, p = .724 and no significant change of this contrast across tasks, β = 35 ms, SE = 19.2, t(32.7) = 1.85, p = .074.

Error rates did not differ significantly by task, β = -0.20, SE = 0.34, z = 0.58, p = .563. There was no significant difference between the unrelated and the form-related condition, β = -0.08, SE = 0.17, z = 0.49, p = .626, and no significant change of this contrast across tasks, β = 0.11, SE = 0.33, z = 0.32, p = .746. More errors were made in the semantic compared to the unrelated condition, β = 0.44, SE = 0.15, z = 2.83, p = .005. This semantic effect differed significantly across tasks, β = 1.32, SE = 0.31, z = 4.26, p < .001. In the picture-word task the semantic interference effect was significant, β = 1.09, SE = 0.21, z = 5.18, p < .001, whereas there was no significant semantic effect in the picture-picture task, β = -0.22, SE = 0.23, z = 0.98, p = .327. Figure 2 displays sign latencies and error probabilities as estimated in the model fits.

Fig. 2. Mean naming latencies and error probabilities for the picture-word (PWI) and picture-picture (PPI) interference tasks, as estimated in the model fits. Error bars represent the 95% CI. Small shapes and densities represent the individual means for each participant.

To summarize, behaviourally there was a facilitation effect for form-related distractors – which was of similar size across tasks. That is, participants named target pictures faster when distractors (either pictures or words) were related through the Spanish name of the target picture. There was a semantic interference effect in the picture-word interference tasks (on error rates only), whereas there was no semantic effect in the picture-picture interference task.

3.2. Electrophysiological results

Table 2 represents the main effects and interactions throughout the different time windows. Only significant results are discussed in this section.

Table 2. Significance table displaying the p-values on the repeated measures ANOVAs performed at 7 time windows. Significant effects are highlighted in bold with the corresponding F-statistics. Corrected values using the Greenhouse-Geisser correction are reported. TW: Time window (in ms) TD: Type of Distractor, T: Task, EC: Electrode Cluster

In the time window 150-250 ms after the onset of the stimuli presentation, ERP analyses revealed a main effect of type of distractor (F(1.88,43.13) = 3.31, p = 0.05). Post-hoc comparisons showed that there were significant differences between form-related and unrelated distractors (t(23) = 2.4, p = 0.02) and between semantically related and unrelated distractors (t(23) = 2.36, p = 0.03). Both related conditions elicited a larger positivity than the unrelated condition. In this time window, there was also a significant interaction between task and electrode cluster (F(2.14, 49.25) = 4.89, p = 0.01), but post-hoc comparisons did not reveal significant differences.

At 250-350 ms post-onset, there were significant interactions between task and electrode cluster (F(1.90, 43.63) = 10.73, p < 0.001); however, none of the post-hoc comparisons yielded significant results.

At the 350-450 ms time window, the analysis showed a main effect of type of distractor (F(1.93, 44.40) = 3.38, p = 0.04). Post-hoc comparisons showed that both form-related distractors (t(23) = 2.27, p = 0.03) and semantically related distractors (t(23) = 2.4, p = 0.02) elicited more positive-going waves compared to unrelated distractors. In addition, there was a significant interaction between task and electrode cluster (F(2.68, 61.55) = 5.24, p = 0.04). Post-hoc comparisons did not reveal significant differences.

At the 450-550 ms time window, there was a significant interaction between task and electrode cluster (F(2.83, 65.03) = 3.43, p = 0.02). No significant results were obtained in post-hoc comparisons.

To summarize the results reported above, in the early time window 150-250 ms post-onset and in the late time window 350-450 ms post-onset, there was a main effect of distractor type, and this factor did not interact with task or electrode cluster. Post-hoc comparisons revealed that form-related and semantically related distractors elicited more positive-going waves compared to unrelated distractors. Figure 3 depicts the ERP waves for each type of distractor across tasks, in the seven regions of interest.

Fig. 3. Event-related potentials from the semantic (sem), form-related (form) and unrelated (unr) conditions (Y axis: Mean amplitude in μ V) from the stimuli presentation (time 0) to 750 ms. Panel (a) depicts the ERP waves for the picture-word interference task (PWI) and panel (b) depicts the ERP waves for the picture-picture interference task (PPI). Nine regions of interest are represented: anterior left (AL), anterior right (AR), central left (CL), central right (CR), centro-posterior left (CPL), centro-posterior right (CPR) and occipital (O).

Figure 4 depicts the scalp map for semantic and phonological effects across tasks for the two critical time windows where significant differences were observed.

Fig. 4. Topographic maps depicting semantic (sem) and form-related (form) effects for the critical time windows. Effects were computed by subtracting the semantic and phonological distractor ERPs from the unrelated distractor ERPs. Voltage scale in microvolts. Panel (a) represents the picture-word interference task (PWI) and panel (b) represents the picture-picture interference task (PPI).

4. Discussion

In the present study, we explored cross-language, cross-modal interactions in deaf LSC-Spanish bilinguals. Participants named pictures in LSC while ignoring visual distractors in the form of Spanish words (picture-word interference task) or pictures (picture-picture interference task). Distractors were either semantically related to the target picture, form-related to the Spanish name of the target picture (phono-translation), or unrelated. For the semantic contrast (vs. unrelated), we observed no significant semantic effect on sign latencies in either task, but participants made more errors in the semantic condition than in the unrelated condition during the picture-word interference task. Electrophysiologically, semantically related distractors elicited less-negative-going waves than unrelated distractors in the time windows 150-250 ms and 350-450 ms post-stimulus onset. For the form-related contrast (vs. unrelated), we observed phonological facilitation across tasks; pictures presented with form-related distractors in Spanish were signed faster than those pictures presented with unrelated distractors. At the ERP level, this relationship elicited a reduced negativity in the same time windows where semantic effects were reported.

These results provide further evidence of cross-linguistic interactions in deaf bimodal bilinguals, both in language comprehension (Lee et al., 2019; Morford et al., 2011) and language production (Emmorey et al., 2020; Giezen & Emmorey, 2016), and from the weaker L2 oral language onto the more dominant L1 sign language, a pattern seen in other studies within the oral modality (Bobb, Von Holzen, Mayor, Mani & Carreiras, 2020; Holzen & Mani, 2014). This indicates that deaf signers had attained sufficient proficiency in their L2 oral language to experience word influences during sign production (Van Hell & Tanner, 2012). Finally, we obtained very similar results in the picture-word interference task and in the picture-picture interference task. Despite differences between the two tasks in the input format of the distractors (written words vs. pictures), the magnitude of the phonological effect was very similar across these tasks. These results suggest that the oral language knowledge of deaf bilinguals influences sign production even when it is not directly involved in the task.

Regarding the semantic manipulation in our study, we only observed a semantic interference effect on error rates in the picture-word interference task, whereas we did not observe any significant semantic effect on sign latencies. The null result in the picture-picture interference task is in line with previous results in the oral domain. Multiple studies have shown no semantic effects from distractor pictures in picture naming (Damian & Bowers, 2003; Navarrete & Costa, 2005; Roelofs, 2008). Furthermore, there is evidence that semantic interference in this task may only be observed if the task specifically promotes attention to the distractor pictures (Jescheniak, Matushanskaya, Mädebach & Müller, 2014; Matushanskaya, Mädebach, Müller & Jescheniak, 2016), which was not the case in the present study.

For the picture-word interference task, the absence of a clear semantic interference on naming latencies is at odds with previous studies in the oral domain (Costa & Caramazza, 1999; Hermans et al., 1998), but corresponds to other results in sign production of bimodal bilinguals. Similar to the present study, Giezen and Emmorey (2016) had reported a null effect on naming latencies, whereas Emmorey et al. (2020) even observed semantic facilitation. This suggests that semantically related distractor words may induce no semantic conflict in sign production in line with predictions of the response exclusion account. However, we believe that this conclusion may be premature for the following reasons. First, we did observe a semantic interference effect on error rates. This was also the case in the study by Emmorey et al. (2020) and suggests that semantically related distractor words induce some level of conflict. Second, the null effect we observed on latencies is not in itself informative because we predicted either semantic interference (lexical competition) or facilitation (non-competitive selection) to occur. It is possible that the null effect in the present study (and in Giezen & Emmorey, 2016) reflects insufficient power of the design to reliably observe semantic interference effects (Brysbaert & Stevens, 2018; Bürki, Elbuy, Madec & Vasishth, 2020). The results by Emmorey et al. (2020) suggest that the true underlying effect may be one of facilitation. However, we believe that more evidence is needed to evaluate the direction and size of semantic distractor effects in bimodal sign production.

Our ERP results – in particular, the observation of less negative ERPs for semantically related distractors in the N400 time window – correspond to previous results in monolingual and bilingual picture-word interference studies (e.g., Blackford et al., 2012; Dell'Acqua et al., 2010; Roelofs et al., 2016; Rose, Aristei, Melinger & Rahman, 2019; Zhu et al., 2015). In line with previous studies, we interpret this N400 attenuation to indicate semantic priming between the target picture and a related distractor. Some authors have argued that N400-attenuation reflects facilitatory priming of the target by the distractor and is therefore incompatible with lexical competition (Blackford et al., 2012; Emmorey et al., 2020). However, prominent lexical competition accounts assume that semantic context effects reflect a trade-off between facilitatory priming of the target picture by the distractor and interfering “reverse” priming of the distractor by the target picture. Following this argument, our result is compatible with both competitive and non-competitive production accounts (for discussion see Piai et al., 2014; Roelofs et al., 2016). Interestingly, we found an ERP-modulation not only in the N400 time window, but also earlier at around 200 ms. As we discuss in the introduction, earlier ERP-correlates of semantic distractor effects have also been found in some previous studies (Aristei et al., 2011; Dell'Acqua et al., 2010; Hoshino & Thierry, 2011; Rose et al., 2019). Such an early modulation fits with common time-course estimates for lexical access in picture naming. A recent study suggested that early ERP-modulations (at around 200ms) may reflect lexical competition, whereas later modulations (at around 400ms) may reflect ongoing semantic priming between target picture and distractor word (Rose et al., 2019).

In sum, we conclude that neither the behavioural nor the electrophysiological evidence unequivocally support the competitive or the non-competitive account. The behavioural results are inconclusive in this regard. The ERP-results correspond to similar findings in previous mono- and bilingual studies but appear compatible with both accounts. Moving forward, more evidence is needed to determine whether there is indeed semantic facilitation instead of semantic interference in bimodal picture-word interference tasks (as suggested by the result by Emmorey et al., 2020) and to clarify the functional relevance of the ERP-results for the behavioural effects.

Concerning phonological effects, the observation that picture names were signed faster in the presence of distractor words form-related to the oral name of the picture is, to our knowledge, the first piece of evidence showing that phonological properties of the oral language modulate how deaf individuals produce signs (see Lee et al., 2019; Hosemann et al., 2020, for similar results in comprehension). This was further validated by the ERPs showing sensitivity to the phonological manipulation. Form-related distractors elicited a reduced negativity compared to unrelated distractors between 150-250 ms, and 350-450 ms post onset. These modulations replicate phono-translation ERP effects reported in the oral modality (Hoshino & Thierry, 2011) and signed modality (e.g., Hosemann et al., 2020). Remarkably, phonological ERP effects were obtained in the same time windows and with the same direction as those obtained for the semantic contrast, although different results were obtained behaviourally for the two manipulations. This indicates that ERP polarities do not have a direct correspondence with behavioural effects (see also Dell'Acqua et al., 2010). In our study, similar ERP-modulations for the semantic and the phonological contrast might indicate priming between the distractor stimulus and the target picture, while not reflecting the functional consequence of such priming (in terms of facilitation or interference). Note that the polarity and timing (especially the early modulation) of the reported ERPs do not fit with the canonical N400 responses reported in picture-word interference tasks (Chauncey, Holcomb & Grainger, 2009). Acknowledging the differences, we followed N400 interpretations in picture-word interference studies and interpret our data as evidence of the priming effect for form-related distractors relative to unrelated targets (see also Hosemann et al., 2020 for a similar interpretation of N400-like effects in bimodal bilinguals).

Behaviourally, the phonological effect found here differed from previous studies with unimodal bilinguals using the picture-word interference paradigm (Costa et al., 2003; Hermans et al., 1998). The so-called phono-translation interference effect occurs when distractors are phonologically related to the translation of the target language (saying perro, ‘dog’ in English, presented with the distractor doll) and it has been interpreted as evidence of lexical competition in bilingual speech production.

Finding phonological facilitation in bimodal bilinguals and phonological interference in unimodal bilinguals could be reconciled by models which posit phonological interference effects arising at the phonological level (within-language competition models). According to this account, in the absence of phonological overlap between language modalities, the activation of the phonological properties of the oral lexicon could not interfere with the activation of the sign language phonology. Thus, these models would account for phonological facilitation effects in sign production arising at the lexical level. In particular, processing of the distractor, word or picture, would lead to lexical activation of the oral phonological neighbours of the distractor. For the form-related distractors this includes activation of the translation equivalents of the target sign. Activation of the target's translation in the oral language (due to phonological priming by the form-related distractor) may then facilitate target retrieval via automatic translation from oral to sign language. Under this assumption, phonological effects would be a consequence from both languages being activated during the task as a result of parallel activation processes.

As mentioned in the introduction, the obtained pattern of semantic and phonological effects, both behavioural and at the ERP level, could be attributed to the “mouthing” of words (or part-words) that co-occurs with sign articulation in code-blending production (Capek, Waters, Woll, MacSweeney, Brammer, McGuire & Campbell, 2008; Giustolisi, Mereghetti & Cecchetto, 2017; Hosemann et al., 2020; Vinson et al., 2010). In this context, it is conceivable that the articulatory buffer is shared for word-distractors and picture mouthings (i.e., the oral language). In this case, the response exclusion hypothesis may predict semantic interference to result from the same post-lexical conflict as for oral production: that is, due to slower exclusion of semantically-related distractor words (relative to an unrelated ones) from the articulatory buffer. This could delay availability of the mouthed phonemes of the picture, which would also delay sign onset if mouthings are produced in synchrony with the sign. More importantly, the phonological facilitation effect we observed could be explained by phonemes of the distractor overlapping (fully or partly) with those of the mouthed picture name, thus facilitating mouthing production of phonologically-related words. In other words, the phonological facilitation effect may reflect mouthing preparation rather than genuine cross-linguistic influence of the oral language on sign preparation. Following the suggestion of an anonymous reviewer we conducted a follow-up analysis to explore this possibility. As in previous reports (Vinson et al., 2010), we observed that tendencies to produce mouthing widely varied between participants. In the present study, thirteen out of twenty-four participants were mouthing during most of the trials, and six participants did not produce mouthing while signing, indicating that mouthing and the manual components of signs could dissociate and were not obligatory mouth patterns of the signs selected. Further analysis suggested that mouthing was not a critical factor for the behavioural and the ERP results. That is, there were no substantial differences between those participants who were overtly mouthing and those participants who did not produce mouthing (or produced it only in a few trials). Thus, our results do not seem to be caused by mouthing productions while signing.

From the perspective of the parallel activation account it is surprising that the phonological facilitation effect appeared to be virtually identical with distractor pictures and distractor words. For distractor pictures, parallel activation of the phonological cohort should be much less direct, and thus weaker, than for distractor words, because phonological activation is necessarily mediated by visual and conceptual processing of the picture. In line with this argument, unimodal studies have found phonological facilitation to be more robust with distractor words than with distractor pictures (Bloem & La Heij, 2003; Jescheniak et al., 2009). For this reason, we prefer an alternative account under which the phonological effect is not a direct result of immediate co-activation of the L2 but reflects the reorganisation of the L1 as a result of L2 language learning processes (Costa, Pannunzi, Deco & Pickering, 2017, 2019). Within this framework, the present results would be reflecting the reorganisation of the sign language lexicon as a consequence of learning an oral language.

Under this account, lexical signs that were a priori not related in the sign language lexicon (e.g., CEREZA and CEREBRO, cherry and brain in English) would become related as a result of the phonological similarity of their corresponding translations in the oral language. It is possible that phonological properties of the oral language are linked to signs via mouthing production, considered to develop with bilingual experience of deaf individuals with the oral modality.

In this line, although not directly proposed for language production, the reading vocabulary acquisition model for deaf children (Hermans, Knoors, Ormel & Verhoeven, 2008) could account for the learning hypothesis. The model describes how sign and oral languages interact in three developmental stages. In the first stage, deaf children only have access to the form of written words and the meaning is necessarily accessed throughout signs. The repeated co-activation of the sign and the written word translation equivalent results in the semantic and syntactic representations of the signs copied into the lexical representation of the written word. Finally, in the last stage, lexical entries contain all the semantic, morphological, and syntactic information. Considering the learning hypothesis, it is also possible that during the second stage, properties of the oral language are linked to sign forms via orthographic/phonological representations and, consequently, relationships in the oral language would ultimately map onto the sign language lexicon.

Either via reading (orthographic links) or mouthing processes (phonological links), or via the two representations combined, the sign lexicon of a native signer would be restructured when learning an oral language, resulting in a lexical network different from that of a deaf individual without such oral language experience. Thus, when processing a given sign not only the properties of the sign language would be activated, but also properties of the oral language which became linked to the sign. If this is the case, the effects observed in the present study may result from activation flow within the sign language lexicon instead of direct activation flow from the oral language lexicon to the sign lexicon. Note that this account is not arguing against oral language co-activation during sign production itself. The crucial difference is that under this account it is not the immediate co-activation of the oral language that causes the phonological facilitation effect but the reorganization of the sign lexicon following the repeated co-activation of both languages in the process of learning the oral language.

5. Summary and conclusion

The present study tested for cross-linguistic effects from an oral language (L2) on sign language production (L1) by deaf bimodal bilinguals. We found evidence for such cross-linguistic effects, most clearly in form of a phonological facilitation effect of distractors which were form-related to the oral language translation of the target signs. The ERP results suggest a lexical locus of the cross-linguistic interaction between sign and oral languages. The critical phonological effect appeared to be similar with distractor pictures and written distractor words. This suggests that cross-linguistic influences of the oral language are not restricted to task contexts involving oral language stimuli (i.e., distractor words). Most importantly, the present results provide the first piece of evidence that deaf bimodal bilinguals are sensitive to the properties of the oral language in sign production.