Is the digit effect a cognate effect? Digits (still) differ from pictures in non-phonologically mediated language switching

Abstract

Language control in bilinguals is often investigated with the language switching paradigm. Switch costs reflect the ease/difficulty of applying this control mechanism. The type of stimuli employed in the experiments may influence switch costs. To date, only one study has compared digit vs picture processing, reporting reduced switch costs for digits (Declerck, Koch & Philipp, 2012). This result was adjudicated to phonological overlap between the languages used. Crucially, it remains an open question whether this digit effect generalises to language combinations without phonological relation. We fill this gap by investigating language switching with two language pairs differing in relative proficiency (L1 Chinese–L2 English, L1 Chinese–L3 French), where cross-language phonological activation is not expected. Overall, a digit effect is observed in the Chinese–English pair. Contrary to Declerck et al.'s (2012) finding, digits increased switch costs. Phonological mediation cannot explain this effect; instead, we suggest its origin lies in within-language word association links.

Introduction

Language switching is an important paradigm to study language selection and control in bilingual language production. Two types of stimuli are often used, Arabic digits (hereafter digits) and pictures. A central difference between digit naming and picture naming is the potential involvement of semantic processing. It is assumed that picture naming requires semantic activation (e.g., Humphreys, Price & Riddoch, 1999; Sitton, Mozer & Farah, 2000). However, whether semantic information is activated in digits naming is a much-debated issue (e.g., Brysbaert, 1995; Herrera & Macizo, 2011, 2012; Macizo & Alvarez, 2018; Reynvoet, Brysbaert & Fias, 2002). Accordingly, these processing differences may produce different results in language switching experiments when the two types of stimuli are used. In this light, Declerck et al. (2012) specifically examined the influence of stimulus type on language switching between L1 German and L2 English and showed that digits reduced switch costs, but only with cognate stimuli, concluding that the digit effect arises at the phonological level. However, whether their observed digit effect generalizes to language combinations without phonological overlap remains an open question. The present study investigates the influence of stimulus type (digits vs pictures) during bilingual naming in a trilingual population. Crucially, we employ two language pairs (L1 Chinese–L2 English and L1 Chinese–L3 French) where cross-language activation at the phonological level is not expected (i.e., in the absence of full or partial cognates) while examining potential effects of proficiency on the stimulus effect.

Digits, pictures, and their influence on language switching

Participants in cued language switching experiments switch language upon a language cue to name a stimulus. The cognitive processes underlying language selection and control are investigated by comparing participants’ performance (reaction times and error rates) in switch trials relative to repetition trials. Switch costs (i.e., longer reaction times and/or more errors in switch trials) are recurrent in this type of experiment. Nevertheless, research findings are inconsistent about the pattern of switch costs, and both participant- and task-related predictors can influence the results (see Bobb & Wodniecka, 2013 for a review). One such factor is the stimulus type; some studies have used pictures (e.g., Christoffels, Firk & Schiller, 2007; Costa & Santesteban, 2004), whereas others have used digits (e.g., Jackson, Swainson, Cunnington & Jackson, 2001; Meuter & Allport, 1999). When comparing digit and picture naming, a central debate is whether semantic processing is involved when participants have to name digits. Some authors assume that naming Arabic digits requires access to the number concept (i.e., a compressed number line on which integers are sequentially ordered), similar to picture naming accessing the object concept (e.g., Brysbaert, 1995; Damian, 2004; Fias, 2001; Fias, Reynvoet & Brysbaert, 2001; Reynvoet et al., 2002). In contrast, other authors hold the opinion that naming these symbols takes place via an asemantic route (i.e., naming can take place through a direct route between Arabic digits and their lexical form, bypassing semantics; e.g., Dehaene, 1992; Roelofs, 2006).

In a seminal study, Brysbaert (1995) observed a numerical distance effect in number priming by which target numbers preceded by primes with a close magnitude were named faster in a priming experiment. This effect was argued to arise due to the co-activation of other closely related numbers – the prime automatically accessed the abstract number line, and the activation spread from one semantic representation to nearby magnitudes. Thus, the numerical distance effect is generally interpreted as evidence for semantic involvement in digit processing. Posterior studies confirmed this finding, providing more evidence about the effect being semantic in nature (e.g., Duyck & Brysbaert, 2002; Koechlin, Naccache, Block & Dehaene, 1999).

On the other hand, Herrera and Macizo (2010, 2011, 2012) have provided evidence for an asemantic route using the semantic blocking paradigm. This paradigm is often employed to examine processing differences between picture and word naming (e.g., Damian, Vigliocco & Levelt, 2001; Kroll & Stewart, 1994). Results usually show that picture naming latencies are longer when pictures are grouped by category in a blocked condition relative to a mixed condition, where items belong to different semantic categories. Crucially, this interference effect disappears when the items to be named are words. These contrasting results are assumed to arise from processing differences during picture and word naming. In picture naming, semantic activation occurs first. Then, semantically related items are co-activated, causing interference at the lemma level. In word naming, on the other hand, direct access to the word form is achieved, resulting in no semantic interference.

Using this paradigm, Herrera and Macizo compared the naming latencies of numerals and pictures in mixed and blocked conditions. Their results showed the expected longer latencies in the blocked conditions for naming pictures. However, a facilitatory effect was observed in number naming (both number words and Arabic digits). Based on these findings, the authors suggested that, as it occurs with words, digit naming is achieved through direct access to the word level too (i.e., an asemantic route). Interestingly, Herrera and Macizo (2011, 2012) also observed that the facilitatory effect during Arabic numbers’ naming in blocked conditions was more pronounced for digits numerically closer to the previous stimuli (i.e., a numerical distance effect). Nevertheless, these authors held the opinion that this distance effect does not necessarily involve semantics. Instead, they placed the locus of the effect at the word level, as numbers often co-occur and are usually learned in a sequence. Therefore, numbers would share strong associative connections in the lexicon, and the numerical distance effect can be attributed to associative links between numbers separated by a small distance.

Yet, a recent study by Macizo and Alvarez (2018) offered a more nuanced (and complex) picture of digit naming. In a picture/digit naming experiment, the authors found a posterior N400 modulation, a marker of semantic access, in addition to the previously reported lack of interference effect in blocked conditions when naming digits – replicating Herrera and Macizo's (2011, 2012) findings. Thus, Macizo and Alvarez (2018) left the question of the influence of semantics in naming Arabic digits unresolved.

In language switching, only Declerck et al. (2012) has directly compared the effects of employing digits and pictures on switch costs. They used L1 German–L2 English stimuli – a digit set (1-9) and a standard picture set (9 pictures included). Critically, the numbers they employed contained cognates. Therefore, the authors added a control stimulus set of pictures with cognate names and a control stimulus set consisting of semantically-related pictures to match the characteristics of cognate digit stimuli. Overall, Declerck and colleagues found that participants named digits faster than pictures. Further, switch costs were smaller during digit naming. Crucially, the difference in switch costs between digits and pictures disappeared when digits were compared to the cognate control pictures. This finding led the authors to conclude that the main difference between digit and picture naming in their language switching experiment did not emerge from the semantic level but the phonological one. That is, the digit effect was, in fact, a cognate effect. In this light, naming cognate digits would lead to phonological activation of the non-target language, facilitating performance at switch trials. In contrast, this would cause interference in repetition trials. Thus, combining these two effects would result in cognate digits reducing switch costs.

Importantly, Declerck et al. used another piece of evidence to support their conclusion about digit and picture naming not differing at the semantic level: a numerical distance effect in switch costs. Their data indicated that closely related digits (i.e., with a numerical distance of three or less) increased switch costs compared to more distant digits. The finding led the authors to suggest that differential semantic involvement cannot be adjudicated as the cause of the digit effect. In fact, had numerical distance been maximal across the digit stimuli, the digit effect would have been even more prominent.

All in all, the findings in Declerck et al. (2012) suggest that digit and picture naming share processing mechanisms (i.e., semantic involvement). Following this conclusion, no difference in switch costs should be found when digits from two languages are non-cognates. Notably, this hypothesis has not been investigated yet – being central to the present experimental design.

At this point, we should note that in recent localist-connectionist bilingual processing models like Multilink (Dijkstra, Wahl, Buytenhuijs, Halem, Al-Jibouri, De Korte & Rekké, 2019), there is no qualitative difference in how lexical items are connected to semantics in different languages. Semantic representations send activation to their corresponding phonological representations cross-linguistically. However, as noted by Dijkstra and colleagues, the strength of the connections between semantics and phonology may be proficiency-dependent (Dijkstra et al., 2019, p. 5). Therefore, we can expect the numerical distance effect on switch costs to differ as a function of relative proficiency. This potential interaction between proficiency and the numerical distance effect would further support a semantic account for digit processing. Nevertheless, this possibility was not further pursued in Declerck et al. (2012), a gap we fill in the present investigation.

The present study addresses two unresolved issues that can shed light on how stimulus type influences language switching. First, if the digit and picture naming asymmetries observed in Declerck et al. (2012) originated at the phonological level, we should expect no differences in switch costs between digits and pictures in our two phonologically non-overlapping language pairs, where cognates are absent. However, observing differential switch costs for digits and pictures would indicate that other representational levels (and not only the phonological one) are involved during language selection. In addition to this, the present study manipulates the relative proficiency of the language pair (L1 Chinese–L2 English vs L1 Chinese–L3 French). In this way, we addressed whether (and how) proficiency modulates the digit numerical distance effect. If this effect arises from semantic co-activation among closely related digits and if the strength of the activation depends on proficiency, we should expect this effect to differ in the two language pairs.

Method

Participants

Thirty native Chinese university students participated (27 female, mean age = 19). They were English graduate students with French as their second foreign language (i.e., third language). According to self-reports, they started learning English at around the age of 10, learning the language for an average of 12.3 years. Also, they started learning French in the university, having done so at the time of the experiment for an average of 1.5 years. Although they were exposed to English in the university daily, Chinese was their main language of communication. Participants reported an average of 4 hours of classroom learning of French plus 3-4 hours of self-study each week. We used the participants’ lexical robustness as an index of their global language proficiency (Costa, Santesteban & Ivanova, 2006; Schwieter & Sunderman, 2008, 2009). Lexical robustness was measured with the Multilingual Naming Test (MINT; Gollan, Weissberger, Runnqvist, Montoya & Cera, 2012), first in Chinese, then in English, and lastly in French. The MINT is a standardised naming test where participants name 68 pictures of varying frequency in the languages of interest. Although the MINT was not designed to measure proficiency in French, it has been validated as a reliable measure for capturing variance in bilinguals’ language proficiency and lexical robustness for English, Spanish, and Mandarin (Gollan et al., 2012; Ivanova, Salmon & Gollan, 2013; Sheng, Lu & Gollan, 2014). To use the MINT to measure participants’ French lexical robustness, we removed six English–French cognates from the original list, which ended up containing 62 items (instead of 68).

One participant was excluded due to low accuracy in the English MINT (16%). Among the remaining participants, the average MINT scores were 89.50% for Chinese (SD = 3.09%, range = 80.88%-97.05%), 59.48% for English (SD = 8.85%, range = 41.17%-77.94%), and 17.30% for French (SD = 4.61%, range = 9.67%-25.80%). Their MINT scores in the three languages significantly differed from each other (Chinese MINT vs English MINT, t = 20.61, p <0.001; Chinese MINT vs French MINT, t = 77.93, p < 0.001 English MINT vs French MINT, t = 32.87, p <0.001). These results evidence that the current participants were clearly L1 dominant and, importantly for the present study, reasonably proficient in English but low proficient in French.

Materials and task

There were two types of stimuli, digits from 1-9 and 9 pictures. The participants were required to name the stimuli in different languages, indicated by a colour cue (red for Chinese, blue for English, and purple for French). The digits were presented in Arabic integer form. The pictures were obtained from Snodgrass and Vanderwart (1980). Prior to the experiment, five English graduate students with French as their third language were asked to mark the pictures – the original stimuli contained 260 pictures – they could name in Chinese, English and French without prompt. In total, they were able to name 22 pictures in the three languages without hesitation or self-correction. The 22 pictures were further sent to 10 different students. They were asked to rate their familiarity with each picture on a 7-point Likert scale. The top 9 pictures with the highest mean ratings were selected as the picture stimuli. See Appendix A for the list of picture stimuli and Appendix C for the digit stimuli. The familiarity rating of each picture in each language can be found in Appendix B. As noted, the words describing the pictures on the list did not include cognates. These words were matched for frequency and number of syllables. We also approximated the physical size of the pictures (289 × 289 pixels) so that they appeared as close to the digits (presented in font size 48) as possible. The experiment was run with E-prime 3.0. The speech onset of the vocal responses was recorded with Chronos, connected to E-prime. The responses were recorded so that the noted errors could be checked to ensure data accuracy.

Procedure

The experiments were carried out in two sessions conducted on consecutive days. In each session, participants named items in the two languages of one of the two pairs. Language pair was counterbalanced across sessions and participants. Prior to the experiment, the subjects were allowed to get familiar with the stimuli (both digits and pictures) for as long as they desired. Before the experiment started, a brief explanation of the task was given orally and was also available in written form. The oral instruction was given in Chinese to ensure the participants understood the experiment procedure clearly, but the written form was available in all three languages. Participants were asked to name the items as quickly and accurately as possible. Each stimulus was preceded by a colour cue. Cue cards were printed and displayed next to the laptop so that they were visible throughout the experiment. A fixation point (+) was presented for 500 ms, followed by the colour cue for 100 ms. After the cue, the stimulus was shown, staying on screen until a response was registered or for a maximum of 3000 ms. There was a 500 ms inter-trial interval. In each session, there were four blocks of digit naming and four blocks of picture naming. Each block consisted of 36 trials. Before being presented with the critical trials with each stimulus type, the subjects were shown 18 practice trials (using the same stimuli as used in the experiment, and each item being practiced twice). After the practice, they had a break and a chance to ask the experimenter questions on the task. In each block, each stimulus was presented four times, twice in each language. The trials were pseudo-randomized, restricting presentations to an equal number of switches and repetitions and no immediate repetitions of stimuli across trials. The order of digit versus picture blocks was counterbalanced across participants. After each block, the participants could have a short break.

Results

Data and analysis code can be found in the second author's OSF repository (https://doi.org/10.17605/OSF.IO/X3AT9). See Table 1 for a summary of the results. One participant was excluded from the analysis due to low accuracy in the English MINT (16%). Thus, the final analyses included data from 29 participants. Response times and error rates were analysed. Response latencies below 200 ms and above 5000 ms were removed (Baayen & Milin, 2010). Following Lo and Andrews (2015), raw response times were used, and Generalized Linear Mixed-Effects Models (Baayen, Davidson & Bates, 2008) were specified with the inverse gaussian family and identity function with the lme4 package (Bates, Mächler, Bolker & Walker, 2015) in R (R Core Team, 2019). For the accuracy analysis, the models employed the binomial family. Categorical independent variables were sum contrasted, whereas numerical independent variables were scaled, centred, and converted to z units.

Table 1. Mean response times (RTs, in milliseconds; standard errors), error rates (%), and switch costs (in milliseconds).

*p < .001

In both analyses, main effects and interactions of interest were included as fixed effects (Brauer & Curtin, 2018). The random structure consisted of all main effects and interactions varying within subjects. Further, we specified Complex Random Intercepts (CRI), following Scandola and Tidoni (2021). This method, which also employs maximal random structures (e.g., Barr, Levy, Scheepers & Tily, 2013; Brauer & Curtin, 2018; Scandola & Tidoni, 2021), aims at finding an optimal trade-off between Type-I and II errors while minimising conversion and overfitting issues. In a CRI model, random slopes are replaced by random intercepts for each grouping factor. All analyses started with a maximal random-effects structure specification. In the case of non-convergence, the CRI that explained the least variance was removed. This procedure continued until a final, convergent model was found. In the response latencies analysis, final models were further criticised (e.g., checking model assumptions), and standardised residuals above 2.5 SD were removed (Baayen & Milin, 2010). See Appendix D for the maximal and final models for RTs and error rates analyses.

Data from the two experiments were analysed together. A maximal model contained fixed effects for Language pair (L1 Chinese–L2 English, L1 Chinese–L3 French), Stimulus (digits vs pictures), and Trial (switch vs repeat), and their interactions. Their CRI random-effects structures contained random intercepts for the same factors and their interactions by subject.

Response time analysis

See Appendix E for the final model's outcome. The final model revealed significant main effects of Language pair (β = −68.61; t = −8.89; p < .001), Stimulus (β = −133.87; t = −22.69; p < .001), and Trial (β = −46.04; t = −9.80; p < .001). The effects indicated overall faster responses in the Chinese–English data, and to digits and repeat trials. Further, Language pair and Stimulus interacted significantly (β = −36.81; t = −5.15; p < .001), showing that the effect of type of stimuli was larger in the Chinese–English dataset. Lastly, a three-way interaction between Language pair, Stimulus, and Trial (β = −21.44; t = −3.78; p < .001) revealed significantly larger switch costs for digits than pictures, but only with the Chinese–English data (38 ms for digits vs 51 ms for pictures).

In order to explore the impact of numerical distance on switch costs, we run a second model where the factor Stimulus was replaced by Numerical distance treated categorically (coded as small – less than or equal to three numbers of difference between two trials – or large – more than three numbers of difference). In this model, we observed significant main effects of Language pair (β = −85.61; t = −6.63; p < .001), Trial (β = −44.03; t = −9.14; p < .001), and Numerical distance (β = −16.81; t = −2.93; p < .01). These effects indicated that RTs were faster with the Chinese–English stimuli, repeat trials, and large numerical distance. A significant interaction between Language pair and Numerical distance was obtained (β = 30.87; t = 2.20; p < .05), indicating that the Numerical distance effect is larger with the Chinese–French stimuli. Importantly, Numerical distance did not significantly interact with Trial, suggesting that the factor does not explain the stimulus type effect observed in the main, original model.

Accuracy analysis

See Appendix F for the final model's outcome. Accuracy was dummy-coded as 1 (correct) or 0 (incorrect). The main effects of Language pair (β = -0.43; t = -3.43; p < .001), and Trial (β = 0.23; t = 2.75; p < .001) were significant. These effects indicated lower error rates with responses in the Chinese–French stimuli and to repeat trials. Further, the interaction between Language pair and Stimulus was significant (β = 0.45; t = 2.58; p < .001), showing that participants in the Chinese–French dataset were more accurate when responding to pictures. Finally, the interaction between Language pair and Trial also reached significance (β = 0.33; t = 2.00; p < .05), indicating increased switch costs in the Chinese–English pair. This finding replicates the one observed in the analysis of response times. In the model replacing Stimulus with Numerical distance, only Trial (i.e., switch cost) yielded a significant effect (β = 0.31; t = 2.63; p < .01).

Discussion

This paper investigated how stimulus type influences language switching in bilingual naming between phonologically distinct languages. More specifically, we examined whether the digit effect on language switching described in Declerck et al. (2012) can still be found when non-cognate stimuli are used and whether digits and pictures are indeed processed similarly. We also explored how the numerical distance between digits interacted with proficiency to further assess the role of semantics in digit processing. We used two language pairs differing in relative proficiency (L1 Chinese–L2 English, L1 Chinese–L3 French) in a within-participant design to answer these questions. Overall, stimulus type influenced switch costs in the Chinese–English pair but not in the Chinese–French data. Switch costs were larger with digits than with pictures when participants switched between Chinese and English. This finding contrasts with the reduced switch costs for digits reported in Declerck et al. (2012). In addition, we found no numerical distance effect on switch costs, being this result irrespective of language pair.

Does phonological and semantic involvement explain the present results?

Declerck et al. (2012) suggested that the digit effect in language switching is a cognate effect. Their conclusion was based on the absence of a difference between digit naming and cognate-matched picture naming and the maintained reduction in switch costs with digits after manipulating the semantics of the picture sets. If this argument is on the right track, we would expect no switch costs asymmetries between digits and pictures with non-cognate stimuli. However, our data did not support this prediction. Finding larger switch costs for digits in the Chinese–English set indicated that the difference in stimulus processing was not modulated by phonology. Thus, the origin of the effect within the present non-cognate stimuli calls for an explanation.

Arguably, processing at the semantic level can be considered as the potential cause of the digit effect. The reason is that digit stimuli might be semantically closer than picture stimuli (e.g., Brysbaert, 1995; Declerck et al., 2012). Nevertheless, we find such a tentative explanation challenging to reconcile with recent connectionist bilingual processing models (e.g., Multilink). Assuming, as the models do, that semantic activation occurs in parallel across languages, activating a particular representation stimulates related cross-language lexical representations too. For instance, the semantic representation of “3” activates its lexical representation in both Chinese (sān) and English (“three”). Likewise, the semantic representations of “2” and “4” would also be largely excited due to increased semantic proximity, sending activation to their cross-language lexical representations. If so, switching between languages when naming these numbers should become easier, as their representations would have been already stimulated at the pre-switch trial. By the same token, naming in repetition trials would be more difficult due to increased competition, and a reduction in switch costs would be expected. Note that this mechanism resembles that argued by Declerck et al. (2012) to account for the reduction of switch costs with cognate digits – in their study, phonological co-activation made repeat trials more difficult but switch trials easier. Notably, the present data did not show the expected decrease in switch costs with digits if processing occurs at the semantic level. Thus, in light of the present results, we conclude that the digit effect does not lie in a more considerable degree of semantic co-activation during digit naming.

This conclusion is further confirmed by the absence of a numerical distance effect in our data. Recall that this effect has been held as a crucial piece of evidence in favour of the semantic processing account for digits (Brysbaert, 1995; Declerck et al., 2012; Duyck & Brysbaert, 2002; Koechlin et al., 1999). This is because digits of closer magnitude would be more semantically similar to each other than farther away digits (Mandera, Keuleers & Brysbaert, 2017). Thus, if semantic co-activation accounts for the digit effect, the effect should be more pronounced with small distance numbers or even driven by them. But, yet again, this was not evidenced in our data, and numerical distance did not influence switch costs.

The asemantic route in digit naming

At this point, and after having considered the a priori two most probable explanations of the digit effect, we should contemplate the possibility of digit naming occurring via an asemantic route, as suggested by Herrera and Macizo (2011, 2012). In their view, supported by their results in semantic blocking experiments, the semantic level is not activated during digit naming. However, more importantly, numbers in close distance (i.e., three or less) establish associative, word-level associations due to these closely related numbers co-occurring more often. As such, during digit naming, naming one digit also activates the closely related digits at the word level – with no semantic involvement. However, following this hypothesis, a question emerges: How do number associations at the word level explain the present increased switch costs with digits? A conjectural explanation can only work under two assumptions. First, in light of the absence of a numerical distance effect, word associations would have been established between all the numbers within the ten (not just between numbers with a distance of three or less, as suggested by Herrera and Macizo). Second, word associations between these numbers only occur within the same language and not cross-linguistically.

Crucially, studies on bilingual number knowledge suggest that mathematical knowledge is linguistically coded in one language, with mathematical computation taking place primarily in one language too (Dehaene, 1997; Dehaene, Spelke, Pinel, Stanescu & Tsivkin, 1999). That is, bilingual speakers do not overpass language boundaries when performing mathematical tasks. In addition, Wagner, Kimura, Cheung and Barner (2015) showed that bilingual children acquire knowledge of number words independently in each language early in life. This acquisition is contingent on their exposure to these words in each particular language. More importantly for the present argument, this knowledge does not transfer easily across languages.

Going back to our results, under this word association account of digit naming, the observed larger switch costs with digits would be explained by an increased difficulty to switch away from a language because the number associates from the same language, and only from that same language, are strongly stimulated. Activating these associates enables a faster selection of these number words at repetition trials, but, crucially, it also causes more interference at switch trials when translation equivalents are required. Moreover, observing the digit effect only in the Chinese–English dataset suggests that word associations were established for both L1 Chinese and L2 English, but not for L3 French. Thus, the low proficiency of our participants in French would have prevented them from establishing such connections.¹

To summarize, our findings challenge the results in Declerck et al. (2012) as we obtain a digit effect in a switching paradigm employing phonologically distinct languages. This result necessarily implies that a phonological account is not enough to explain the digit effect in these experiments. Further, we argue that the present results do not find straightforward accommodation under an account resorting to differences emerging at the semantic level. Instead, we conclude that our data point towards word-level associations for digit naming. Strong word associations established among within-language numbers and dependent on proficiency can explain the present digit effect by making switching more effortful. Whether this account is on the right track should be further investigated in future research. For instance, studies may consider (1) manipulating the potential degree of association in both digit and picture stimuli; (2) employing different language combinations; and (3) examining different bilingual populations – where experience with the critical stimuli varies in quantity and quality. All this while trying to increase sample sizes (Brysbaert, 2021) despite the inherent difficulties of recruiting participants in bilingualism research. We consider that our understanding of bilingual switching and the functioning of the bilingual lexicon would considerably benefit from such a pursuit.