Hostname: page-component-cd9895bd7-jn8rn Total loading time: 0 Render date: 2024-12-18T14:01:00.500Z Has data issue: false hasContentIssue false
  • English
  • Français

Enhanced temporal binding of audiovisual information in the bilingual brain

Published online by Cambridge University Press:  05 July 2018

GAVIN M. BIDELMAN Open the ORCID record for GAVIN M. BIDELMAN [Opens in a new window]  and
SHELLEY T. HEATH
GAVIN M. BIDELMAN*
Affiliation:
School of Communication Sciences & Disorders, University of Memphis, Memphis, TN, USA Institute for Intelligent Systems, University of Memphis, Memphis, TN, USA University of Tennessee Health Sciences Center, Department of Anatomy and Neurobiology, Memphis, TN, USA
SHELLEY T. HEATH
Affiliation:
School of Communication Sciences & Disorders, University of Memphis, Memphis, TN, USA
*
Address for correspondence: Gavin M. Bidelman, PhD, School of Communication Sciences & Disorders, University of Memphis, 4055 North Park Loop, Memphis, TN, 38152[email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

We asked whether bilinguals’ benefits reach beyond the auditory modality to benefit multisensory processing. We measured audiovisual integration of auditory and visual cues in monolinguals and bilinguals via the double-flash illusion where the presentation of multiple auditory stimuli concurrent with a single visual flash induces an illusory perception of multiple flashes. We varied stimulus onset asynchrony (SOA) between auditory and visual cues to measure the “temporal binding window” where listeners fuse a single percept. Bilinguals showed faster responses and were less susceptible to the double-flash illusion than monolinguals. Moreover, monolinguals showed poorer sensitivity in AV processing compared to bilinguals. The width of bilinguals’ AV temporal integration window was narrower than monolinguals’ for both leading and lagging SOAs (Biling.: -65–112 ms; Mono.: -193 – 112 ms). Our results suggest the plasticity afforded by speaking multiple languages enhances multisensory integration and audiovisual binding in the bilingual brain.

Keywords

audiovisual integrationbilingualismexperience-dependent plasticitymultisensory facilitationtemporal binding window
Type
Research Article
Information
Bilingualism: Language and Cognition , Volume 22 , Issue 4 , August 2019 , pp. 752 - 762
Copyright
Copyright © Cambridge University Press 2018 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

* The authors thank Haley Sanders for assistance in data collection and Kelsey Mankel for comments on earlier versions of this manuscript.

Supplementary material can be found online at https://doi.org/10.1017/S1366728918000408

References

International Telecommunications Union (ITU). (1998). Relative timing of sound and vision for broadcasting (pp. 1–5). Technical Report, Geneva, Switzerland.Google Scholar
ATSC. (2003). Relative Timing of Sound and Vision for Broadcast Operations. In Implementation Subcommittee Finding: Doc. IS-191, 26 June, 2003.Google Scholar
Banks, B., Gowen, E., Munro, K. J., & Adank, P. (2015). Audiovisual cues benefit recognition of accented speech in noise but not perceptual adaptation. Frontiers in Human Neuroscience, 9, 422. doi:10.3389/fnhum.2015.00422Google Scholar
Bernstein, L. E., Auer, E. T. Jr, & Takayanagi, S. (2004). Auditory speech detection in noise enhanced by lipreading. Speech Communication, 44 (1–4), 5–18. doi:https://doi.org/10.1016/j.specom.2004.10.011Google Scholar
Bialystok, E. (2009). Bilingualism: The good, the bad, and the indifferent. Bilingualism: Language and Cognition, 12 (1), 3–11.Google Scholar
Bialystok, E., Craik, F. I., & Freedman, M. (2007). Bilingualism as a protection against the onset of symptoms of dementia. Neuropsychologia, 45 (2), 459–464. doi:10.1016/j.neuropsychologia.2006.10.009Google Scholar
Bialystok, E., & DePape, A. M. (2009). Musical expertise, bilingualism, and executive functioning. Journal of Experimental Psychology: Human Perception and Performance, 35 (2), 565–574. doi:10.1037/a0012735Google Scholar
Bialystok, E., Majumder, S., & Martin, M. M. (2003). Developing phonological awareness: Is there a bilingual advantage? Applied Psycholinguistics, 24 (01), 27–44. doi:10.1017/S014271640300002XGoogle Scholar
Bidelman, G. M. (2016). Musicians have enhanced audiovisual multisensory binding: Experience-dependent effects in the double-flash illusion. Experimental Brain Research, 234 (10), 3037–3047.Google Scholar
Bidelman, G. M., & Dexter, L. (2015). Bilinguals at the “cocktail party”: Dissociable neural activity in auditory-linguistic brain regions reveals neurobiological basis for nonnative listeners' speech-in-noise recognition deficits. Brain and Language, 143, 32–41.Google Scholar
Bidelman, G. M., Gandour, J. T., & Krishnan, A. (2011). Cross-domain effects of music and language experience on the representation of pitch in the human auditory brainstem. Journal of Cognitive Neuroscience, 23 (2), 425–434. doi:10.1162/jocn.2009.21362Google Scholar
Bidelman, G. M., Hutka, S., & Moreno, S. (2013). Tone language speakers and musicians share enhanced perceptual and cognitive abilities for musical pitch: Evidence for bidirectionality between the domains of language and music. PLoS One, 8 (4), e60676. doi:10.1371/journal.pone.0060676Google Scholar
Burfin, S., Pascalis, O., Ruiz Tada, E., Costa, A., Savariaux, C., & Kandel, S. (2014). Bilingualism affects audiovisual phoneme identification. Frontiers in Psychology, 5, 1179. doi:10.3389/fpsyg.2014.01179Google Scholar
Cecere, R., Gross, J., & Thut, G. (2016). Behavioural evidence for separate mechanisms of audiovisual temporal binding as a function of leading sensory modality. European Journal of Neuroscience, 43 (12), 1561–1568. doi:10.1111/ejn.13242Google Scholar
Cecere, R., Rees, G., & Romei, V. (2015). Individual differences in alpha frequency drive crossmodal illusory perception. Current Biology, 25 (2), 231–235. doi:https://doi.org/10.1016/j.cub.2014.11.034Google Scholar
Erber, N. P. (1975). Auditory-visual perception of speech. Journal of Speech and Hearing Disorders, 40 (4), 481–492.Google Scholar
Erickson, L. C., Zielinski, B. A., Zielinski, J. E. V., Liu, Turkeltaub, P. E., Leaver, A. M., & Rauschecker, J. P. (2014). Distinct cortical locations for integration of audiovisual speech and the McGurk effect. Frontiers in Psychology, 5 (534). doi:10.3389/fpsyg.2014.00534Google Scholar
Foss-Feig, J. H., Kwakye, L. D., Cascio, C. J., Burnette, C. P., Kadivar, H., Stone, W. L., & Wallace, M. T. (2010). An extended multisensory temporal binding window in autism spectrum disorders. Experimental Brain Research, 203 (2), 381–389.Google Scholar
Hervais-Adelman, A., Pefkou, M., & Golestani, N. (2014). Bilingual speech-in-noise: Neural bases of semantic context use in the native language. Brain and Language, 132, 1–6.Google Scholar
Innes-Brown, H., & Crewther, D. (2009). The impact of spatial incongruence on an auditory-visual illusion. PLoS One, 4 (7), e6450. doi:10.1371/journal.pone.0006450Google Scholar
Kaganovich, N., Schumaker, J., Leonard, L. B., Gustafson, D., & Macias, D. (2014). Children with a history of SLI show reduced sensitivity to audiovisual temporal asynchrony: An ERP study. Journal of Speech, Language, and Hearing Research, 57 (4), 1480–1502. doi:10.1044/2014_JSLHR-L-13-0192Google Scholar
Kaposvari, P., Csete, G., Bognar, A., Csibri, P., Toth, E., Szabo, N., Vecsei, L., Sary, G., & Kincses, Z. T. (2015). Audio-visual integration through the parallel visual pathways. Brain Research, 1624, 71–77. doi:10.1016/j.brainres.2015.06.036Google Scholar
Krizman, J., Bradlow, A. R., Lam, S. S.-Y., & Kraus, N. (2016). How bilinguals listen in noise: linguistic and non-linguistic factors. Bilingualism: Language and Cognition, 1–10. doi:10.1017/S1366728916000444Google Scholar
Krizman, J., Skoe, E., Marian, V., & Kraus, N. (2014). Bilingualism increases neural response consistency and attentional control: evidence for sensory and cognitive coupling. Brain and Language, 128 (1), 34–40. doi:10.1016/j.bandl.2013.11.006Google Scholar
Kuhl, P. K., Ramírez, R. R., Bosseler, A., LinJ., -F. L. J., -F. L., & Imada, T. (2014). Infants’ brain responses to speech suggest analysis by synthesis. Proceedings of the National Academy of Sciences of the United States of America, doi:10.1073/pnas.1410963111. doi:10.1073/pnas.1410963111Google Scholar
Kuhl, P. K., Williams, K. A., Lacerda, F., Stevens, K. N., & Lindblom, B. (1992). Linguistic experience alters phonetic perception in infants by 6 months of age. Science, 255 (5044), 606–608.Google Scholar
Lee, H. L., & Noppeney, U. (2011). Long-term music training tunes how the brain temporally binds signals from multiple senses. Proceedings of the National Academy of Sciences of the United States of America, 108 (51), E1441–1450. doi:10.1073/pnas.1115267108Google Scholar
Lee, H. L., & Noppeney, U. (2014). Music expertise shapes audiovisual temporal integration windows for speech, sinewave speech and music. Frontiers in Psychology, 5 (868), 1–9. doi:10.3389/fpsyg.2014.00868Google Scholar
Li, P., Sepanski, S., & Zhao, X. (2006). Language history questionnaire: A web-based interface for bilingual research. Behavioral Research Methods, 38 (2), 202–210.Google Scholar
Macmillan, N. A., & Creelman, C. D. (2005). Detection theory: A user's guide (2nd ed.). Mahwah, N.J.: Lawrence Erlbaum Associates, Inc.Google Scholar
Man, K., Kaplan, J. T., Damasio, A., & Meyer, K. (2012). Sight and sound converge to form modality-invariant representations in temporoparietal cortex. Journal of Neuroscience, 32 (47), 16629–16636. doi:10.1523/jneurosci.2342-12.2012Google Scholar
Mishra, J., & Gazzaley, A. (2012). Attention distributed across sensory modalities enhances perceptual performance. Journal of Neuroscience, 32 (35), 12294–12302. doi: https://doi.org/10.1523/JNEUROSCI.0867-12.2012Google Scholar
Mishra, J., Martinez, A., & Hillyard, S. A. (2008). Cortical processes underlying sound-induced flash fusion. Brain Research, 1242, 102–115. doi:10.1016/j.brainres.2008.05.023Google Scholar
Mishra, J., Martinez, A., Sejnowski, T. J., & Hillyard, S. A. (2007). Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. Journal of Neuroscience, 27, 4120–4131.Google Scholar
Musacchia, G., Sams, M., Skoe, E., & Kraus, N. (2007). Musicians have enhanced subcortical auditory and audiovisual processing of speech and music. Proceedings of the National Academy of Sciences of the United States of America, 104 (40), 15894–15898. doi: 10.1073/pnas.0701498104Google Scholar
Navarra, J., & Soto-Faraco, S. (2007). Hearing lips in a second language: visual articulatory information enables the perception of second language sounds. Psychological Research, 71 (1), 4–12. doi:10.1007/s00426-005-0031-5Google Scholar
Neufeld, J., Sinke, C., Zedler, M., Emrich, H. M., & Szycik, G. R. (2012). Reduced audio–visual integration in synaesthetes indicated by the double-flash illusion. Brain Research, 1473, 78–86. doi:https://doi.org/10.1016/j.brainres.2012.07.011Google Scholar
Pons, F., Bosch, L., & Lewkowicz, D. J. (2015). Bilingualism modulates infants' selective attention to the mouth of a talking face. Psychol Sci, 26 (4), 490–498. doi:10.1177/0956797614568320Google Scholar
Powers, A. R., Hillock, A. R., & Wallace, M. T. (2009). Perceptual training narrows the temporal window of multisensory binding. Journal of Neuroscience, 29, 12265–12274.Google Scholar
Raij, T., Ahveninen, J., Lin, F.-H., Witzel, T., Jääskeläinen, I. P., Letham, B., Israeli, E., Sahyoun, C., Vasios, C., Stufflebeam, S., Hämäläinen, M., & Belliveau, J. W. (2010). Onset timing of cross-sensory activations and multisensory interactions in auditory and visual sensory cortices. European Journal of Neuroscience, 31 (10), 1772–1782. doi:10.1111/j.1460-9568.2010.07213.xGoogle Scholar
Reetzke, R., Lam, B. P. W., Xie, Z., Sheng, L., & Chandrasekaran, B. (2016). Effect of simultaneous bilingualism on speech intelligibility across different masker types, modalities, and signal-to-noise ratios in school-age children. PLoS One, 11 (12), e0168048. doi:10.1371/journal.pone.0168048Google Scholar
Ressel, V., Pallier, C., Ventura-Campos, N., Diaz, B., Roessler, A., Avila, C., & Sebastian-Gallés, N. (2012). An effect of bilingualism on the auditory cortex. Journal of Neuroscience, 32 (47), 16597–16601. doi: 10.1523/JNEUROSCI.1996-12.2012Google Scholar
Rogers, C. L., Lister, J. J., Febo, D. M., Besing, J. M., & Abrams, H. B. (2006). Effects of bilingualism, noise, and reverberation on speech perception by listeners with normal hearing. Applied Psycholinguistics, 27 (03), 465–485. doi:10.1017/S014271640606036XGoogle Scholar
Ronquest, R. E., Levi, S. V., & Pisoni, D. B. (2010). Language identification from visual-only speech signals. Attention, perception & psychophysics, 72 (6), 1601–1613. doi:10.3758/APP.72.6.1601Google Scholar
Rosenthal, O., Shimojo, S., & Shams, L. (2009). Sound-induced flash illusion is resistant to feedback training. Brain Topography, 21 (3-4), 185–192. doi:10.1007/s10548-009-0090-9Google Scholar
Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C., & Foxe, J. J. (2007). Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cerebral Cortex, 17 (5), 1147–1153. doi:10.1093/cercor/bhl024Google Scholar
Schroeder, S. R., Marian, V., Shook, A., & Bartolotti, J. (2016). Bilingualism and Musicianship Enhance Cognitive Control. Neural Plasticity, 2016, 4058620. doi:10.1155/2016/4058620Google Scholar
Shams, L., Kamitani, Y., & Shimojo, S. (2000). What you see is what you hear. Nature, 408 (14), 788.Google Scholar
Shams, L., Kamitani, Y., & Shimojo, S. (2002). Visual illusion induced by sound. Cognitive Brain Research, 14, 147–152.Google Scholar
Shams, L., Kamitani, Y., Thompson, S., & Shimojo, S. (2001). Sound alters visual evoked potentials in humans. Neuroreport, 12 (17), 3849–3852.Google Scholar
Soto-Faraco, S., Navarra, J., Weikum, W. M., Vouloumanos, A., Sebastian-Gallés, N., & Werker, J. F. (2007). Discriminating languages by speech-reading. Perception and Psychophysics, 69 (2), 218–231.Google Scholar
Stanislaw, H., & Todorov, N. (1999). Calculation of signal detection theory measures. Behavior Research Methods, Instruments, & Computers, 31 (1), 137–149. doi:10.3758/BF03207704Google Scholar
Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.Google Scholar
Tabri, D., Smith, K. M., Chacra, A., & Pring, T. (2010). Speech perception in noise by monolingual, bilingual and trilingual listeners. International Journal of Language and Communication Disorders, 1–12.Google Scholar
van Eijk, R. L. J., Kohlrausch, A., Juola, J. F., & van de Par, S. (2008). Audiovisual synchrony and temporal order judgments: Effects of experimental method and stimulus type. Perception and Psychophysics, 70 (6), 955–968. doi:10.3758/pp.70.6.955Google Scholar
Vatikiotis-Bateson, E., Eigsti, I.-M., Yano, S., & Munhall, K. G. (1998). Eye movement of perceivers during audiovisual speech perception. Perception and Psychophysics, 60, 926–940.Google Scholar
von Hapsburg, D., Champlin, C. A., & Shetty, S. R. (2004). Reception thresholds for sentences in bilingual (spanish/english) and monolingual (english) listeners. Journal of the American Academy of Audiology, 15, 88–98.Google Scholar
Wallace, M. T., & Stevenson, R. A. (2014). The construct of the multisensory temporal binding window and its dysregulation in developmental disabilities. Neuropsychologia, 64C, 105–123. doi:10.1016/j.neuropsychologia.2014.08.005Google Scholar
Wojtczak, M., Beim, J. A., Micheyl, C., & Oxenham, A. J. (2012). Perception of across-frequency asynchrony and the role of cochlear delays. Journal of the Acoustical Society of America, 131 (1), 363–377. doi:10.1121/1.3665995Google Scholar
Younkin, A. C., & Corriveau, P. J. (2008). Determining the Amount of Audio-Video Synchronization Errors Perceptible to the Average End-User. IEEE Transactions on Broadcasting, 54 (3), 623–627. doi:10.1109/TBC.2008.2002102Google Scholar
Zhang, J., Stuart, A., & Swink, S. (2011). Word recognition by English monolingual and Mandarin-english bilingual speakers in continuous and interrupted noise. Canadian Journal of Speech-Language Pathology and Audiology, 35 (4), 322–331.Google Scholar
Supplementary material: PDF

Bidelman and Heath supplementary material

Figure S1

Download Bidelman and Heath supplementary material(PDF)
PDF 47 KB