Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-hc48f Total loading time: 0 Render date: 2024-12-27T09:31:48.315Z Has data issue: false hasContentIssue false

17 - Multisensory Perception and Mental Imagery

from Part II - Imagery-Based Forms of the Imagination

Published online by Cambridge University Press:  26 May 2020

By
Christopher C. Berger
Edited by
Anna Abraham
Anna Abraham
Affiliation:
University of Georgia
Get access

Summary

Mental imagery is what we experience when we imagine seeing a specific object, hearing a particular sound, or feeling a particular touch, and it is perhaps the most fundamental aspect of our imagination. Historically, research on mental imagery has explored the phenomenological and neurological similarities between mental imagery and sensory perception to understand the quasi-perceptual nature of these conjured “images” we experience in the “mind’s eye.” However, this line of research has traditionally focused on the similarities of mental imagery and perception within each sensory modality, and the relationship between mental imagery in one sense and its effects on perception in another sense or on our perception of the world around us as a whole has largely been ignored. This chapter will extend the study of the relationship between mental imagery and perception into a multisensory context, and utilize insights from research in neuroscience and multisensory perception to explore how mental imagery in one sense can affect ongoing perception in another. The chapter will also examine the similarities in how the brain processes and integrates imagined and real crossmodal sensory stimuli. Lastly, the chapter will discuss how the integration of mental imagery and real sensory stimuli can lead to brain plasticity across the senses and change how we perceive the world around us in the future.

Keywords

Mental imagerymultisensory integrationmultisensory perceptionmultisensory illusions
Type
Chapter
Information
The Cambridge Handbook of the Imagination , pp. 258 - 275
Publisher: Cambridge University Press
Print publication year: 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Alais, D., and Burr, D. (2004). The Ventriloquist Effect Results from Near-Optimal Bimodal Integration. Current Biology: CB, 14(3), 257–262.Google Scholar
Albright, T. D. (2012). On the Perception of Probable Things: Neural Substrates of Associative Memory, Imagery, and Perception. Neuron, 74(2), 227–245.CrossRefGoogle ScholarPubMed
Allen, A. K., Wilkins, K., Gazzaley, A., and Morsella, E. (2013). Conscious Thoughts from Reflex-Like Processes: A New Experimental Paradigm for Consciousness Research. Consciousness and Cognition, 22(4), 1318–1331.CrossRefGoogle ScholarPubMed
Andersen, T. S., Tiippana, K., and Sams, M. (2004). Factors Influencing Audiovisual Fission and Fusion Illusions. Cognitive Brain Research, 21(3), 301–308.Google Scholar
Barraclough, N. E., Xiao, D., Baker, C. I., Oram, M. W., and Perrett, D. I. (2005). Integration of Visual and Auditory Information by Superior Temporal Sulcus Neurons Responsive to the Sight of Actions. Journal of Cognitive Neuroscience, 17(3), 377–391.CrossRefGoogle Scholar
Beauchamp, M. S., Argall, B. D., Bodurka, J., Duyn, J. H., and Martin, A. (2004a). Unraveling Multisensory Integration: Patchy Organization within Human STS Multisensory Cortex. Nature Neuroscience, 7(11), 1190–1192.CrossRefGoogle ScholarPubMed
Beauchamp, M. S., Lee, K. E., Argall, B. D., and Martin, A. (2004b). Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus. Neuron, 41(5), 809–823. www.ncbi.nlm.nih.gov/pubmed/15813999.Google Scholar
Beauchamp, M. S., Nath, A. R., and Pasalar, S. (2010). fMRI-Guided Transcranial Magnetic Stimulation Reveals that the Superior Temporal Sulcus is a Cortical Locus of the McGurk Effect. Journal of Neuroscience, 30(7), 2414–2417.CrossRefGoogle ScholarPubMed
Berger, C. C., and Ehrsson, H. H. (2013). Mental Imagery Changes Multisensory Perception. Current Biology, 23, 1367–1372.CrossRefGoogle ScholarPubMed
Berger, C. C., and Ehrsson, H. H. (2014). The Fusion of Mental Imagery and Sensation in the Temporal Association Cortex. Journal of Neuroscience, 34(41), 13684–13692.CrossRefGoogle ScholarPubMed
Berger, C. C., and Ehrsson, H. H. (2017). The Content of Imagined Sounds Changes Visual Motion Perception in the Cross-Bounce Illusion. Scientific Reports, 7, 40123.Google Scholar
Berger, C. C., and Ehrsson, H. H. (2018). Mental Imagery Induces Crossmodal Sensory Plasticity and Changes Future Auditory Perception. Psychological Science, 29(6), 926–935.CrossRefGoogle ScholarPubMed
Bischoff, M., Walter, B., Blecker, C. R., et al. (2007). Utilizing the Ventriloquism-Effect to Investigate Audiovisual Binding. Neuropsychologia, 45(3), 578–586.Google Scholar
Bonath, B., Noesselt, T., Martinez, A., et al. (2007). Neural Basis of the Ventriloquist Illusion. Current Biology: CB, 17(19), 1697–1703.Google Scholar
Bruce, C., Desimone, R., and Gross, C. G. (1981). Visual Properties of Neurons in a Polysensory Area in Superior Temporal Sulcus of the Macaque. Journal of Neurophysiology, 46(2), 369–384. www.ncbi.nlm.nih.gov/pubmed/6267219.Google Scholar
Calvert, G. A., Campbell, R., and Brammer, M. J. (2000). Evidence from Functional Magnetic Resonance Imaging of Crossmodal Binding in the Human Heteromodal Cortex. Current Biology: CB, 10(11), 649–657. www.ncbi.nlm.nih.gov/pubmed/10837246.Google Scholar
Cichy, R. M., Heinzle, J., and Haynes, J.-D. (2011). Imagery and Perception Share Cortical Representations of Content and Location. Cerebral Cortex, 22(2), 372–380.CrossRefGoogle ScholarPubMed
Craver-Lemley, C., and Reeves, A. (1987). Visual Imagery Selectively Reduces Vernier Acuity. Perception, 16(5), 599–614.CrossRefGoogle ScholarPubMed
Dils, A. T., and Boroditsky, L. (2010). Visual Motion Aftereffect from Understanding Motion Language. Proceedings of the National Academy of Sciences of the United States of America, 107, 16396–16400.Google Scholar
Driver, J., and Noesselt, T. (2008). Multisensory Interplay Reveals Crossmodal Influences on “Sensory-Specific” Brain Regions, Neural Responses, and Judgments. Neuron, 57(1), 11–23.Google Scholar
Farah, M. J. (1985). Psychophysical Evidence for a Shared Representational Medium for Mental Images and Percepts. Journal of Experimental Psychology. General, 114(1), 91–103. www.ncbi.nlm.nih.gov/pubmed/3156947.Google Scholar
Farah, M. J. (1989a). Mechanisms of Imagery-Perception Interaction. Journal of Experimental Psychology. Human Perception and Performance, 15(2), 203–211. www.ncbi.nlm.nih.gov/pubmed/2525596.Google Scholar
Farah, M. J. (1989b). The Neural Basis of Mental Imagery. Trends in Neurosciences, 12(10), 395–399. www.ncbi.nlm.nih.gov/pubmed/8137002.Google Scholar
Frissen, I., Vroomen, J., and de Gelder, B. (2012). The Aftereffects of Ventriloquism: The Time Course of the Visual Recalibration of Auditory Localization. Seeing and Perceiving, 25(1), 1–14.Google Scholar
Frissen, I., Vroomen, J., de Gelder, B., and Bertelson, P. (2005). The Aftereffects of Ventriloquism: Generalization Across Sound-Frequencies. Acta Psychologica, 118(1–2), 93–100.CrossRefGoogle ScholarPubMed
Ghazanfar, A. A., Chandrasekaran, C., and Logothetis, N. K. (2008). Interactions Between the Superior Temporal Sulcus and Auditory Cortex Mediate Dynamic Face/Voice Integration in Rhesus Monkeys. Journal of Neuroscience, 28(17), 4457–4469.Google Scholar
Ghazanfar, A. A., and Schroeder, C. E. (2006). Is Neocortex Essentially Multisensory? Trends in Cognitive Sciences, 10(6), 278–285.CrossRefGoogle ScholarPubMed
Grassi, M., and Casco, C. (2009). Audiovisual Bounce-Inducing Effect: Attention Alone Does Not Explain Why the Discs Are Bouncing. Journal of Experimental Psychology. Human Perception and Performance, 35(1), 235–243.Google Scholar
Grassi, M., and Casco, C. (2012). Revealing the Origin of the Audiovisual Bounce-Inducing Effect. Seeing and Perceiving, 25(2), 223–233.Google Scholar
Halpern, A. R. (1988). Mental Scanning in Auditory Imagery for Songs. Journal of Experimental Psychology. Learning, Memory, and Cognition, 14(3), 434–443. www.ncbi.nlm.nih.gov/pubmed/2969942.CrossRefGoogle ScholarPubMed
Howard, I. P., and Templeton, W. B. (1966). Human Spatial Orientation. London, UK: Wiley.Google Scholar
Hubbard, T. L. (2010). Auditory Imagery: Empirical Findings. Psychological Bulletin, 136(2), 302–329.Google Scholar
Johns, L. C., Rossell, S., Frith, C., et al. (2001). Verbal Self-Monitoring and Auditory Verbal Hallucinations in Patients with Schizophrenia. Psychological Medicine, 31, 705–715.CrossRefGoogle ScholarPubMed
Kayser, C., and Logothetis, N. K. (2009). Directed Interactions between Auditory and Superior Temporal Cortices and their Role in Sensory Integration. Frontiers in Integrative Neuroscience, 3(May), 1–11.Google Scholar
Kosslyn, S. M. (1973). Scanning Visual Images: Some Structural Implications. Perception & Psychophysics, 14(1), 90–94.CrossRefGoogle Scholar
Kosslyn, S. M. (1994). Image and Brain: The Resolution of the Imagery Debate. Cambridge, MA: MIT Press.Google Scholar
Kosslyn, S. M., Ball, T. M., and Reiser, B. J. (1978). Visual Images Preserve Metric Spatial Information: Evidence from Studies of Image Scanning. Journal of Experimental Psychology. Human Perception and Performance, 4(1), 47–60. www.ncbi.nlm.nih.gov/pubmed/627850.Google Scholar
Kosslyn, S. M., Ganis, G., and Thompson, W. L. (2001). Neural Foundations of Imagery. Nature Reviews. Neuroscience, 2(9), 635–642.Google Scholar
Lewald, J. (2002). Rapid Adaptation to Auditory-Visual Spatial Disparity. Learning & Memory, 9(5), 268–278.Google Scholar
Macmillan, N. A., and Kaplan, H. L. (1985). Detection Theory Analysis of Group Data: Estimating Sensitivity from Average Hit and False-Alarm Rates. Psychological Bulletin, 98(1), 185–199.Google Scholar
Magnotti, J. F., Basu Mallick, D., Feng, G., et al. (2015). Similar Frequency of the McGurk Effect in Large Samples of Native Mandarin Chinese and American English Speakers. Experimental Brain Research, 233(9), 2581–2586.Google Scholar
Marchant, J. L., Ruff, C. C., and Driver, J. (2012). Audiovisual Synchrony Enhances BOLD Responses in a Brain Network Including ultisensory STS While Also Enhancing Target-Detection Performance for Both Modalities. Human Brain Mapping, 33(5), 1212–1224.CrossRefGoogle Scholar
Mast, F. W., Berthoz, A., and Kosslyn, S. M. (2001). Mental Imagery of Visual Motion Modifies the Perception of Roll-Vection Stimulation. Perception, 30(8), 945–957.Google Scholar
McGurk, H., and MacDonald, J. (1976). Hearing Lips and Seeing Voices. Nature, 264(23), 746–748. www.nature.com/nature/journal/v264/n5588/abs/264746a0.html.Google Scholar
Nath, A. R., and Beauchamp, M. S. (2011). Dynamic Changes in Superior Temporal Sulcus Connectivity during Perception of Noisy Audiovisual Speech. Journal of Neuroscience, 31(5), 1704–1714.Google Scholar
Noesselt, T., Rieger, J. W., Schoenfeld, M. A., et al. (2007). Audiovisual Temporal Correspondence Modulates Human Multisensory Superior Temporal Sulcus plus Primary Sensory Cortices. Journal of Neuroscience, 27(42), 11431–11441.Google Scholar
Noppeney, U., Josephs, O., Hocking, J., Price, C. J., and Friston, K. J. (2008). The Effect of Prior Visual Information on Recognition of Speech and Sounds. Cerebral Cortex, 18(3), 598–609.Google Scholar
O’Craven, K. M., and Kanwisher, N. (2000). Mental Imagery of Faces and Places Activates Corresponding Stimulus-Specific Brain Regions. Journal of Cognitive Neuroscience, 12(6), 1013–1023. www.ncbi.nlm.nih.gov/pubmed/11177421.Google Scholar
Pearson, J., Clifford, C. W. G., and Tong, F. (2008). The Functional Impact of Mental Imagery on Conscious Perception. Current Biology, 18(13), 982–986.Google Scholar
Perky, C. W. (1910). An Experimental Study of Imagination. American Journal of Psychology, 21(3), 422–452.Google Scholar
Perrodin, C., Kayser, C., Logothetis, N. K., and Petkov, C. I. (2014). Auditory and Visual Modulation of Temporal Lobe Neurons in Voice-Sensitive and Association Cortices. Journal of Neuroscience, 34(7), 2524–2537.Google Scholar
Plaze, M., Paillère-Martinot, M.-L., Penttilä, J., et al. (2011). “Where do Auditory Hallucinations Come From?”A Brain Morphometry Study of Schizophrenia Patients with Inner or Outer Space Hallucinations. Schizophrenia Bulletin, 37(1), 212–221.Google Scholar
Pylyshyn, Z. W. (1973). What the Mind’s Eye Tells the Mind’s Brain: A Critique of Mental Imagery. Psychological Bulletin, 80(1), 1–12.CrossRefGoogle Scholar
Pylyshyn, Z. W. (2002). Mental Imagery: In Search of a Theory. The Behavioral and Brain Sciences, 25, 157–182.Google Scholar
Recanzone, G. H. (1998). Rapidly Induced Auditory Plasticity: The Ventriloquism Aftereffect. Proceedings of the National Academy of Sciences of the United States of America, 95(February), 869–875.Google Scholar
Schlack, A., and Albright, T. D. (2007). Remembering Visual Motion: Neural Correlates of Associative Plasticity and Motion Recall in Cortical Area MT. Neuron, 53(6), 881–890.CrossRefGoogle ScholarPubMed
Segal, S. J., and Fusella, V. (1970). Influence of Imaged Pictures and Sounds on Detection of Visual and Auditory Signals. Journal of Experimental Psychology, 83(3), 458–464. www.ncbi.nlm.nih.gov/pubmed/5480913.Google Scholar
Sekuler, R., Sekuler, A. B., and Lau, R. (1997). Sound Alters Visual Motion Perception. Nature, 385(6614), 308. www.ncbi.nlm.nih.gov/pubmed/9002513.Google Scholar
Seltzer, B., and Pandya, D. N. (1994). Parietal, Temporal, and Occipital Projections to Cortex of the Superior Temporal Sulcus in the Rhesus Monkey: A Retrograde Tracer Study. The Journal of Comparative Neurology, 343(3), 445–463.Google Scholar
Shams, L., Kamitani, Y., and Shimojo, S. (2000). What You See Is What You Hear. Nature, 408(6814), 788. doi:10.1038/35048669.Google Scholar
Shimojo, S., and Shams, L. (2001). Sensory Modalities Are Not Separate Modalities: Plasticity and Interactions. Current Opinion in Neurobiology, 11(4), 505–509.Google Scholar
Spence, C., and Deroy, O. (2012). Hearing Mouth Shapes: Sound Symbolism and the Reverse McGurk Effect. I-Perception, 3(8), 550–552.Google Scholar
Stein, B. E., and Stanford, T. R. (2008). Multisensory Integration: Current Issues from the Perspective of the Single Neuron. Nature Reviews. Neuroscience, 9(4), 255–266.Google Scholar
Stevenson, R. A., and James, T. W. (2009). Audiovisual Integration in Human Superior Temporal Sulcus: Inverse Effectiveness and the Neural Processing of Speech and Object Recognition. NeuroImage, 44(3), 1210–1223.Google Scholar
Sweeny, T. D., Guzman-Martinez, E., Ortega, L., Grabowecky, M., and Suzuki, S. (2012). Sounds Exaggerate Visual Shape. Cognition, 124(2), 194–200.Google Scholar
Szycik, G. R., Stadler, J., Tempelmann, C., and Münte, T. F. (2012). Examining the McGurk Illusion Using High-Field 7 Tesla Functional MRI. Frontiers in Human Neuroscience, 6(April), 95.Google Scholar
van Wassenhove, V., Grant, K. W., and Poeppel, D. (2007). Temporal Window of Integration in Auditory-Visual Speech Perception. Neuropsychologia, 45(3), 598–607.Google Scholar
Wallace, M. T., Roberson, G. E., Hairston, W. D., et al. (2004). Unifying Multisensory Signals Across Time and Space. Experimental Brain Research. Experimentelle Hirnforschung. Expérimentation Cérébrale, 158(2), 252–258.CrossRefGoogle ScholarPubMed
Weber, R. J., and Castleman, J. (1970). The Time It Takes to Imagine. Perception & Psychophysics, 8(3), 165–168.Google Scholar
Wegner, D. M. (1994). Ironic Processes of Mental Control. Psychological Review, 101(1), 34–52.Google Scholar
Werner, S., and Noppeney, U. (2010a). Distinct Functional Contributions of Primary Sensory and Association Areas to Audiovisual Integration in Object Categorization. Journal of Neuroscience, 30(7), 2662–2675.Google Scholar
Werner, S., and Noppeney, U. (2010b). Superadditive Responses in Superior Temporal Sulcus Predict Audiovisual Benefits in Object Categorization. Cerebral Cortex, 20(8), 1829–1842.Google Scholar
Winawer, J., Huk, A. C., and Boroditsky, L. (2010). A Motion Aftereffect from Visual Imagery of Motion. Cognition, 114(2), 276–284.Google Scholar
Woods, T. M., and Recanzone, G. H. (2004). Visually Induced Plasticity of Auditory Spatial Perception in Macaques. Current Biology, 14, 1559–1564.Google Scholar
Wozny, D. R., and Shams, L. (2011). Recalibration of Auditory Space Following Milliseconds of Cross-Modal Discrepancy. Journal of Neuroscience, 31(12), 4607–4612.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×