Hostname: page-component-78c5997874-fbnjt Total loading time: 0 Render date: 2024-11-15T21:17:05.915Z Has data issue: false hasContentIssue false
  • English
  • Français

Taking simulation semantics out of the laboratory: towards an interactive and multimodal reappraisal of embodied language comprehension

Published online by Cambridge University Press:  07 July 2014

KASPER KOK  and
ALAN CIENKI
KASPER KOK
Affiliation:
Vrije Universiteit Amsterdam
ALAN CIENKI
Affiliation:
Vrije Universiteit Amsterdam and Moscow State Linguistic University
Get access Rights & Permissions [Opens in a new window]

abstract

Recent embodied theories of meaning known as ‘simulation semantics’ posit that language comprehension engages, or even amounts to, mental simulation. What is meant here by ‘language comprehension’, however, deviates from the perspectives on interpersonal communication adhered to by researchers in social psychology and interactional linguistics. In this paper, we outline four alternative perspectives on comprehension in spoken interaction, each of which highlights factors that have remained largely outside the current purview of simulation theories. These include perspectives on language comprehension in terms of (i) striving for inter-subjective conformity; (ii) recognition of communicative intentions; (iii) prediction and anticipation in a dynamic environment; and (iv) integration of multimodal cues. By contrasting these views with simulation theories of comprehension, we outline a number of fundamental differences in terms of the kind of process comprehension is assumed to be (passive and event-like versus active and continuous), as well as the kind of stimulus that language is assumed to be (comprising unimodal units versus being multimodal and distributed across conversational turns). Finally, we discuss potential points of connection between simulation semantics and research on spoken interaction, and touch on some methodological implications of an interactive and multimodal reappraisal of simulation semantics.

Keywords

simulation semanticscomprehensioninteractionsocial cognitionmultimodality
Type
Research Article
Information
Language and Cognition , Volume 9 , Issue 1 , March 2017 , pp. 1 - 23
Copyright
Copyright © UK Cognitive Linguistics Association 2014 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

references

Altenberg, B. (1987). Prosodic patterns in spoken English: studies in the correlation between prosody and grammar for text-to-speech conversion. Lund: Lund University Press.Google Scholar
Austin, J. (1962). How to do things with words. London: Oxford University Press.Google Scholar
Aziz-Zadeh, L., Wilson, S., Rizzolatti, G., & Iacoboni, M. (2006). Congruent embodied representations for visually presented actions and linguistic phrases describing actions. Current Biology, 16(18), 1818−1823.Google Scholar
Bakhtin, M. (1981). The dialogic imagination. Austin: University of Texas.Google Scholar
Barsalou, L. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22(4), 577−609.Google Scholar
Barsalou, L. (2003). Situated simulation in the human conceptual system. Language and Cognitive Processes, 18(5/6), 513−562.Google Scholar
Barsalou, L. (2005). Situated conceptualization. In Cohen, H. & Lefebvre, C. (Eds.), Handbook of categorization in cognitive science (pp. 619–650). St Louis: Elsevier.Google Scholar
Barsalou, L. (2009). Simulation, situated conceptualization, and prediction. Philosophical Transactions of the Royal Society B: Biological Sciences, 364, 1281−1289.Google Scholar
Barwise, J., & Perry, J. (1981). Situations and attitudes. Journal of Philosophy, 78(11), 668−691.Google Scholar
Barwise, J., & Perry, J. (1983). Situations and attitudes. Cambridge, MA: MIT Press.Google Scholar
Beattie, G., & Shovelton, H. (1999). Mapping the range of information contained in the iconic hand gestures that accompany spontaneous speech. Journal of Language and Social Psychology, 18(4), 438−462.Google Scholar
Bergen, B. K. (2012). Louder than words: the new science of how the mind makes meaning. New York: Basic Books.Google Scholar
Bergen, B. K., & Chang, . (2005). Embodied Construction Grammar in simulation-based language understanding. In Östman, J-O & Fried, M (Eds.), Construction Grammars: cognitive grounding and theoretical extensions (pp. 147−190). Amsterdam & Philadelphia: John Benjamins.Google Scholar
Bergen, B. K., Lindsay, S., Matlock, T., & Narayanan, S. (2007). Spatial and linguistic aspects of visual imagery in sentence comprehension. Cognitive Science, 31(5), 733−764.Google Scholar
Bergen, B. K., & Wheeler, K. (2005). Sentence understanding engages motor processes. Paper presented at the Proceedings of the 27th Annual Conference of the Cognitive Science Society, Mahwah, NJ.Google Scholar
Bernardis, P., & Gentilucci, M. (2006). Speech and gesture share the same communication system. Neuropsychologia, 44(2), 178−190.Google Scholar
Bolinger, D. (1986). Intonation and its parts: melody in spoken English. Stanford: Stanford University Press.Google Scholar
Borghi, A., Glenberg, A., & Kaschak, M. P. (2004). Putting words in perspective. Memory & Cognition, 32(6), 863−873.Google Scholar
Borghi, A., & Riggio, L. (2009). Sentence comprehension and simulation of object temporary, canonical and stable affordances. Brain Research, 1253, 117−128.Google Scholar
Brazil, D. (1997). The communicative value of intonation in English. Cambridge: Cambridge University Press.Google Scholar
Brunyé, T., Ditman, T., Mahoney, C., Augustyn, J., & Taylor, H. (2009). When you and I share perspectives pronouns modulate perspective taking during narrative comprehension. Psychological Science, 20(1), 27−32.Google Scholar
Carruthers, P., & Smith, P. (1996). Theories of theories of mind. Cambridge: Cambridge University Press.Google Scholar
Chafe, W. (1994). Discourse, consciousness, and time: the flow and displacement of conscious experience in speaking and writing. Chicago: University of Chicago Press.Google Scholar
Chafe, W., & Tannen, D. (1987). The relation between written and spoken language. Annual Review of Anthropology, 16, 383−407.Google Scholar
Chao, L., Haxby, J., & Martin, A. (1999). Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nature Neuroscience, 2(10), 913−919.Google Scholar
Chwilla, D., Kolk, H., & Vissers, C. (2007). Immediate integration of novel meanings: N400 support for an embodied view of language comprehension. Brain Research, 1183, 109−123.Google Scholar
Cienki, A. (2012). Usage events of spoken language and the symbolic units we (may) abstract from them. In Badio, J. & Kosecki, K. (Eds.), Cognitive processes in language (pp. 149−158). Bern: Peter Lang.Google Scholar
Clark, H. (1973). Space, time, semantics, and the child. New York: Academic Press.Google Scholar
Clark, H. (1996). Using language. Cambridge: Cambridge University Press.Google Scholar
Clark, H. (1997). Dogmas of understanding. Discourse Processes, 23(3), 567−598.Google Scholar
Clark, H. (1999). On the origins of conversation. Verbum, 2, 147−161.Google Scholar
Clark, H., & Brennan, S. (1991). Grounding in communication. In Resnick, L. B. Levine, J. M., & Teasley, S. D. (Eds.), Perspectives on socially shared cognition (pp. 127−149). Washington, DC: APA Books.Google Scholar
Clark, H., & Krych, M. (2004). Speaking while monitoring addressees for understanding. Journal of Memory and Language, 50(1), 62−81.Google Scholar
Clark, H., & Marshall, C. (1981). Definite reference and mutual knowledge. In Joshi, A. K. Webber, B. L., & Sag, I. A. (Eds.), Elements of discourse understanding (pp. 10−63). Cambridge: Cambridge University Press.Google Scholar
Connell, L. (2007). Representing object colour in language comprehension. Cognition, 102(3), 476−485.Google Scholar
Croft, W. (2009). Towards a social cognitive linguistics. In Evans, V. & Pourcel, S. (Eds.), New directions in cognitive linguistics (pp. 395–420). Amsterdam: John Benjamins.Google Scholar
Dale, R., Fusaroli, R., Duran, N., & Richardson, D. (2013). The self-organization of human interaction. In Ross, H. (Ed.), Psychology of Learning and Motivation, Vol. 59 (pp. 43−95). Online: <http://www.sciencedirect.com/science/article/pii/B9780124071872000022>.Google Scholar
Desai, R., Binder, J., Conant, L., & Seidenberg, M. (2010). Activation of sensory–motor areas in sentence comprehension. Cerebral Cortex, 20(2), 468−478.Google Scholar
Dove, G. (2010). On the need for embodied and dis-embodied cognition. Frontiers in Psychology, 1(242).Google Scholar
Egorova, N., Shtyrov, Y., & Pulvermüller, F. (2013). Early and parallel processing of pragmatic and semantic information in speech acts: neurophysiological evidence. Frontiers in Human Neuroscience, 7(86).Google Scholar
Evans, V. (2009). How words mean. Oxford: Oxford University Press.Google Scholar
Gallese, V. (2007). Before and below ‘theory of mind’: embodied simulation and the neural correlates of social cognition. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1480), 659−669.Google Scholar
Gallese, V., & Goldman, A. (1998). Mirror neurons and the simulation theory of mind-reading. Trends in Cognitive Sciences, 2(12), 493−501.Google Scholar
Garrod, S., & Anderson, A. (1987). Saying what you mean in dialogue: a study in conceptual and semantic co-ordination. Cognition, 27(2), 181−218.Google Scholar
Garrod, S., & Pickering, M. (2004). Why is conversation so easy? Trends in Cognitive Sciences, 8(1), 8−11.Google Scholar
Geeraerts, D. (2010). Recontextualizing grammar: underlying trends in thirty years of Cognitive Linguistics. In Tabakowska, E., Choinski, M., & Wiraszka, L. (Eds.), Cognitive linguistics in action: from theory to application and back (pp. 71−102). Berlin: Mouton de Gruyter.Google Scholar
Gibbs, R. (2006). Metaphor interpretation as embodied simulation. Mind & Language, 21(3), 434−458.Google Scholar
Glenberg, A., & Gallese, V. (2012). Action-based language: a theory of language acquisition, comprehension, and production. Cortex, 48(7), 905−922.Google Scholar
Glenberg, A., & Kaschak, M. (2002). Grounding language in action. Psychonomic Bulletin & Review, 9(3), 558−565.Google Scholar
Glenberg, A., & Robertson, D. (1999). Indexical understanding of instructions. Discourse Processes, 28(1), 1−26.Google Scholar
Goldman, A. (1992). In defense of the simulation theory. Mind & language, 7(1/2), 104−119.Google Scholar
Gopnik, A. (1995). How to understand beliefs. Behavioral and Brain Sciences, 18(02), 398−400.Google Scholar
Graesser, A., Millis, K., & Zwaan, R. (1997). Discourse comprehension. Annual Review of Psychology, 48(1), 163−189.Google Scholar
Graesser, A., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101(3), 371−395.Google Scholar
Hagoort, P., Hald, L., Bastiaansen, M., & Petersson, K. (2004). Integration of word meaning and world knowledge in language comprehension. Science, 304, 438−441.Google Scholar
Hauk, O., Johnsrude, I., & Pulvermüller, F. (2004). Somatotopic representation of action words in human motor and premotor cortex. Neuron, 41(2), 301−307.Google Scholar
Hauk, O., & Pulvermüller, F. (2004). Neurophysiological distinction of action words in the fronto‐central cortex. Human Brain Mapping, 21(3), 191−201.Google Scholar
Havas, D., Glenberg, A., & Rinck, M. (2007). Emotion simulation during language comprehension. Psychonomic Bulletin & Review, 14(3), 436−441.Google Scholar
Holler, J., & Beattie, G. (2003). Pragmatic aspects of representational gestures: Do speakers use them to clarify verbal ambiguity for the listener? Gesture, 3(2), 127−154.Google Scholar
Hostetter, A. (2011). When do gestures communicate? A meta-analysis. Psychological Bulletin, 137(2), 297−315.Google Scholar
Hostetter, A., & Alibali, M. (2008). Visible embodiment: gestures as simulated action. Psychonomic Bulletin & Review, 15(3), 495−514.Google Scholar
Kaschak, M., & Glenberg, A. (2000). Constructing meaning: the role of affordances and grammatical constructions in sentence comprehension. Journal of Memory and Language, 43(3), 508−529.Google Scholar
Kaschak, M., Madden, C., Therriault, D., Yaxley, R., Aveyard, M., Blanchard, A., & Zwaan, R. (2005). Perception of motion affects language processing. Cognition, 94(3), B79−B89.Google Scholar
Kelly, S., Özyürek, A., & Maris, E. (2010). Two sides of the same coin: speech and gesture mutually interact to enhance comprehension. Psychological Science, 21(2), 260−267.Google Scholar
Kendon, A. (1994). Do gestures communicate? A review. Research on Language and Social Interaction, 27(3), 175−200.Google Scholar
Kendon, A. (2004). Gesture: visible action as utterance. Cambridge: Cambridge University Press.Google Scholar
Klatzky, R., Pellegrino, J., McCloskey, B., & Doherty, S. (1989). Can you squeeze a tomato? The role of motor representations in semantic sensibility judgments. Journal of Memory and Language, 28(1), 56−77.Google Scholar
Krauss, R., & Fussell, S. (1988). Other-relatedness in language processing: discussion and comments. Journal of Language and Social Psychology, 7(3/4), 263−279.Google Scholar
Krauss, R., & Fussell, S. (1996). Social psychological models of interpersonal communication. In Higgins, E. T. & Kruglanski, A. (Eds), Social psychology: a handbook of basic principles (pp. 655–701). New York: Guilford Press.Google Scholar
Langacker, R. (1987). Foundations of Cognitive Grammar: theoretical prerequisites. Volume 1. Stanford: Stanford University Press.Google Scholar
Langacker, R. (1997). The contextual basis of cognitive semantics. In Nuyts, J. & Pedersen, E. (Eds.), Language and conceptualization (pp. 229–252). Cambridge: Cambridge University Press.Google Scholar
Langacker, R. (2008). Cognitive Grammar: a basic introduction. Oxford: Oxford University Press.Google Scholar
Linell, P. (1982). The written language bias in linguistics. Linkoping: Deptartment of Communication Studies, University of Linkoping. Republished in revised form in 2005 by Routledge (London).Google Scholar
Linell, P. (1998). Approaching dialogue: talk, interaction and contexts in dialogical perspectives. Amsterdam: John Benjamins Google Scholar
Linell, P. (2007). Dialogicality in languages, minds and brains: Is there a convergence between dialogism and neuro-biology? Language Sciences, 29(5), 605−620.Google Scholar
Mahon, B., & Caramazza, A. (2008). A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content. Journal of Physiology, 102(1/3), 59−70.Google Scholar
Marghetis, T., & Bergen, B. (in press). Embodied meaning, inside and out: the coupling of gesture and mental simulation. In Müller, C., Cienki, A., Fricke, E., Ladewig, S., McNeill, D., & Bressem, J. (Eds.), Body – language – communication: an international handbook on multimodality in human interaction, Vol. 2. Berlin: Mouton De Gruyter.Google Scholar
Masson, M., Bub, D., & Warren, C. (2008). Kicking calculators: contribution of embodied representations to sentence comprehension. Journal of Memory and Language, 59(3), 256−265.Google Scholar
Matlock, T. (2004). Fictive motion as cognitive simulation. Memory & Cognition, 32(8), 1389−1400.Google Scholar
McNeill, D. (1992). Hand and mind: what gestures reveal about thought. Chicago: University of Chicago Press.Google Scholar
Menenti, L., Pickering, M., & Garrod, S. (2012). Toward a neural basis of interactive alignment in conversation. Frontiers in Human Neuroscience, 6(185), 1−9.Google Scholar
Molnar-Szakacs, I., Wu, A., Robles, F., & Iacoboni, M. (2007). Do you see what I mean? Corticospinal excitability during observation of culture-specific gestures. PLoS One, 2(7), e626.Google Scholar
Montgomery, K., Isenberg, N., & Haxby, J. (2007). Communicative hand gestures and object-directed hand movements activated the mirror neuron system. Social Cognitive and Affective Neuroscience, 2(2), 114−122.Google Scholar
Ochs, E., Schegloff, E., & Thompson, S. (1996). Interaction and grammar. Cambridge: Cambridge University Press.Google Scholar
Özyürek, A., Willems, R., Kita, S., & Hagoort, P. (2007). On-line integration of semantic information from speech and gesture: insights from event-related brain potentials. Journal of Cognitive Neuroscience, 19(4), 605−616.Google Scholar
Parker, I. (1992). Discourse dynamics: critical analysis for social and individual psychology. London: Routledge.Google Scholar
Pickering, M., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences, 27(02), 169−190.Google Scholar
Pickering, M., & Garrod, S. (2009). Prediction and embodiment in dialogue. European Journal of Social Psychology, 39(7), 1162−1168.Google Scholar
Pulvermüller, F., Mohr, B., & Schleichert, H. (1999). Semantic or lexico-syntactic factors: What determines word-class specific activity in the human brain? Neuroscience Letters, 275(2), 81−84.Google Scholar
Rączaszek-Leonardi, J. (2009). Symbols as constraints: the structuring role of dynamics and self-organization in natural language. Pragmatics & Cognition, 17(3), 657−676.Google Scholar
Richardson, D., Spivey, M., Barsalou, L., & McRae, K. (2003). Spatial representations activated during real-time comprehension of verbs. Cognitive Science, 27(5), 767−780.Google Scholar
Robinson, E. (2000). The cognitive foundations of pragmatic principles: implications for theories of linguistic and cognitive representation. In Nuyts, J. & Pedersen, E. (Eds.), Language and conceptualization (pp. 253–271). Cambridge: Cambridge University Press.Google Scholar
Sato, M., Mengarelli, M., Riggio, L., Gallese, V., & Buccino, G. (2008). Task related modulation of the motor system during language processing. Brain and Language, 105(2), 83−90.Google Scholar
Shannon, C., & Weaver, W. (1948). The mathematical theory of communication. Urbana: University of Illinois Press.Google Scholar
Skipper, J., Goldin-Meadow, S., Nusbaum, H., & Small, S. (2007). Speech-associated gestures, Broca’s area, and the human mirror system. Brain and Language, 101(3), 260−277.Google Scholar
Speer, N., Zacks, J., & Reynolds, J. (2007). Human brain activity time-locked to narrative event boundaries. Psychological Science, 18(5), 449−455.Google Scholar
Sperber, D., & Wilson, D. (1986). Relevance: communication and cognition. Cambridge, MA: Harvard University Press.Google Scholar
Stanfield, R., & Zwaan, R. (2001). The effect of implied orientation derived from verbal context on picture recognition. Psychological Science, 12(2), 153−156.Google Scholar
Stolorow, R., & Atwood, G. (1992). Contexts of being: the intersubjective foundations of psychological life. Hillsdale: Analytic Press.Google Scholar
Swerts, M., & Geluykens, R. (1994). Prosody as a marker of information flow in spoken discourse. Language and Speech, 37(1), 21−43.Google Scholar
Tettamanti, M., Buccino, G., Saccuman, M., Gallese, V., Danna, M., Scifo, P., Fazio, F., Rizzolatti, G., Cappa, S. F., & Perani, D. (2005). Listening to action-related sentences activates fronto-parietal motor circuits. Journal of Cognitive Neuroscience, 17(2), 273−281.Google Scholar
Thibault, P. (2005). The interpersonal gateway to the meaning of mind: unifying the inter- and intraorganism perspective on language. In Hasan, R., Matthiessen, C., & Webster, J. (Eds.), Continuing discourse on language: a functional perspective (pp. 117–156). London: Equinox.Google Scholar
Van Berkum, J., Van Den Brink, D., Tesink, C., Kos, M., & Hagoort, P. (2008). The neural integration of speaker and message. Journal of Cognitive Neuroscience, 20(4), 580−591.Google Scholar
Van Dam, W., Rueschemeyer, S., Lindemann, O., & Bekkering, H. (2010). Context effects in embodied lexical-semantic processing. Frontiers in Psychology, 1(150).Google Scholar
Van Dijk, T., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press.Google Scholar
Van Elk, M., Slors, M., & Bekkering, H. (2010). Embodied language comprehension requires an enactivist paradigm of cognition. Frontiers in Psychology, 1(234).Google Scholar
Varela, F., Thompson, E., & Rosch, E. (1991). The embodied mind: cognitive science and human experience. Cambridge, MA: MIT press.Google Scholar
Vigliocco, G., Warren, J., Siri, S., Arciuli, J., Scott, S., & Wise, R. (2006). The role of semantics and grammatical class in the neural representation of words. Cerebral Cortex, 16(12), 1790−1796.Google Scholar
Wallentin, M., Nielsen, A. H., Vuust, P., Dohn, A., Roepstorff, A., & Lund, T. E. (2011). BOLD response to motion verbs in left posterior middle temporal gyrus during story comprehension. Brain and Language, 119(3), 221−225.Google Scholar
Willems, R., Benn, Y., Hagoort, P., Toni, I., & Varley, R. (2011). Communicating without a functioning language system: implications for the role of language in mentalizing. Neuropsychologia, 49(11), 3130−3135.Google Scholar
Willems, R., de Boer, M., de Ruiter, J., Noordzij, M., Hagoort, P., & Toni, I. (2010). A dissociation between linguistic and communicative abilities in the human brain. Psychological Science, 21(1), 8−14.Google Scholar
Willems, R., & Francken, J. (2012). Embodied cognition: taking the next step. Frontiers in Psychology, 3(582).Google Scholar
Willems, R., & Hagoort, P. (2007). Neural evidence for the interplay between language, gesture, and action: a review. Brain and Language, 101(3), 278−289.Google Scholar
Willems, R., & Varley, R. (2010). Neural insights into the relation between language and communication. Frontiers in Human Neuroscience, 4(203).Google Scholar
Wilson, A., & Golonka, S. (2013). Embodied cognition is not what you think it is. Frontiers in Psychology, 4(58).Google Scholar
Winter, B., & Bergen, B. (2012). Language comprehenders represent object distance both visually and auditorily. Language and Cognition, 4, 1−16.Google Scholar
Yaxley, R., & Zwaan, R. (2007). Simulating visibility during language comprehension. Cognition, 105(1), 229−236.Google Scholar
Zwaan, R. (2003). The immersed experiencer: toward an embodied theory of language comprehension. Psychology of Learning and Motivation, 44, 35−62.Google Scholar
Zwaan, R. (2009). Mental simulation in language comprehension and social cognition. European Journal of Social Psychology, 39(7), 1142−1150.Google Scholar
Zwaan, R. (2014). Embodiment and language comprehension: reframing the discussion. Trends in Cognitive Sciences 18(5), 229−234.Google Scholar
Zwaan, R., Madden, C., Yaxley, R., & Aveyard, M. (2004). Moving words: dynamic representations in language comprehension. Cognitive Science, 28(4), 611−619.Google Scholar
Zwaan, R., Magliano, J., & Graesser, A. (1995). Dimensions of situation model construction in narrative comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(2), 386−397.Google Scholar
Zwaan, R., & Radvansky, G. (1998). Situation models in language comprehension and memory. Psychological Bulletin, 123(2), 162−185.Google Scholar
Zwaan, R., Stanfield, R., & Yaxley, R. (2002). Language comprehenders mentally represent the shapes of objects. Psychological Science, 13(2), 168−171.Google Scholar
Zwaan, R., & Taylor, L. (2006). Seeing, acting, understanding: motor resonance in language comprehension. Journal of Experimental Psychology: General, 135(1), 1−11.Google Scholar