The pervasive use of technology has changed dramatically how we create meaning and interact with one another. Smartphones, tablets and laptops have equipped us with a plethora of applications that we can use creatively to communicate in ever more effective ways. We can now have a text-based conversation on our devices with photos, web links (possibly to audiovisual media) and even GPS locations. Videoconferencing platforms enable us to combine voice and text chat, share all kinds of files (documents, pictures, audio/video recordings, whiteboards) and work on them collaboratively. The immersive platforms of virtual worlds integrate written and oral communication tools, which can be complemented with an array of bodily movements that avatars can reproduce in different scenarios while engaging in action. Additionally, in online games, and more specifically in Massively Multiplayer Online Games (MMOGs), gamers interact as avatars with other players in complex ways resorting to different communication modes to solve problems, complete quests, plan strategies and coordinate action to achieve the game goals.
The widespread availability and sophistication of multimodal communication tools have captured the imagination of practitioners and scholars alike, who have identified interesting learning opportunities and fields of enquiry for language teaching. Many studies have been carried out in the past three decades looking at the affordances that digital tools have for facilitating meaningful communication among peers and for enhancing second language (L2) learning. However, most of this research has been looking at meaning-making processes in communication from a traditionally linguistic perspective. Only recently have scholars initiated the methodological turn towards looking at communication processes from a multimodal perspective in order to grasp the complexity of all the layers and modes involved in meaning creation.
Multimodality, as defined by Kress and van Leeuwen (Reference Kress and van Leeuwen2001: 20), is “the use of several semiotic modes in the design of a semiotic product or event, together with the particular way in which these modes are combined – they may for instance reinforce each other […], fulfil complementary roles […] or [be] hierarchically ordered”. In computer assisted language learning (CALL), different semiotic modes are also involved in the meaning-making process – exchanges may take place orally and/or in writing, and may benefit from additional information such as non-verbal cues, graphics or social presence indicators. All of these provide learners with new ways to negotiate meaning in language learning.
A growing body of research on multimodality studies is emerging in the field of communication and CALL. Most scholars analyse this concept from a social semiotic perspective (Kress & Jewitt, Reference Kress and Jewitt2003; Kress, Reference Kress2010), based on van Leeuwen’s socio-semiotic theory (Reference van Leeuwen2004), or through a multimodal interactional analysis approach (Norris, Reference Norris2004), both regarding language use in general (Bednarek & Martin, Reference Bednarek and Martin2011; Jing, Reference Jing2011; Kojima, Brown, Rothermel, Carlson & Matzuzaki et al., Reference Kojima, Brown, Rothermel, Carlson and Matsuzaki2012) and CALL in particular (Guichon & McLornan, Reference Guichon and McLornan2008; Levy & Stockwell, Reference Levy and Stockwell2013). According to Levy, digitally mediated communication is creatively multimodal, including “multi-purpose, multifunctional technologies that involve layers of complexity and application in L2 learning that are unique among the technologies of the modern world” (Levy, Reference Levy2009: 779).
Multimodality in desktop videoconferencing has been analysed with a particular focus on gaze and eye contact (Satar, Reference Satar2013), webcam framing (Codreanu & Combe Celik, Reference Codreanu and Combe Celik2013), webcam use and gestures (Develotte, Guichon & Vincent, Reference Develotte, Guichon and Vincent2010) and eye movement (Stickler & Shi, Reference Stickler and Shi2015). Regarding multimodality in virtual worlds, scholars have researched the interaction between avatars and objects (Panichi, Reference Panichi2015) as well as the role of non-verbal communication (Wigham, Reference Wigham2012; Wigham & Chanier, Reference Wigham and Chanier2013), the virtual scenario and avatar actions affecting exchanges (Jauregi, Canto, de Graaff, Koenraad & Moonen, Reference Jauregi, Canto, de Graaff, Koenraad and Moonen2011) and social presence (Satar, Reference Satar2015). More recently, attention has been devoted to the problem-solving nature of games (Reinders, Reference Reinders2012; Cornillie, Thorne & Desmet, Reference Cornillie, Thorne and Desmet2012) in immersive virtual environments that create the conditions for embodied interaction. Gamers collaboratively solve problems and achieve game goals through multimodal communication and by coordinating actions (Zheng, Newgarden & Young, Reference Zheng, Newgarden and Young2012; Zheng, Wagner, Young & Brewer, Reference Zheng, Wagner, Young and Brewer2009).
Student participation and how it is affected by multimodality has also attracted some attention, resulting in studies on learner practices in audiographic exchanges (Ciekanski & Chanier, Reference Ciekanski and Chanier2008) and participation patterns (Hampel & Stickler, Reference Hampel and Stickler2012). Other researchers have analysed the value of using digital stories as multimodal products in the foreign language classroom (Oskoz & Elola, Reference Oskoz and Elola2014) or how foreign language students make use of the different codes that are available to them in a digital story to construct meaning (Yang, Reference Yang2012). Multimodality must certainly be also taken into account when it comes to task design (Berglund, Reference Berglund2009; Canto, de Graaff & Jauregi, Reference Canto, de Graaff and Jauregi2014; Hampel, Reference Hampel2006, Reference Hampel2010; Hauck, Reference Hauck2010; Stockwell, Reference Stockwell2010).
Some scholars have looked at how providing multiple modes of computer mediated communication might influence learners’ L2 learning processes. Sauro (Reference Sauro2009) has focused on bimodal synchronous interaction using voice and text chat; Abrams (Reference Abrams2003) explored the differences between synchronous chat and asynchronous bulletin boards; Pérez (Reference Pérez2003) analysed chat and emails; and Canto, Jauregi and van den Bergh (Reference Canto, Jauregi and van den Bergh2013) compared classroom face-to-face interactions, video-communication and interaction in virtual worlds.
This special issue of ReCALL aims to present a collection of innovative papers that will pave the way for future research on multimodality in CALL, addressing its many dimensions and the interactions between them (linguistic and iconic systems, sounds, images, kinesis, and learner embodiment, among others). The papers in this volume present different approaches to carrying out multimodal analysis that best capture the essence of multimodal meaning co-creation enacted in the specific pedagogical learning context. The environments used in the studies could not be more diverse: from the MMOGs or virtual worlds to desktop videoconferencing software, digital stories, a TV series in combination with the programme’s online resources and face-to-face interaction. Furthermore, the languages covered in this Special Issue are also varied – papers deal with multimodal interactions in English, Spanish, German and Turkish.
Sabine Tan, Kay L. O’Halloran and Peter Wignell address the need to utilise multimodal analysis to disentangle the intricacies enacted in meaning creation by adopting a multimodal social semiotic approach. The authors compare how specific semiotic systems (group dynamics, dialogue, gaze, gesture and space) are used in conventional face-to-face language teaching and in three Second Life language learning activities. The detailed analysis of the excerpts used from each setting shows that there are noticeable differences in how meaning is constructed in a face-to-face and virtual world language teaching context with regard to the semiotic systems analysed, which the authors claim to be crucial when it comes to creating an effective learning environment. They conclude that 3D virtual worlds need to progress further technically as avatars need to be able to direct gaze purposefully during interaction and use gestures while communicating in order to provide language students with some of the key affordances found in conventional face-to-face settings, which they find necessary to create a rich learning environment.
Kristi Newgarden and Dongping Zheng analyse how interaction via a multiplayer game like World of Warcraft supports L2 learners’ language skills. The participants, university students, were enrolled in a semester-long English course. They played the game in groups made up of native and non-native speakers of English. Gaming data and the Skype conversations were recorded and learners’ verbalisation and avatar actions analysed, focusing on recurrent languaging activities (city activities, learning a skill, planning moves, playing around, questing, random fighting, talking about past and future gaming or travelling), communicative activities (meaning-making, facilitating gameplay and taking care of others’ needs), languaging modes (moving only, verbalising only, doing both, coordinated verbalisation and movement and multitasking) and initiation/response moves. The results show that the recurrent languaging activities of the game facilitate contextualised rich practice of communication in L2 at different Common European Framework of Reference levels.
Müge Satar highlights the importance of online language learning and teaching in multimodal contexts and analyses how the semiotic resources integrated in desktop videoconferencing can help learners to construct meaning during online interaction sessions. In her study, a total of ten English trainee teachers from Turkey took part in 3–4 one-hour interaction sessions during which they were asked to complete a series of open-ended tasks to stimulate interpersonal interaction. The transcript of the sessions is the main data source, although interviews and questionnaires were also used for triangulation. The analysis revealed that five semiotic resources were used by the participants, i.e. paralinguistic contextualisation cues, different codes of the body, facial expression and voice to express emotions, use of gestures and overlapping speech. Following the results of the study, Satar suggests exploring multimodal resources in intercultural communication and investigating the role of other available semiotic resources in meaning-making in online language learning and communication contexts.
Ana Oskoz and Idoia Elola explore how students make use of multimodality in the creation of digital stories. Their research looks into the interaction with genres including multimodal semiotic resources in an advanced Spanish writing course within the social semiotics and activity theory frameworks. In their study, the researchers analyse how learners managed to form an integrated multimodal text by using and synthesising linguistic, visual, audio, and spatial modes of representation. Namely, six undergraduate students from a US university were enrolled in a semester-long writing course that included one session per week. Researchers analysed participants’ reflections on their creative processes, questionnaires in connection with the activity theory framework, and three online journal entries in which students reflected upon the creation of the digital stories. The results of this study show that short-term oriented actions associated with the new genre resulted in a better understanding of their long-term objective, that learners managed to perceive the interplay between multiple modes, and that such modes were successfully manipulated to move from a traditional text to the narrative of a multimodal digital story. All in all, according to the authors, these results lead to the conclusion that including digital stories in the foreign language classroom can be a valuable pedagogical practice.
Zsuzsanna Abrams suggests combining multimodal analysis and task-based language teaching. In her study, a total of 30 first-year students enrolled in a beginners’ German course at an American university took part in this four-week long study. A number of task-based activities were designed around the German TV series Rosenheim-Cops and the programme’s multimedia resources available online, such as “making of” videos, the series’ Facebook page or other users’ reviews of the programme. All tasks involved analysing how different layers of meaning (written and oral interactions, music, gestures and body posture among others) contribute to the final message. They were designed in such a way that students developed lexical, sociolinguistic and pragmatic knowledge as well as intercultural awareness and L2 digital literacy. The tasks proposed guided students in the analysis of the multiple codes that construct meaning, therefore enabling novice learners of German to access authentic audiovisual materials in the target language.
We would like to thank the authors for the quality and timeliness of their contributions, which highlight the relevance of the topic and point to exciting new avenues of research. We hope that this Special Issue will help advance knowledge in the field of multimodality and lay foundations for a constructive and critical debate offering new possibilities for future empirical and theoretical research in CALL.
Acknowledgements
We would like to express our gratitude to Jozef Colpaert, Mats Deutschmann, Melinda Dooly, Regine Hampel, Mirjam Hauck, Susanna Nocchi, Robert O’Dowd, Luisa Panichi, Bryan Smith, Glenn Stockwell and Uschi Stickler, who kindly agreed to review the papers published in this Special Issue. Our thanks also go to Sylvie Thouësny, for all the support provided throughout the editing process, and especially to ReCALL’s editors, Françoise Blin and Alex Boulton, for giving us the opportunity to make this Special Issue a reality.
