Wearables in sociodrama: An embodied mixed-methods study of expressiveness in social interactions

Abstract

This mixed-methods study investigates the use of wearable technology in embodied psychology research and explores the potential of incorporating bio-signals to focus on the bodily impact of the social experience. The study relies on scientifically established psychological methods of studying social issues, collective relationships and emotional overloads, such as sociodrama, in combination with participant observation to qualitatively detect and observe verbal and nonverbal aspects of social behavior. We evaluate the proposed method through a pilot sociodrama session and reflect on the outcomes. By utilizing an experimental setting that combines video cameras, microphones, and wearable sensors measuring physiological signals, specifically, heart rate, we explore how the synchronization and analysis of the different signals and annotations enables a mixed-method that combines qualitative and quantitative instruments in studying embodied expressiveness and social interaction.

Introduction

Wearable technologies play an important role in innovations across many disciplines and are transforming the way we conduct social science research. They raise new challenges for social and psychological inquiry, and in particular for methodological explorations related to multimodal and multisensory research.

This article describes an innovative methodology that can be utilized to research and analyze the embodied impact of social issues and dynamics. It is part of the ongoing transdisciplinary project “Transition to 8” that aims at applying tools and methodologies for studying the multisensorial and experiential impact of social local issues on citizens and bridging them with technology and contemporary art. More specifically, through the project an online platform will be developed to enable the creation of artistic projects based on biometric and other data derived from using wearables to record the reactions of the human body in the framework of sociodrama sessions, during which participants express themselves and deliberate on important collective social issues. Within this context, the main objective of this work is to define the methods that will allow deeper multimodal analysis and bridge the gap between quantitative and qualitative approaches.

To do so we propose a mixed-methods study that incorporates wearables during sociodrama sessions. The wearables are used complementary with other methods of ethnography studies, such as human on-site observation, audio and video recording, thematic analysis of transcription, observation and annotation of the recordings.

This article presents and discusses the aforementioned methodology in the context of (a) a proposed study that will explore through sociodrama sessions the perspectives and experiences of residents in Eleusis regarding their living conditions and (b) a pilot sociodrama study that was conducted. The article is structured as follows: In section “Theoretical Background and Related Work,” we present the theoretical background for this research, we explain the embodied turn in psychology, define sociodrama and briefly describe related work of using wearables in psychology. In section “Proposed Study and Methodology,” we describe in detail the research methodology in the context of the proposed study “Pilot Study and Outcomes,” we present the outcomes of the pilot sociodrama study, including the experimental data collection and analysis. Finally, in section “Discussion,” we reflect on the outcomes and their use in psychology research and in section “Conclusion and Future Work,” we conclude the article and present future research plans.

Theoretical Background and Related Work

The Embodied Turn

In recent years, there has been an increasing interest in using methodologies that go beyond the domain of verbal descriptions and include embodied aspects in research leading the so called “embodied turn” or “corporeal turn” in human sciences (Brown et al., 2011; Seikkula et al., 2015). Indeed, there is evidence to suggest that most of the communication between people takes place nonverbally (Mehrabian, 1981, 2009) estimated that non-verbal gestures and expressions communicated 93% of people’s feelings and attitudes. Similarly, Birdwhistell (1952) argued that no more than 30% of a social interaction and its attached meaning happens verbally. Even in an everyday context, if a person argues she feels sad but her facial expression and body gestures indicate that she is angry, people tend to believe more what is communicated nonverbally.

According to Denham and Onwuegbuzie (2013), studying nonverbal communication could help qualitative researchers in various ways, such as increasing validity through triangulation, capturing underlying messages, gaining new insights on nonverbal elements and discovering potential contradictions between verbal and nonverbal behavior. Yet, as Brown et al. (2011) noted, embodiment in research poses ontological and methodological challenges. Although there has been a significant shift toward the inclusion of the body in research, little is mentioned regarding the ways in which current methodologies can move the focus away from discourse (Brown et al., 2011). Furthermore, there is currently no unified methodological approach. The existed embodied methodologies are informed by various theoretical frameworks ranging from phenomenology to neuroscience (Spatz, 2017).

Whilst these conceptualizations differ significantly, there is a common thread among them of approaching the embodied aspect of human experience and, in particular, incorporating the felt or affective dimensions. According to Willis and Cromby (2020), this “affective turn” signifies an interest in emotion and affect, and places these phenomena both in biology and language. Therefore, this theoretical perspective does not seek to replace the focus on language but instead aims to integrate the verbal and embodied components of human experience. Indeed, our experience is always embodied experience and meaning is continuously generated both through the body and language (Cromby, 2012).

Qualitative researchers have used an array of methods for collecting and analyzing the nonverbal and embodied components of human behavior and interaction. According to Seikkula et al. (2015), prosody-pitch, the rhythm and the timbre of the voice, body posture, gesture, and facial expression are some of the nonverbal or paraverbal elements that are vital to human communication and interaction. Other researchers have focused on mutual smiling episodes in a therapeutic context (Darwiche et al., 2008), on synchronized behaviors of participants, especially body movements (Seikkula et al., 2015) and on silences in people’s discourse (Mazzei, 2008). Furthermore, Gorden (1975) has developed a typology of nonverbal communication data. Some indicators for studying body language are kinesics (such as body movements and postures), proximity between people, temporal speech markers (such as silences and hesitation) and paralinguistic aspects (such as tenor, strength, and color of voice). Similarly, Ekman (1999) studied how emotions are expressed through facial expression in various social contexts, (McNeill, 1992) focused on different types of gestures and Onwuegbuzie et al. (2009) highlighted the meaning that can be communicated via gestures.

Consequently, researchers can use various frameworks for incorporating the embodied component of human interaction and communication. There are also numerous instruments that researchers can use to collect the data, such as audio and video recordings, photographs, checklists, matrices, and diagrams (Onwuegbuzie et al., 2009). More recently, partly due to technological advances, researchers have deployed physiological measures through the use of biometric devices and wearables (Cromby, 2012).

Sociodrama

A sociodrama is a dramatic play in which several individuals act out assigned roles for the purpose of studying and remedying problems in group or collective relationships. Its founder was Jacob Moreno, a social scientist who invented psychodrama and later sociodrama in an effort to explore his sociological interests while using psychodrama techniques. Sociodrama is an experiential group-as-a-whole procedure that can be used to explore social issues, social justice concerns, collective trauma, prejudice, stigma, interpersonal tension, intergroup conflict, healing, justice, reparation, and other social factors operating in complex human settings (Moreno, 1943; Marineau, 1989; Moreno et al., 2000). It can be used for exploration, engaging, problem solving, and teaching (Agazarian and Carter, 1993; Moreno et al., 2000). Sociodrama is suggested, and has been used, as a creative means for groups to work through political and social issues that concern them (Browne, 2005).

In the last decades, through the use of social theater (Boal, 1979; Saldaña, 2011) and other arts-based collaborative approaches (Conrad and Sinner, 2015), sociodrama, apart from an innovative method of qualitative analysis of social issues (Saldafia, 2003; Saldaña, 2011), has developed into a major transformative intervention for community development (Weiner et al., 2011; Conrad and Sinner, 2015).

Sociodrama is a methodology applicable to all sizes and types of groups, even to large groups. It starts with the present participants, in the here and now of the meeting. Instead of a group discussion a sociodramatic group demands individuals who are involved to engage each other in specific dramatic activities in order to discuss, explore and enact the possible solutions on issues that concern them (Kellermann, 1998). Its purpose is by clarifying, educating and energizing, to make people find their spontaneity and creativity, love and empathy inside themselves, with the ultimate goal of increasing internal control, balance and integration as controversies and hostilities erupt in the world, in a local and/or in a larger area. That is why the sociodramatic groups are groups of horizontal hierarchy, that is, the group process is cocreated by the group leader as well as by all the participants (Moreno, 1953; Agazarian and Peters, 1981).

Both verbal and nonverbal communication are involved in sociodrama. In verbal communication, thoughts, emotions or information are exchanged between individuals through the use of speech. Sociodrama also allows for a great deal of nonverbal communication that involves signals transmitted through facial expressions, posture, eye contact, gestures, tone of voice, body language, physical movements, space, touch, paralanguage, and other ways (Aguiar, 2001; Bargh, 2013).

Wearables in Research

Wearables or biometric devices have been used in various studies ranging from analyzing the impact of music on humans (Rickard, 2004; Kelley et al., 2014), collecting data at events such as rituals and spectacles (Konvalinka et al., 2011; Fischer et al., 2014), investigating consumers’ behavior (Chamberlain and Broderick, 2007) and exploring interactions and dialogues within psychotherapeutic settings (Seikkula et al., 2015; Kykyri et al., 2017). Many researchers have measured cardio-vascular activity, such as heart rate (HR), heart rate variability (HRV), and finger temperature (Kreibig, 2010; Seikkula et al., 2015).

Similarly, electrodermal activity (EDA) is often used to capture changes in skin conductance and galvanic skin response (GSR) (Sorinas et al., 2020). In addition, changes in blood pressure, as well as respiration rate and variability are common physiological responses measured by biometric devices (Calvo et al., 2010; Sorinas et al., 2020). HR and skin conductance are the measures most often reported in studies (Kreibig, 2010), and they have both been used with the purpose of capturing levels of arousal (Kreibig, 2010). There is also some evidence suggesting that changes in skin conductance reflect the general level of activation while performing a task, while HR depends more on the emotional response rather than on task requirements (Iffland et al., 2014).

The use of biometric devices in psychology and the social sciences relies on an underlying assumption that physiological measures capture bodily reactions that are not observable in visible behavior or in verbal accounts (Chamberlain and Broderick, 2007; Seikkula et al., 2015). Drawing on neuroscience research and theories of emotion, bodily changes and in particular physiological arousal are linked with the rising of emotions (Rickard, 2004). For example, EDA has been found to increase with most emotions (Kreibig, 2010). According to Rickard (2004), skin conductance is a more reliable indicator of emotions than language-based methods, such as self-reports. HR changes have also been strongly associated with different affective states and responses, such as anger, anxiety, joy, fear, and sadness (Chamberlain and Broderick, 2007; Kreibig, 2010).

In many recent theories of emotion, the autonomous nervous system (ANS) is perceived as a major component of emotional responses (Kreibig, 2010). The ANS is both an activating and restorative system. Specifically, it consists of two subsystems, the sympathetic nervous system (SNS), that increases activation and stimulates, and the parasympathetic nervous system (PNS), that has a restorative function. The ANS, also called an involuntary nervous system, responds to stimulus immediately, before conscious thought. Therefore, studying physiological reactions can provide valuable information for the affective dimension during social interactions (Seikkula et al., 2015). For example, when a stimulus is perceived as significant, there is a sympathetic activation of the ANS and an increase in physiological arousal occurs. Arousal is defined as the level of activation, ranging from total calmness to high alertness (Chamberlain and Broderick, 2007).

It should be noted that there is no preexisting and specific procedure or mapping when analyzing biometric data within the context of qualitative methodology. Furthermore, given the fact that biometrics measure arousal levels and no specific emotions, the data is analyzed in combination with qualitative observations. As mentioned earlier, this “affective turn” in research focuses on both the verbal and nonverbal elements of human experience, which that are perceived as occurring simultaneously and continuously (Willis and Cromby, 2020). Developing a method of mapping biometric data onto different layers of verbal and nonverbal data in order to prepare the ground for a form of combined analysis of several data layers is the goal and, we contend, the innovative contribution of this study.

Proposed Study and Methodology

The present interdisciplinary mixed-methods study explores the use of wearable technology and the importance of using bio-signals along with digital media and participant observation in the context of a multimodal and multisensory ethnographic inquiry interested in social issues relevant to the community of Eleusis. More specifically, the proposed methods and pilot study, aims to establish a qualitative and quantitative methods to be employed in an ethnographic study, where residents of Eleusis participate in sociodrama sessions in order to express themselves and deliberate on important social topics, such as the environment, unemployment and migration. The study is in line with an increasing eclecticism and creativity within multimodal research that is responsive to social issues, employing arts-based theatrical, performative as well as digital and biometric data for knowledge production and dissemination (Gergen and Gergen, 2011; Leavy, 2015; DeValiant et al., 2020). In particular, it the study reported here uses sociodrama as a “potential stage” for creating transformative research (Ius, 2020) and an active method for research related to the community and the appreciation of local collective knowledge (da Penha Nery and Gisler, 2019). It also adopts innovative methodologies that go beyond the realm of language and verbal descriptions, such as digital media (e.g., cameras, video) and wearable bio-signals (Willis and Cromby, 2020).

Study Design

The study aims to explore the perspectives of residents in Eleusis regarding their living conditions and their impact on their lives, at both individual and collective levels. The study design, incorporates sociodrama sessions, in which Eleusis residents are called upon to express themselves and deliberate as a group on the topics of environment, unemployment, and migration, that have been identified as central concerns for the Eleusis community. In line with multimodal research, the sessions are video and audio-recorded and bio-signals are obtained from selected participants, generating visual, audio and bio-signals. After preprocessing and synchronizing, the data are subjected to several layers of analysis, resulting in complex multilayered depictions of the patterns of expressions of Eleusis residents’ perspectives regarding living in the community of Eleusis. It is important to clarify that in article we contribute with the methods and a pilot study and we report on the technical and practical challenges, while the complete ethnographic research and interpretation of the verbal aspects is an ongoing process.

Epistemology

This study adopts a critical realist epistemological perspective. Critical realism is ontologically realist, assuming the existence of an external reality that is independent of human minds, and epistemologically relativist, acknowledging that different methods produce different perspectives on reality (Williams, 2018). Although it offers a general theoretical framework, critical realism is not linked to a specific methodology, hence it can inform a wide variety of quantitative, qualitative, or mixed-methods designs (Sturgiss and Clark, 2020). Moreover, as it focuses on understanding the process, how and why things happen, it is most appropriate for qualitative research approaches that explore the dynamic nature of social processes (Sturgiss and Clark, 2020). Moreover, critical realism uses a theoretical and researcher driven analytical process (Fletcher, 2017), which, in our study, fits the sociodrama process. At the same time, as we are interested in exploring participants’ experiences and perspectives on their conditions of living in Eleusis and the impact of this on their lives, we also adopt a phenomenological perspective. Phenomenological epistemologies focus on capturing and being true to participants’ experiential worlds (Willig, 2013). Combining a critical realist approach to knowledge production with a phenomenological focus on participants’ subjective experience is considered appropriate in qualitative research, as they are epistemologically compatible and share enough common premises (Willig, 2012).

Ethics and Quality Assurance

When it comes to complex, real-world contexts, mixed methods research offers a larger set of more flexible tools for dealing with ethical dilemmas than single-method approaches (Hesse-Biber, 2010; Preissle et al., 2015). Reflexivity as a methodological consideration is central to ethics in mixed methods research, along with positionality and efforts to minimize bias (Cain et al., 2019).

Study procedures are in accordance with the ethical standards of the responsible institutional or regional committee on human experimentation or in accordance with the Helsinki Declaration of 1975 as revised in 1983. Ethics approval was sought and gained from the Ethics Committee of Athena Research Center. ¹

Written informed consent, after full briefing regarding the requirements and ethical safeguards of the study, will be provided by each participant as a prerequisite to their participation in the sociodrama sessions. Files of the nonpseudonymised video, audio and bio-signals will be kept in an encrypted server, to which only the research team members will have access, following signing a confidentiality agreement. In-depth qualitative analysis will be conducted to pseudonymized data, after all identifying features have been removed. Only processed and pseudonymized data will be given to artists and used in research reports and academic publications. The study complies with the Data Protection Regulation (EU) 2016/679 of the European Parliament. Specifically, a detailed management and storage plan for all data is envisioned, including storage of data on encrypted and password-protected files, secured on University computers, defining persons responsible for data storage and management and regulating access to data by research team membership status.

Data quality and analysis may be a measure of ethics in mixed methods studies and, thus, should be addressed (Cain et al., 2019). We have designed data collection and analysis processes that endeavor to fulfill the quality assessment criteria for validity and reliability, as defined in both qualitative and (Lincoln et al., 1985; Walsh and Downe, 2006). More specifically, designing a multimodal study that incorporates several layers of qualitative and quantitative data within a multidisciplinary team requires intensive processes of building common ground, through familiarizing with other research approaches, translating concepts and tools to other research and disciplinary languages and building shared technical capacity in order to arrive at commonly agreed decisions and research protocols. Through systematic team work, triangulation and consensual validation processes have been adopted throughout the design, data collection and data analysis phases, and this we consider to be a major strength of this study. This collaborative process also ensures that the study fulfills the quality criteria of transparency, as all steps and aspects of the research process are continuously recorded and explained, transferability, as the knowledge acquired is already made relevant to different disciplines, as well as applicability, as the potential impact of the study is at the center of the team’s concerns.

Sample

Residents of Eleusis will be asked to participate in sociodrama sessions, with a minimum number of 20 adult individuals per group. In total, two groups per subject will be conducted, with a total number of approximately 120 participants. Regarding inclusion criteria, participants must be adult residents of Eleusis, who speak and understand the Greek language, as the sessions will be conducted in Greek. There will be no exclusion criteria. We aim to recruit a wide range of participants, with varied characteristics, in order to have a sample that will be as representative of the population of Eleusis as possible. This will be ensured through a broad dissemination of the participation call to the population of Eleusis through local agencies and through distributing participants in sociodrama sessions in a way that ensures participant variability within each session.

Data Collection Setting and Process

Data collection will take place in the framework of sociodrama sessions in Eleusis city in Greece, in a dedicated venue, in a big room with the capacity for allowing 20–25 people to comfortably walk and move in space. Sociodrama is an experiential procedure for social exploration characterized also as a “potential stage” for creating transformative research on social issues (Ius, 2020). Participants will be asked to work through issues that have been designated as topics of concern for Eleusis, namely environment, unemployment and migration. Each sociodrama session will be dedicated to one of the issues above, and will last for approximately one-and-a-half hours. The sessions will be facilitated by two experienced sociodramatists, members of the research team.

Prospective participants are aware beforehand that they are called upon to participate in a sociodrama session and the topics they will be asked to engage with during the session. Detailed information regarding sociodrama is given to participants beforehand and they are called to attend an introductory meeting with the other participants, if they wish. In the beginning of the session, the sociodrama facilitators introduce the rationale and process of the session and initiate activities, through which participants introduce themselves to one another and start getting a sense of belonging to a group. The facilitators then assign a series of roles to individuals and subgroups of participants, and instruct them to enact these roles in subgroup or whole group activities. Usually, participants alternate between roles and subgroups, so that they experience all the aspects of the dilemmatic topic under consideration. There is guidance on how to play a role; participants are encouraged to get into their roles and feel free to express them in any way that feels right to them. The detailed introduction to the setting, the initial team building exercises and the freedom in performing the assigned roles, all are designed to decrease performance anxiety, to foster ease and spontaneity, and immerse participants in the “here and now” of the enactment process. Moreover, it is important to mention that during a sociodrama we do not ask participants to act, but to participate in a specific task as if they were someone else and express their personal point of view. We clarify to them that they are not judged on how they perform as if they were theater actors but they are asked to act as they feel, expressing opinions that their role they play would support.

The main part of the session consists of several phases of role enactment and working through the designated issues. The session is concluded with participants’ reflections on the experience of participating in this process and the new perspectives on the debated topic that they have obtained through this.

Sociodrama provides an open stage and a procedure through which various collective dynamics can be played out. In the process, every participant has a voice as an individual and as a member of the group. Participants engage with the topic at hand through thinking, talking, feeling, moving and acting, allowing their thoughts, feelings and actions to unfold in a free-flowing way without having to follow strict rules. Participants may sit or stand and talk, they may move to parts of the space to engage in subgroup actions, they might move within the space while enacting a role and so forth Verbal, nonverbal and bodily signals, expressions and interactions all contribute to the unfolding process, and therefore have to be recorded.

Wearable, Audio, and Video Equipment

To record human and human body activity during each sociodrama session, three different types of equipment will be used: cameras, microphones and wearables to record physiological data and movement.

GoPro Hero4 Silver cameras will be placed around the room to record the activity of the crowd from a panoramic point of view utilizing the wide-angle-lens setting option. Also, two more cameras set on the regular-angle-lens (no wide-angle-lens) will be placed in locations that allow the recording of postures and facial expressions of participants. Figure 1 demonstrates the top view of the room and the placement of the equipment in it.

Figure 1. Top-view of the room, displaying the location of the cameras and microphones.

The sound will be recorded by all cameras. However, to achieve a better quality of sound for the thematic analysis that we describe in the sections below, we will employ Zoom microphones. The placement of the microphones can be seen in Figure 1.

The wearable equipment that will be used is two Heartypatch sensors from Protocentral, previously used in Kalampratsidou et al. (2019) and Kalampratsidou and Torres (2020). Heartypatch sensors record ECG (electrocardiogram) signal wirelessly. To achieve this, we place the sensor on the left side of the chest (by the area of the heart) of randomly selected sociodrama participants using nonreusable sticky gel electrodes and we turn the sensor on. The sensor is automatically connected to the WiFi networks that it is preprogrammed to connect; the PC should be connected to the same network. Then, we execute the Python code that we wrote to load and save the streamed data.

Recordings Synchronization

To synchronize the data recorded from the various devices, we will time-stamp all the data collected on the Heartypatch devices. To synchronize the audio and video data with the bio-signal recordings, one of the GoPro Cameras, named as main (see Figure 1), will be initiated through a Python script employing the Python module goprocam. ² In this code, the time that the recording starts will also be time-stamped and saved. Once we know the time that the main camera started and the times of the signals collected, we will be able to sync the rest of the cameras and audio data with the main camera by listening or watching the material, and therefore all audio and video data will be synchronized with the bio-signals.

In a secondary effort to have control of the device synchronization, during sociodrama one of the researchers, who is responsible for setting up the equipment, will time-stamp manually the times that each equipment piece is started as well as the timeline of the sociodrama and possible failures will be noted during the session.

Data Analysis

Different types of data from sound, video and bio-signals files will be analyzed comparatively and in conjunction. In the initial phase of the analysis, these different types of data files will be synchronized and will be coded for the presence of particular interactional and group process instances. Subsequently, particular interactional and group process episodes will be selected for further analysis. Selected fragments of specific episodes will be subjected to in-depth qualitative analysis by using the Video Annotation software program. Such software programs, such as ELAN by The Language Archive, Hyperresearch by Researchware Inc., and so forth offer the possibility to choose, code and analyze segments of video-recording, through inserting annotations on video episodes, transcribing and analyzing visual and verbal content. The software’s user interface intuitiveness is considered to be exceptional and easy to use. Particular attention will be paid to the coordination between different levels of participant expression—verbal, nonverbal and bodily—for each participant. We will also be analyzing how these levels interact at group level, in relation to happenings at the level of group interaction. Finally, we will attempt to associate these coordinated patterns of individual and group expression with the topics under discussion, in order to arrive at conclusions regarding how different aspects of living in Eleusis impact on its residents, both individually and collectively.

Once the analysis has been completed, the written result reports will provide accounts of the complex patterns of processes uncovered. At the same time, the pseudonymised extracts of raw data—video, audio, biometric, verbal—will serve as both testimonies and vivid enactments of the processes depicted through the analysis. In the first sense, as testimonies, the raw data extracts can be used to document and support the analysis, accompanying written reports of the study findings. As vivid enactments, on the other hand, they have an evocative power that goes beyond a proof of data. For this reason, selected extracts will be given to the artists to aid their attunement with Eleusis residents’ experiences and spark their inspiration for the music they will be called upon to produce.

Bio-Signals Processing and Metrics

To work with the EEG signal we first filtered it to remove instrumentational noise and then extracted the heart-rate signal. Also, the audio signal was employed which was examined as another bio-signal. The EEG signal was processed and plotted in Matlab R2018b. Raw ECG data were streamed from the Heartypatch sensor and saved on a Python interface developed by our team. The data were then filtered using a Butterworth IIR band pass filter for 5—30 Hz at second order, to remove any baseline wandering and to accurately detect the R-peaks. The range of the band pass filter was selected based on the fact that a QRS complex is present in the frequency range of 5—30 Hz. From the filtered EEG signal, the heart-rate was estimated by extracting the peaks (or beats) from the ECG signal and by counting the beats that happened within the past 1 min. This estimation repeats each time a new beat is detected. Therefore, we were able to generate a graph that demonstrates the change of the HR for every beat detected than the heart-rate updated every minute. The study of the audio signal was done through spectrograms. The spectrograms were generated with iZotope RX 7 Audio Editor software and visualize three parameters: time, loudness, and pitch. Time is measured in seconds and corresponds to the X-axis, loudness is measured in Decibels (dB) and corresponds to the Y-axis, and pitch is measured in Hertz (Hz) using the Mel scale and is represented by the color range. Lower frequencies are visualized in blue colors, mid frequencies in green colors, and higher frequencies in red colors.

Pilot Study and Outcomes

A pilot sociodrama session was organized with the goal of validating the methodology and identifying the practical and technical implications of incorporating wearables and using sociodrama as a setting for collecting multimodal data. For example, while sociodrama as a practice applies to tens or even hundreds of people, and can be held in an open space, the need to incorporate technologies poses certain limitations related to the need for protection of the devices, electricity, WiFi, placement of microphones and cameras. In parallel, the COVID-19 safety regulations created some extra restrictions in the availability of space and social distancing conditions. Finally, the setting had to be located in the city of Eleusis, for reasons of ecological validity and to be accessible to its citizens who will be called upon to participate in the sessions. Following consideration of a number of spaces that could fulfill these requirements, a large room at the “Workers’ Center of Eleusis” was selected for the pilot sociodrama session.

Pilot Sociodrama Experimental Session and Sample

The pilot sociodrama session was held in June of 2021 in Eleusis city, and was coordinated by two experienced sociodramatists (one male and one female), the same sociodramatists who are scheduled to coordinate the main sociodrama sessions of the project. As the pilot session had a preparatory goal for the main sessions/experiments, we did not involve Eleusis citizens as participants but it was held among the project team members. In particular, participants of the session were nine persons (four male and five female), with three of them working and living in Eleusis, one of them being a native of the city. The participants belonged to all three research teams that collaborated in the implementation of the “Transition to 8” project. Specifically, three participants from the psychology research team (National and Kapodistrian University of Athens), three participants from the technology research team (Athena Research Center) and three participants from the culture management team (Mentor in Culture & Heritage company). The session lasted 2 hr. In addition, three more persons were present during the session: one technology expert from the technology research team, who made sure that the equipment was working properly throughout the duration of the session, and two psychology experts from the psychology research team were observing the session and taking notes, acting as participant observers in ethnographic work.

The theme of the sociodrama session was the notion of place and our connection with cities in general and Eleusis in particular. During the session the participants had to enact particular roles that relate differently to events and facts about the city. In particular, three subgroups were proposed by the sociodramatists: (a) European Commission members, who decide that Eleusis is an appropriate candidate for a European Capital of Culture and announce their decision, (b) residents of Eleusis, who react to these news, and (c) citizens of other Greek cities, who react to these news. The nine participants were randomly divided into three groups and they were asked to enact these roles. All the groups tried all three roles and at the end of the sessions, during debriefing, they were asked to reflect on which role was more difficult and why.

Distribution of Wearable Equipment

Two participants were invited to wear the HeartyPatch Sensors. The selection was random and only after their approval we helped them place the sensor on their chest. In the manuscript, these two participants are referred to as P1 and P2.

Pilot Sociodrama Structure

The pilot sociodrama was divided into three main sections: “Introduction,” “Sociodrama,” and “Conclusion.” Each section contains several subsections, during which participants were directed to execute specific tasks. Below we demonstrate the sections and subsections as well as the tasks contained within them.

1. 1. Introduction—Engaging interaction

   1. (a) Walk and team questions: Participants were directed to walk in the room, then find the location of their research team, present how many they are and if someone is missing.

   2. (b) Maps: Participants imagine that they walk on a map and position themselves depending on the answers to the following questions:

      1. (i) Where do you live?

      2. (ii) Where is the place of your origin?

      3. (iii) What is the place that has impacted you culturally?

         Note: Before each direction participants were asked to walk until they were given the direction. Then they had to place themselves in the map based on the direction and stand there until all participants present the choice of their location.

   3. (c) Project connection: Participants were asked to touch the shoulder of the person who connected them with (or brought them in) the project

   4. (d) How much do you know Eleusis?: The sociodramatist specifies that a book in the middle of the room represents Eleusis, then she directs the participants to place themselves in a distance that indicates how much they know about Eleusis.

   5. (e) Perspective toward Eleusis: Using a book to represent Eleusis, participants are asked to walk in space at various distances to Eleusis and express in words and with their body posture their perspective toward the city.

   6. (f) Creating the subgroups: The sociodramatist randomly splits participants in three subgroups

2. 2. Sociodrama—Role playing

   1. (a) Role 1: Greeks (non Eleusinians): The Greeks (all Greek population except Eleusinians) respond to the news that Eleusis will be a European Capital of Culture for 2023.

   2. (b) Role 2: Eleusinians: Eleusianians learn that their city is going to be European Capital of Culture for 2023.

   3. (c) Role 3: European Commission: The European Commission decides the city that will be European Capital of Culture each year. Eleusis was selected among other candidates as European Capital of Culture for 2023 and the Commission should announce the decision.

      Note: Roles 1, 2, and 3 are listed in the sequence that were performed by P1 (and her team, which includes P2), whose bio-signal data will be discussed later. As it was already mentioned, all the groups tried all three roles and of course the sequence of the roles differs from group to group.

3. 3. Conclusion—debriefing

   1. (a) Walk and talk to Eleusis: A book in the middle of the room represents Eleusis and participants are directed to approach Eleusis and talk to the city.

   2. (b) Discussion: All participants, sociodramatists, observers and the person responsible for the equipment, discuss their experience of participating in, coordinating or observing the sociodrama session.

   3. (c) Circle of trust: Sociodramatists invited all participants to join the circle of trust and after the end of the session take with them anything they enjoyed during the session and leave in the middle of the circle what they wish to leave aside.

Coordination of Qualitative and Quantitative Data

In the following section, we provide the details of a preliminary synchronization of data and data segmentation. In particular, we attempt data segmentation for synchronization and analysis in two different ways: firstly using bio-signals as a guide and secondly using audio data.

Video data annotation and timestamp extraction

ELAN is a tool that enables video and audio annotation. In this work, ELAN v6.0 was used to synchronize the video and audio data recorded on the different devices. The recording of the main GoPro camera was used as a point of reference. Next, we annotated the beginning and ending of all the tasks performed during each subsection. Examples of annotated sections and subsections can be seen in Figures 2–4.

Figure 2. ELAN annotation of the main section “Introduction” and its subsections. The total duration of the introduction was 30.83 min.

Figure 3. ELAN annotation of the “Sociodrama” section and its subsections. The sociodrama lasted 33.82 min.

Figure 4. ELAN annotation of the conclusionary section, which lasted 30.55 min.

The timestamps of the annotated sections are then extracted in milliseconds—among other options—in a CSV file format, as Table 1 demonstrates. The timestamps were transferred in Matlab or other tools, where the bio-signal processing and observation took place. This step enables the synchronization of the quantitative and qualitative data, as well as the flexible move from one data-type to another for transferring and exchanging information.

Table 1. Times of main sections in milliseconds

Episodes localization through heart activity

After having specified the times and duration of each section, it is now possible to parameterize the bio-signals based on the section they belong. The ECG data of P1 is displayed in the upper plot of Figure 5. The signal was processed and plotted in Matlab R2018b. Raw ECG data were streamed from the Heartypatch sensor and saved on a Python interface developed by our team. The data were then filtered using a Butterworth IIR band pass filter for 5–30 Hz at second order, to remove any baseline wandering and to accurately detect the R-peaks. The range of the band pass filter was selected based on the fact that a QRS complex is present in the frequency range of 5–30 Hz. Figure 5 displays data from the “Introduction” section and its specific subsections “Walk and questions,” “Maps,” “Project connections,” and “How much do you know Eleusis” as an example of data color-mapped based on the corresponding activity section they belong. In the figure, the first approximately 5 min colored black are the data collected before the Pilot Sociodrama session started. The zero values were added to synchronized ECG with the video data of the main camera that was used as a point of reference to sync the data from the different equipment pieces. Therefore, at time 0 min the main camera started recording, the different equipment pieces followed up, at 4.17 min time the Heartypatch sensor started recording and the pilot session started at 4.50 min.

Figure 5. Segmentation of filtered ECG signal and the corresponding heart-rate based on the timestamps extracted from the ELAN software.

Then, the heart-rate is estimated by extracting the peaks (or beats) from the ECG signal and by counting the beats that happened within the past 1 min. This estimation repeats each time a new beat is detected. The outcome of the introductory section is presented in the lower plot of Figure 5.

Observing the HR fluctuations, it is easy to notice that there are specific time-zones during which high heart-rate values appear. For example, during the task “How much do you know Eleusis?” around 18–19 min, we can see the repetitive presence of high heart-rate values within the task. Therefore, the maximum heart-rate value was extracted for each task and its corresponding first and last time of appearance. The outcomes of these estimation are presented in Figure 6, where the blue vertical line defines the first and last appearance of max heart-rate value and the shaded area in between the duration of the time-zones (Table 2).

Table 2. Timestamps of the heart activity episodes extracted based on the max heart-rate value

Note. The table displays the start-time, end-time, and the duration of the episodes.

Figure 6. Visualization of the volume and time of the first and last max heart-rate value (blue lines) for each subsection. The shaded area between these two line indicates the duration of the occurrence.

The extraction of the times of these episodes, presented in Table 2, enable the transfer of the information to ELAN in order to annotate the media data (video and audio) and observe the physical activity that was taking place between the participants during and around these time-zones.

In this work, episodes refer to any specific pattern or outstanding behaviors that can be noted in either bio-signals or media data. By extracting their time and duration, we can then move to a different data type and observe what is happening during, before and/or after the episode in other data-types. Figure 6 and Table 2 demonstrate heart activity based episodes and specifically the time-zones that are characterized by maximum heart-rate values. In the following section of the manuscript, we will observe in ELAN the physical activities that are taking place before and during these episodes.

Observing the physical activity of heart extracted episodes in video data

By extracting the time-stamps of the heart episodes, it was possible to go back to ELAN and annotate the media files with the episodes. Then, we were able to observe in the media files what was happening during the time-zones of high heart-rate values but also what were the events before that caused the heart-rate increase. Thus, below the description of “before” and “during” each episode are listed.

1. A. Walk and questions

   Before: Starting at 00:06:45 the sociodramatist asked P1’s research team how many members were in their team and how many were missing. Figure 7 presents a video-frame of this interaction. During: The team started talking about the members and their participation.

   

   Figure 7. (a) P1 (indicated by a yellow dot) is with her research team and they communicate with the sociodramatist (indicated by a green dot) the participation of their team a few seconds before the high heart-rate is reached. P2 (indicated by an orange dot) is also part of the same research team. (b) P1 and P2 are touching their advisor and communicating with the sociodramatist before the high heart-rate (episode) of “Project connection” is reached.

2. B. Maps

   Before: P1 is walking around the room to finally move toward the place of her origin.

   During: P1 places herself on the imaginary map at the very first millisecond of the episode and stands there for the remaining time.

   Note: In this task, three mapping tasks were given (a) “where do you live?” (b) “where is the place of your origin?” (c) “what is the place that has impacted you culturally?”. Before each task was given, participants had to walk around in the room. Therefore, in this subsection we observe three main fluctuations of the HR.

3. C. Project connection

   Before: P1 is again with her research team and touching her advisor—as all participants were directed to touch the person who connected them with the project. Sociodramatist asks P1 and P2 about their connection to the project, they both confirm it was their common advisor. “Would you like to say something to your advisor?,” the sociodramatist asks. P1 and P2 pause for a few seconds, then they both agree that they want to thank her. “Would you like to respond to them?” the sociodramatist asks the advisor. “I want to say thank you as well,” the advisor responds and they all burst out laughing at 00:16:36.000. Figure 7 demonstrates a video-frame of this interaction.

   After: Another research team talks about who connected them to the project and how they joined.

4. D. How much do you know Eleusis?

   Before: All participants are walking in the room.

   After: Sociodramatist gives the task “I want you stand in a distance from Eleusis (represented by a book in the center of the room) that demonstrates how much you know the city.”

In two out of four cases, the HR increase seems to be connected with the walking task that was executed right before the max HR values are reached. For the other two subsections, “Walk and questions” and “Project connection,” that were characterized by the observers as emotional periods, the start and end time of activity that took place before the episodes were extracted (see Table 3). In the next section, the audio data of these two sections were investigated to “see” beyond what we could hear.

Table 3. Parts of the video empirically identified as before-episode activity

Observing the heart extracted before-episodes in audio data

From the audio files, the before-episode parts were exported and the spectrograms of those were further studied (see Figures 8 and 9). The spectrograms were generated with iZotope RX 7 Audio Editor software and visualize three parameters: time, loudness, and pitch. Time is measured in seconds and corresponds to the X-axis, loudness is measured in Decibels (dB) and corresponds to the Y-axis, and pitch is measured in Hertz (Hz) using the Mel scale and is represented by the color range. Lower frequencies are visualized in blue colors, mid frequencies in green colors and higher frequencies in red colors.

Figure 8. Spectrogram of the before-episode audio recording of subsection “Walk and questions.”

Figure 9. Spectrogram of the before-episode audio of subsection “Project connection.” At 23” a collective burst-out of laughter corresponds to a significant increase in loudness and pitch. From 9 to 14 s, there is a pause in conversation.

Figures 8 and 9 demonstrate an overall fluctuation in loudness and pitch during discussion. Specifically, Figure 8 demonstrates a continuous conversation without any interruptions. In contrast, Figure 9 demonstrates conversation with interruptions. In particular, in the period 9–14 s, we observe the pause, in the segments with the absence of magenta color, during the conversation between sociodramatist, P1, and P2 that was described in the previous section. Also at second 23, we observe a significant increase in loudness and pitch that corresponds to a collective burst-out of laughter and is visualized by the density of different colors.

Episodes localization through audio data

As mentioned earlier, in parallel with using HR as an index for the selection of episodes for further study, we adopted an alternative strategy of using audio data for the same purpose. This is part of a preliminary investigation of possible modes of selecting data segments, that we call “episodes,” for further synchronization and analysis, in order to determine which data modes bear more fruitful results. The audio data of the whole pilot session is used for segmentation and feature extraction. It is important to note that the audio data comes from the group sounds, in contrast to the heart data that corresponds to P1. So in this section the episodes are extracted not from a personalized but from a global measurement.

Figure 10 demonstrates the group audio data of the whole pilot session and the estimated root mean square (RMS) value of the audio volume of each subsection. It is noticeable that the “loudest” subsection is the third round of role playing, reaching as high as 0.1758. Role playing 1 and 2 and “the circle of trust” follow with approximately similar values (0.1259, 0.1266, and 0.1294 respectively). It is also important to observe the subsections with minimum values. In this case, first comes “How much do you know Eleusis?” with an RMS level of 0.0803, then follows “Walk and questions” (RMS = 0.0864) and “Walk and talk to Eleusis” (RMS = 0.0869).

Figure 10. The audio data of the whole pilot sociodrama session, colored based on main section segmentation (bottom graph). The top graph demonstrates the root mean square (RMS) of audio volume (y-value) for each subsection in sequence, whereas the x-value corresponds to the time that the subsection begins. These time values are highlighted with black lines in both bottom and top graphs. The text above the dots indicates the name of the subsection that the value corresponds to.

In the next section, we will define as audio based episodes the min and max RMS volume as extracted by the pilot session. As a result, through ELAN video annotation, we will observe the corresponding sections “How much do you know about Eleusis?” and “Role 3.”

Observing the physical activity of audio extracted episodes in video data

After extracting the audio based episodes, we move back to ELAN to observe what is happening in each of the two subsections.

1. A. “Role 3”—the loudest subsection Subgroup 1 is the European Commission (P1 and P2 are members of this sub-group). Initially they hold an internal meeting to decide about the European Capital of Culture. They decide for “Eleusis.” Then, P1 announces the news to Eleusinians, sub-group 2. The latter make a lot of noise and applaud to celebrate the event. One of the sub-group members enthusiastically announces the news to the rest of the Greeks, sub-group 3 (Figure 11a). But they do not respond as expected; some members feel that their cities are degraded, some others cannot believe their ears. The European Commission sub-group is surprised: “Why are those people complaining?”. Disagreements expand among participants and between the subgroups (Figure 11b).

   

   Figure 11. Moments from “Role 3.”

2. B. “How much do you know about Eleusis?”—the quietest subsections In this subsection, the participants are directed to move to a distance from Eleusis (represented by a book placed in the center of the room) depending on how much they know about the city. The more they know the closer to the city they can move. Then, the sociodramatist asks each participant to express briefly what they would like to know about Eleusis. One after the other participants respond to this call. Some participants become very emotional when it’s their turn. For some of them Eleusis was their birthplace, for others the city of residence, but not for all of them.

Discussion

One of the main challenge we are addressing with this pilot study (section “Pilot Study and Outcomes”) is to test the proposed methods (described in section “Proposed Study and Methodology”) before conducting more sessions with citizens of Eleusis in the future. We focused on how to align the different channels of information and how to find a common vocabulary and tools for transdisciplinary work. So far we achieved to organize a workflow that allow us to go deeper into the analysis qualitatively and quantitatively and hopefully explore what are the pros and cons of each one of the instruments. In general the video seemed to work as a spinal core for aligning the different information and the timestamps of the video provide reference points in order to “zoom in and out” within the sociodrama session. While in this work we do not yet interpret the verbal part of the process, which is an ongoing process, we expect that the thematic analysis and biosignals will work as complementary to identify “important moments” during the session (Willis and Cromby, 2020). In what follows we reflect on the first outcomes from the pilot study which applies the proposed methods and tackle a number of technical challenges and epistemological decisions.

Outcome Discussion

The results of the work, presented in the previous section, demonstrate a continuous transfer from one data-type to the other to observe how actions detected in one data-type are coordinated with the others. We started by synchronizing all data (video, audio, and bio-signals), which was a key point of our analysis. Then, the data were annotated in ELAN by watching the videos. The initial annotation was based on the main sections of the session (“Introduction,” “Sociodrama,” and “Conclusion”) and the tasks that were given in each one of them. The exact description of the main sections and their subsections are presented in section “Pilot Sociodrama Structure.” Transferring the times of the annotation to the ECG signal (Figure 5), we could extract the heart-rate variability within each task. In this manner, the times of max heart-rate values were extracted to annotate the beginning and end of heart based episodes (Figure 6). The corresponding video data were then watched, to only realize that it may be more meaningful to watch what happened before the episode and as a consequence caused the episode. In this work, we empirically defined the beginning of the before-episode section, by watching the videos. Alternatively, we could use the time of the local min heart-rate (appearing right before the max heart-rate) as a more mathematical method. After the before-episode time segments were extracted, their corresponding audio signal was further evaluated. In Figures 8 and 9, we were able to scrutinize how silent moments of social puzzlement, burst-out of laughter and regular talk differ in pitch and loudness.

In a different sequence of observational analysis, the initial annotation times were applied in audio data (section “Episodes localization through audio data”). In this case, the RMS of the audio volume for each subsection of the pilot session was estimated. The outcomes presented in Figure 10 displayed the loudest and quietest tasks. This indicator was used as another episode extractor, this time based on audio data. The audio episodes lead the researchers back to ELAN to elaborately examine the two extreme cases. The loudest subsection was “Role 3.” We could deduce that, after the experience of playing “Role 1” and “Role 2,” the team was possibly feeling more comfortable to enjoy the task and better argue for their roles. As such, the voices’ volumes were raised and the disagreements were more freely expressed, in some cases not even caring if they were talking over each other. In contrast, during the quietest subsection, “How much do you know Eleusis?,” participants were talking one after the other and for several participants expressing how little they feel they know their own hometown or place of living was quite emotional. Eleusis is a city with a very long history, from the ancient Eleusinian Mysteries to today’s sociopolitical issues, there is a lot that we need to understand and to learn. The more you know about a place the more you respect it… In the subsection “How much do you know Eleusis?” P1 talked from 00:21:10.000 to 00:22:05.00. Going back to Figure 6, we can see that during this time there appears a local maximization of her heart-rate; however it is not significant enough to be detected as the max heart-rate of the subsection.

Overall, two different storylines were explored, one initiated by the ECG data and the other one by the audio data. Apart from the separate information that each storyline has revealed, it is important to emphasize that the ECG storyline unfolds through the lenses of P1, whereas the audio storyline has a holistic approach, since it records behaviors of the whole group. Therefore, in the latter case, we cannot have an understanding for each participant separately, unless bio-signals become part of the storyline. In this case, the more participants’ bio-signals are collected the better understanding of the group versus individuals will be gained. Moreover, this will enable comparisons between individuals. For example, given a situation of group conflict, it could enable the examination of how each participant experienced the conflict.

Indeed, we could come up with many possible ways of moving from one data-type to the other, since all reasonable storylines carry valuable information. Also, the more data types we have the more variations of storylines we can create. More data types could mean either recording more participants or collecting more types of bio-signals. The latter is further analyzed in section “Wearability and invasiveness of the wearable equipment.” Moreover, different metrics used to extract episodes could lead to the creation of storylines variations.

Video Segmentation and Episode Extraction in the Study

The pilot session lasted 2 hr and resulted in video, sound and bio-signal (HR) data as well as notes from the observers. If this process is going to be repeated for various groups, this multiplies the amount of data to be analyzed; for both technical and practical reasons, segmenting the data or choosing timeframes that deserve more attention provides a solution to handling this complex and multilayered data.

In this work, we considered two options: (a) segmenting the material based on the sociodrama main sections and subsections and (b) segmenting the material based on Episodes. The first option is more straightforward, as it is based on obvious, distinctive preplanned changes based on the directions given by the sociodramatists. Locating Episodes is more complex, as it is a question of finding “when a psychologically or socially important event occurs during the session.” Moreover, decisions on the criteria for locating Episodes are theoretically driven, and this has advantages and disadvantages for an openended exploratory study, such as this one. It is likely that a segmentation of data on the basis of sections and subsections is a useful strategy for the original processing of the data, while a segmentation on the basis of episodes is more appropriate in the phase of analysis-proper, when specific explorative research questions will have been formulated.

It might be already clear that the detection of episodes could happen in various ways and that it is definitely not a strictly defined process. Episodes could be created by applying a mathematical function in bio-signals, such as extracting the times of max or min values of the parameter studied or by estimating the RMS or average value of a particular set of data, to name some examples. Episodes could also be indicated through the audio-visual data. For instance, in this work, we specified the beginning of the before-episode segments by watching the video data. Similarly, we could detect all moments of walking, laughing or yelling in video data and then transfer the information to bio-signals to study those time-segments. Finally, episodes could also be determined by the observers. Observers have a unique perspective on the process, as they watch the sociodrama session live. This means that they sense any emotional bursts, frustrations, discomforts but also joy and engagement. This is a layer of understanding caused by the presence of “warm bodies” that is missing from just watching the video. Therefore, we consider the input of observers quite essential; and although we did not utilize their input in this manuscript, we plan to include it in future publications.

Another aspect of episodes is that sometimes they are just an indication of something that has already happened or will happen. Therefore, examining the before- and after-episode time-segments is often crucial in order to better understand the episode. For example, when the times of max heart values were extracted, we noticed it was important to see what happened before in order to reach max heart-rate. However, if the research team had chosen to define episodes by detecting the times of min heart-rate, then they might have realized that both before- and after-episodes are vital.

The before- and after-episodes time-segments could be defined either empirically by watching the audio–video material (as it was done for the present work) or using a mathematical equation. As such, in the instance of max HR defined episodes, researchers could use the time of min heart-rate value appearing right before the max value defining the episode to set the initiation of the before-episode. To draw any valid conclusions, we need to systematically check all elevated, and reversely low, HR episodes and investigate how they relate to other layers of verbal, nonverbal and bodily data, in order to draw patterns of correspondence between different forms of data.

The annotation and segmentation of video or other recordings, in this sense, when it comes to embodied experience can be seen as additional tools to an ethnographic approach (El Raheb and Ioannidis, 2021), combined with more traditional instruments that help researchers understand “what happened” on-site (Issari and Pourkos, 2015). In the case of sociodrama, the recordings of both the video and wearables in this mixed-method methodology provide additional datasets for analysis and interpretation as means of “shedding light” from different perspectives, toward understanding the social and individual experience, which is by nature subjective and complex. In our future work, we plan to study how these episodes, that are identified by any of the aforementioned means, are perceived by the participants themselves.

Challenges and Future Work Improvements

Balance between quantitative and qualitative analysis

One of the biggest challenges that our approach faced is maintaining a connection between qualitative and quantitative analysis. In our opinion, a successful holistic approach requires the synchronous utilization of both analysis methods but also the possible exchange of information between them. To achieve this, it was important to enable the synchronous recording of all equipment utilized. This way, it was later possible to move from an event that was observed in video and/or audio data at a specific point of time to the corresponding bio-data that were recorded at that time.

Wearability and invasiveness of the wearable equipment

Various signals can be recorded nowadays, using a variety of sensors; and the more invasive the experimental task allows you to be, the more options you have. Thus, someone could use a full-body costume to capture position data utilizing a high-end motion-capture system or choose an inertial measurement unit (IMU) to capture the movement of a single body part, such as the thorax. Similar limitations apply when recording an electroencephalogram (EEG), which is a piece of expensive and quite obstructive equipment. As a result, there is a trade-off whether you need more data or less invasiveness and more naturalness in your task. In a study investigating the social behavior of a group of people, there are a lot of bio-signals that could carry fruitful information, however allowing natural interaction is vital. For example, heart activity carries information about the calmness of the participants and their physical participation in the task. The ECG equipment used in this study was not obtrusive nor obvious to other participants. In fact, it was not possible to visually understand who was wearing the equipment in the course of the sociodrama session. The use of unobtrusive equipment increases the naturalness of the interaction and thus the ecological validity of the data obtained that is a primary concern when trying to understand social interaction.

Other physiological signals that could be utilized without being very invasive are GSR, which indicates psychological arousal (Kreibig, 2010; Iffland et al., 2014; Sorinas et al., 2020), and body movement, which includes various information, from the horizontal transition of the human body in the space to the participation and engagement of a specific body part or the full body in the social interaction

Collecting bio-signals from a group

Collecting and studying several bio-signals during a task could enable the investigation of different layers of the nervous system, from the central nervous system (CNS) through ECG data analysis to Peripheral Nervous system (PNS) through movement data analysis. Another perspective is studying the Autonomic versus Volitional Nervous System. Of course, the more data we collect from the potential group of people that participates in the task, the more understanding for the whole and the parts of the group we could gain. However, collecting various bio-signals from several participants is a challenge. This is because when studying a group of people there is not only the need to synchronize the recordings of the different sensors-possibly with different sampling rates- of one person but of all people, as well.

Luckily nowadays there are single equipment units that record more than one physiological signal simultaneously, such as Feel by MyFeel, all sensors of Shimmer, E4 by Empatica, Emotibit, Opal by APDM (previously used in Kalampratsidou and Torres (2016). These technologies demonstrate solutions, that solve the issue of synchronizing the recording of various sensor-types in one person (although the different sampling rate per sensor-type may still be a fact). Synchronizing the recordings of several people may still be an issue to consider depending on the brand. In most cases though, the time-stamped data will solve the problem, if the corresponding software enables the parallel recording of various pieces.

Taking all these into consideration, Shimmer3 GSR+ is the equipment that we plan to use in our future sociodrama sessions. Shimmer3 GSR+ enables the recording of GSR, which is connected to emotional arousal (Sorinas et al., 2020). It also registers PPG (Photoplethysmography) signal which demonstrates volumetric changes of blood in peripheral circulation and heart-rate can be estimated. Finally, it records nine DoF inertial sensing, which corresponds to linear and angular acceleration and speed of body movement (Kalampratsidou and Torres, 2019) presents methods of studying audio data in relation to obscure patterns of walking that could be adjusted and employed in our future study). The corresponding software, named Consensys, enables the real-time streaming of the data from up to seven sensors through Bluetooth connection or the storage of the data to the SD card of the sensor and their later on extraction, which is what we plan to use in our set-up. The sampling rate of the sensors can be set manually through the software and the battery of the sensors can last long enough to cover the needs of the session. For all these reasons we settled on employing several Shimmer3 GSR+ sensors for the final study.

“Directing the stage” of the session

Another need that emerged from the pilot sociodrama was that of “directing the stage.” The placement of the cameras and the microphones should be related to the placement of the teams during the main sociodrama. Therefore, the sociodrama facilitator should guide the teams on the right location, so that they are close to the corresponding microphone, as Figure 1 demonstrates. Also, all cameras (and particularly the nonwide ones) should be placed so that they best record the faces of the participants without intervening. Lastly, if there are not enough wearable pieces for all participants, then the participants who will wear the equipment should be either randomly invited or they should be discretely appointed by the sociodramatist if they play a key role in the session. In either case, we need their consent before we help them put the equipment on.

Collaboration Needs

Another point that seems important for this mixed methods study is the need for tools for online synchronous and asynchronous collaboration within the interdisciplinary research team for the phases of “codifying,” analyzing and interpreting the data. For instance, in this pilot work the video annotation, the segmentation of the sociodrama sections, and the episode extractions were technically executed by the members of the research team with more technical background, following notes and discussion among the whole team, including the sociodramatists and psychologists who were less familiar with the technicalities of the ELAN software. Alternatively, psychologists could possibly annotate the data using different methods or interfaces that require basic or no technical knowledge, taking into account participant observation note-taking by hand and the overall analysis of verbal data. During the preparation and organization of the material, we realized that reaching a common vocabulary and understanding through a single software that enables synchronous and asynchronous collaboration is critical. For example, during this work ELAN software was used to bridge the gap between audiovisual and physiological data transitions. However, this multimodal approach introduces new interfaces that the whole team (with computing or no computing background) needs to be familiar with and work directly on them. Finding a common ground where the outcomes of each research perspective could be discussed is a challenge for this project but is important for the future of the scientific community. This point also suggests the need for collaborative, web-based tools for annotation and multimodal synchronization that keep a balance between technical specifications and complex functionalities with intuitive interfaces for researchers with nontechnical background.

Considerations and Limitations

At this stage, it is important to discuss some considerations and limitations about the interpretation of bio-data. Bio-signals offer an opportunity for researchers to capture aspects of physiological activity that are not visible in participants’ behavior and verbal accounts or may even contradict verbal communication. However, the interpretation of these data should be taken with caution and be characterized by transparency. As we have already discussed, bio-data is used in combination with other data, such as behavior, movement, language and other qualitative observations. For example, if bio-signals indicate high arousal for a participant, we can interpret this arousal as a specific emotion only if this is supported by the participant’s narrative and visible behavior. At the same time, various other aspects that might impact physiological arousal are taken into consideration when interpreting the data, such as the impact of physical activity and movement during sociodrama and potential nervousness of speaking or performing in front of a group. In addition, one thing that needs to be taken into account, in is the cost of learning and investing in the method and material and how this balances with the added value on the interpretive level.

Conclusion and Future Work

In this article, we present a mixed-method approach for multimodal research in embodied psychology that incorporates wearables for monitoring HR activity, video, and audio recordings during sociodrama sessions. We validated this approach through a pilot session in Eleusis city and reflected on the outcomes, that we analyzed by synchronizing the data through the ELAN annotation tool. With the aim of contributing to the field of nonverbal communication and embodied social experience research, we discuss the opportunities and challenges that we have identified during this research process based on literature research and interdisciplinary experimentation.

The collection of the research data through sociodrama sessions and the incorporation of the wearables has shown promising first results for multimodal research. A number of challenges are discussed, such as technical challenges and operational requirements regarding the setting, for example, balancing needs of recording with an ecological setting. Nevertheless, a number of opportunities emerged by the incorporation of the wearables’ recordings and analysis as complementary with traditional instruments of ethnographic qualitative analysis. The wearables can shed light on different aspects of the individual and collective experience of the sociodrama, for example, identifying tensions in the body of the participants during silent moments. Using such equipment, it would also be interesting to study the interpersonal reactions. For example, if a participant shows an emotional arousal peak in GSR at a moment or an interval, we could then examine how the bio-signals of the participants interacting with that person are impacted. These kinds of emerging questions highlight not only the potential of incorporating wearables in sociodrama for embodied psychology research, but also the need for an iterative, interdisciplinary and practice-based process toward developing and optimizing the methodology for such research.

In this article, we have synchronized and analyzed the signals of video, audio and heart-rate, and we have provided an insight for the analysis of the episodes. In our future work, which includes sociodrama sessions with the citizens of Eleusis, we aim to incorporate wearables for capturing additional biophysical signals that can detect emotional arousal, such as galvanic skin reaction or movement using Shimmer and to ask more participants (up to six persons simultaneously) to wear the equipment, so that we can study individual versus group reactions. We also consider signals that might be useful, such as breathing, although we are concerned about the invasiveness of the equipment. In addition, we aim at the multimodal exploration of more episodes within a session; and taking into account field notes collected by the participant observation. Finally, our future plans include combining analysis of nonverbal signals and qualitative analysis of the transcribed conversations, as well as comparing the multimodal recordings of episodes with the perceived or verbally expressed views of the episodes by the participants. Overall, wearable technology opens up new challenges and can play an important role in the context of multimodal and multisensory research in social sciences permitting for richer insights (embodied aspects included) regarding psychosocial life. It also calls for collaborative models of constructing and reporting analyses as well for presenting research in a meaningful manner for research audiences.