12 - Automatic Analysis of Bodily Social Signals

Summary

The human body plays an important role in face-to-face interactions (Knapp & Hall, 2010; McNeill, 1992). We use our bodies to regulate turns, to display attitudes and to signal attention (Scheflen, 1964). Unconsciously, the body also reflects our affective and mental states (Ekman & Friesen, 1969). There is a long history of research into the bodily behaviors that correlate with the social and affective state of a person, in particular in interaction with others (Argyle, 2010; Dittmann, 1987; Mehrabian, 1968). We will refer to these behaviors as bodily social signals. These social and affective cues can be detected and interpreted by observing the human body's posture and movement (Harrigan, 2008; Kleinsmith & Bianchi-Berthouze, 2013). Automatic observation and analysis has applications such as the detection of driver fatigue and deception, the analysis of interest and mood in interactions with robot companions, and in the interpretation of higher-level phenomena such as mimicry and turn-taking.

In this chapter, we will discuss various bodily social signals, and how to analyze and recognize them automatically. Human motion can be studied on many levels, from the physical level involving muscles and joints, to the level of interpreting a person's fullbody actions and intentions (Poppe, 2007, 2010; Jiang et al., 2013). We will focus on automatically analyzing movements with a relatively short time scale, such as a gesture or posture shift. In the first section, we will discuss the different ways of measurement and coding, both from motion capture data and images and video. The recorded data can subsequently be interpreted in terms of social signals. In the second section, we address the automatic recognition of several bodily social signals. We will conclude the chapter with a discussion of challenges and directions of future work

Measurement of Body Motion

Body movement can be observed and described quantitatively, for example, in terms of joint rotations or qualitatively with movement labels. While social signals are typically detected and identified as belonging to a certain category, body motion is typically described quantitatively. Therefore, the detection of bodily social signals is often based on a quantitative representation of the movement. From the perspective of computation, body motion is most conveniently recorded and measured using motion capture (mocap) devices. However, their obtrusive nature, cost, and the fact that they typically cannot be used outside the laboratory has limited their employment.