Automatic emotion recognition has widely focused on analysing and inferring the expressions of six basic emotions – happiness, sadness, fear, anger, surprise, and disgust. Little attention has been paid to social emotions such as kindness, unfriendliness, jealousy, guilt, arrogance, shame, and understanding the consequent social behaviour. Social context plays an important factor on labeling and recognizing social emotions, which are difficult to recognise out of context.

Social emotions are emotions that have a social component such as rage arising from a perceived offense (Gratch, Mao, & Marsella, 2006), or embarrassment deflecting undue attention from someone else (Keltner & Buswell, 1997). Such emotions are crucial for what we call social intelligence and they appear to arise from social explanations involving judgments of causality as well as intention and free will (Shaver, 1985).

To date, most of the automatic affect analysers in the literature have performed one-sided analysis by looking only at one party irrespective of the other party with which they interact (Gunes & Schuller, 2013). This assumption is unrealistic for automatic analysis of social emotions due to the inherent social aspect and bias that affect the expressiveness of the emotions in a social context or group setting. Therefore, the recent interest in analysing and understanding group expressions (e.g., Dhall & Goecke, 2012) will potentially contribute to the progress in automatic analysis of social emotions.

Recent developments in social media and social websites have opened up new avenues for the employment of user-driven and user-generated emotional and affective tone such as amused, touched, and empathy in social interactions. Accordingly, a number of researchers refer to automatic analysis of social emotions as ‘social affective analysis’ (e.g., social affective text mining) (Bao et al., 2012). Such works have focused on automatic prediction of social emotions from text content by attempting to establish a connection between affective terms and social emotions (Bao et al., 2012).

In this chapter we want to first provide a short introduction into the “classic” audio features used in this field and methods leading to the automatic recognition of human emotion as reflected in the voice. From there, we want to focus on the main trends leading up to the main challenges for future research. It has to be stated that a line is difficult to draw here – what are contemporary trends and where does “future” start. Further, several of the named trends and challenges are not limited to the analysis of speech, but hold for many if not all modalities. We focus on examples and references in the speech analysis domain.

“Classic Features”: Perceptual and Acoustic Measures

Systematic treatises on the importance of emotional expression in speech communication and its powerful impact on the listener can be found throughout history. Early Greek and Roman manuals on rhetoric (e.g., by Aristotle, Cicero, Quintilian) suggested concrete strategies for making speech emotionally expressive. Evolutionary theorists, such as Spencer, Bell, and Darwin, highlighted the social functions of emotional expression in speech and music. The empirical investigation of the effect of emotion on the voice started with psychiatrists trying to diagnose emotional disturbances and early radio researchers concerned with the communication of speaker attributes and states, using the newly developed methods of electroacoustic analysis via vocal cues in speech. Systematic research programs started in the 1960s when psychiatrists renewed their interest in diagnosing affective states, nonverbal communication researchers explored the capacity of different bodily channels to carry signals of emotion, emotion psychologists charted the expression of emotion in different modalities, linguists and particularly phoneticians discovered the importance of pragmatic information, all making use of ever more sophisticated technology to study the effects of emotion on the voice (see Scherer, 2003, for further details).

While much of the relevant research has exclusively focused on the recognition of vocally expressed emotions by naive listeners, research on the production of emotional speech has used the extraction of acoustic parameters from the speech signal as a method to understand the patterning of the vocal expression of different emotions.

Automated surveillance of human activities has traditionally been a computer vision field interested in the recognition of motion patterns and in the production of high-level descriptions for actions and interactions among entities of interest (Cedras & Shah, 1995; Aggarwal & Cai, 1999; Gavrila, 1999; Moeslund, Hilton, & Krüger, 2006; Buxton, 2003; Hu et al., 2004; Turaga et al., 2008; Dee & Velastin, 2008; Aggarwal & Ryoo, 2011; Borges, Conci, & Cavallaro, 2013). The study on human activities has been revitalized in the last five years by addressing the so-called social signals (Pentland, 2007). In fact, these nonverbal cues inspired by the social, affective, and psychological literature (Vinciarelli, Pantic, & Bourlard, 2009) have allowed a more principled understanding of how humans act and react to other people and to their environment.

Social Signal Processing (SSP) is the scientific field making a systematic, algorithmic and computational analysis of social signals, drawing significant concepts from anthropology and social psychology (Vinciarelli et al., 2009). In particular, SSP does not stop at just modeling human activities, but aims at coding and decoding human behavior. In other words, it focuses on unveiling the underlying hidden states that drive one to act in a distinct way with particular actions. This challenge is supported by decades of investigation in human sciences (psychology, anthropology, sociology, etc.) that showed how humans use nonverbal behavioral cues, like facial expressions, vocalizations (laughter, fillers, back-channel, etc.), gestures, or postures to convey, often outside conscious awareness, their attitude toward other people and social environments, as well as emotions (Richmond & McCroskey, 1995). The understanding of these cues is thus paramount in order to understand the social meaning of human activities.

The formal marriage of automated video surveillance with Social Signal Processing had its programmatic start during SISM 2010 (the International Workshop on Socially Intelligent Surveillance and Monitoring; http://profs.sci.univr.it/∼cristanm/ SISM2010/), associated with the IEEE Computer Vision and Pattern Recognition conference. At that venue, the discussion was focused on what kind of social signals can be captured in a generic surveillance scenario, detailing the specific scenarios where the modeling of social aspects could be the most beneficial.

After 2010, SSP hybridizations with surveillance applications have grown rapidly in number and systematic essays about the topic started to compare in the computer vision literature (Cristani et al., 2013).

Introduction

Social signal processing (SSP) is the computing domain aimed at modeling, analysis, and synthesis of social signals in human–human and human–machine interactions (Pentland, 2007; Vinciarelli et al., 2008, 2012; Vinciarelli, Pantic, & Bourlard, 2009). According to different theoretic orientations, social signals can be defined in different ways, for example, “acts or structures that influence the behavior or internal state of other individuals” (Mehu & Scherer, 2012; italics in original), “communicative or informative signals which … provide information about social facts” (Poggi & D'Errico, 2012; italics in original), or “actions whose function is to bring about some reaction or to engage in some process” (Brunet & Cowie, 2012; italics in original). The definitions might appear different, but there seems to be consensus on at least three points.

1. • Social signals are observable behaviors that people display during social interactions.

2. • The social signals of an individual A produce changes in others (e.g., the others develop an impression or a belief about A, react to A with appropriate social signals, or coordinate their social signals with those of A).

3. • The changes produced by the social signals of A in others are not random, but follow principles and laws.

In a computing perspective, the observations above lead to the key idea that shapes the field of Social Signal Processing, namely that social signals are the physical, machine detectable trace of social and psychological phenomena not otherwise accessible to direct observation. In fact, SSP addresses the following three main problems.

1. • Modeling: identification of principles and laws that govern the use of social signals.

2. • Analysis: automatic detection and interpretation of social signals in terms of the principles and laws above.

3. • Synthesis: automatic generation of artificial social signals following the principles and laws above.

Correspondingly, this book is organized into four main sections of which the first three focus on the three problems outlined above while the fourth one introduces current applications of SSP technologies.

Introduction

People have long been intrigued by the notion of a virtual space that could offer an escape from reality to new sensory experiences. As early as 1965, Sutherland envisioned that the ‘ultimate display’ would enable users to actively interact with the virtual space as if it were real, giving them “a chance to gain familiarity with concepts not realizable in the physical world” (Sutherland, 1965, p. 506). William Gibson appears to have shared this vision when coining the term ‘cyberspace’ in his 1984 novel Neuromancer, defining it as “a consensual hallucination experienced daily by billions of legitimate operators, in every nation, by children being taught mathematical concepts …” (p. 51). While the image may have seemed farfetched at the time, the mounting popularity of home video game consoles, massively multiplayer online role playing games (MMORPGs), and massive open online courses (MOOCs) all demonstrate that virtual reality (VR) is an increasingly integral component of our everyday lives.

Despite some romanticized versions of VR, much of the previous literature focused on its dangers. Early studies voiced concerns about how individuals would no longer be able to receive true emotional and social support online (e.g., Kraut et al., 1998) and more recent research focused on Internet addiction (e.g., Lam et al., 2009) as well as the antisocial effects of playing violent games (e.g., Bartholow, Bushman, & Sestir, 2006). Overall, the results suggest that spending extensive amounts of time in VR can result in apathetic or even violent attitudes and behavior toward others. Indeed, early conversations between Jaron Lanier, one of the pioneers of the technology, and William Gibson, who consulted Lanier while writing his manifesto, focused on this tension. Lanier envisioned prosocial uses for the technology he championed, but Gibson felt compelled to write about the less wholesome applications, saying, “Jaron, I tried. But it's coming out dark” (Lanier, 2013, p. 329).

In terms of academic research, there is a group of scholars who focus on a more positive outlook; the unique affordances of virtual environments actually promote prosocial behavior when leveraged. Recent developments show that even brief virtual interventions can increase environmental awareness, reduce racial bias, and enhance general altruistic behavior. These interventions have been found to be especially powerful when the user feels fully immersed in the virtual world.

Introduction

According to the Oxford Dictionary of Sociology, “Role is a key concept in sociological theory. It highlights the social expectations attached to particular social positions and analyses the workings of such expectations” (Scott & Marshall, 2005). Furthermore, “Role theory concerns one of the most important features of social life, characteristic behaviour patterns or roles” (Biddle, 1986). Besides stating that the notion of role is crucial in sociological inquiry, the definitions introduce the two main elements of role theory, namely expectations and characteristic behaviour patterns. In particular, the definitions suggest that the expectations of others – typically associated to the position someone holds in a given social context – shape roles in terms of stable and recognizable behavioural patterns.

Social signal processing (SSP) relies on the similar key idea that social and psychological phenomena leave physical, machine detectable traces in terms of both verbal (e.g., lexical choices) and nonverbal (prosody, postures, facial expressions, etc.) behavioural cues (Vinciarelli, Pantic, & Bourlard, 2009; Vinciarelli et al., 2012). In particular, most SSP works aim at automatically inferring phenomena like conflict, personality, mimicry, effectiveness of delivery, etc. from verbal and nonverbal behaviour. Hence, given the tight relationship between roles and behavioural patterns, SSP methodologies appear to be particularly suitable to map observable behaviour into roles, i.e. to perform automatic role recognition (ARR). Not surprisingly, ARR was one of the earliest problems addressed in the SSP community and the proposed approaches typically include three main steps, namely person detection (segmentation of raw data streams into segments corresponding to a given individual), behavioural cues extraction (detection and representation of relevant behavioural cues), and role recognition (mapping of detected cues into roles). Most of the works presented in the literature propose experiments over two main types of data, i.e. meeting recordings and broadcast material. The probable reason is that these contexts are naturalistic, but sufficiently constrained to allow effective automatic analysis.

The rest of this chapter is organized as follows: role recognition technology, which introduces the main technological components of an ARR system; previous work, which surveys the most important ARR approaches proposed in the literature; open issues, which outlines the main open issues and challenges of the field; and the last section, which draws some conclusions.

When people interact, their behaviors are greatly influenced by the impressions they have of one another's personalities, abilities, attitudes, intentions, identities, roles, and other characteristics. In fact, many important outcomes in life – outcomes as diverse as friendships, professional success, income, romantic relationships, influence over others, and social support – depend to a significant extent on the impressions that people make on others. Knowing that others respond to them on the basis of their public impressions, people devote considerable thought and energy to conveying impressions that will lead others to treat them in desired ways. In many instances, the impressions people project of themselves are reasonably accurate attempts to let other people know who they are and what they are like (Murphy, 2007). At other times, people may convey impressions of themselves that they know are not entirely accurate, if not blatantly deceptive, when they believe that fostering such images will result in desired outcomes (Hancock & Toma, 2009).

Social and behavioral scientists refer to people's efforts to manage their public images as self-presentation or impression management (Goffman, 1959; Schlenker, 2012). Some researchers use different terms for the process of controlling one's public image depending on whether the efforts are honest or deceitful and whether they involve impressions of one's personal characteristics or information about one's social roles and identity. But we will use the terms interchangeably to refer to any intentional effort to convey a particular impression of oneself to another person without respect to the accuracy or content of the effort.

Tactics of Self-presentation

Nearly every aspect of people's behavior provides information from which others can draw inferences about them, but actions are considered self-presentational only if they are enacted, at least in part, with the goal of leading other people to perceive the individual in a particular way. People convey information about their personal and social characteristics using a wide array of tactics.

Verbal Claims

The most direct self-presentational tactics involve verbal statements that make a particular claim regarding one's personal or social characteristics.

Introduction

In everyday life, people usually describe others as being more or less talkative or sociable, more or less angry or vulnerable to stress, more or less planful or behaviorally controlled. Moreover, people exploit these descriptors in their everyday life to explain and/or predict others’ behavior, attaching them to well-known as well as to new acquaintances. In all generality, the attribution of stable personality characteristics to others and their usage to predict and explain their behavior is a fundamental characteristics of human naive psychology (Andrews, 2008).

As agents that in increasingly many and varied ways participate in and affect the lives of humans, computers need to explain and predict their human parties’ behavior by, for example, deploying some kind of naive folk-psychology in which the understanding of people's personality can reasonably be expected to play a role. In this chapter, we address some of the issues that attempts at endowing machines with the capability of predicting people's personality traits.

Scientific psychology has developed a view of personality as a higher-level abstraction encompassing traits, sets of stable dispositions toward action, belief, and attitude formation. Personality traits differ across individuals, are relatively stable over time, and influence behavior. Between-individual differences in behavior, belief, and attitude can therefore be captured in terms of the dispositions/personality traits that are specific to each individual, in this way providing a powerful descriptive and predictive tool that has been widely exploited by, for example, clinical and social psychology, educational psychology, and organizational studies.

The search for personality traits has been often pursued by means of factor-analytic studies applied to lists of trait adjectives, an approach based on the lexical hypothesis (Allport & Odbert, 1936), which maintains that the most relevant individual differences are encoded into the language, and the more important the difference, the more likely it is to be expressed as a single word.

Introduction

Multimodal social-emotional interactions play a critical role in child development and this role is emphasized in autism spectrum disorders (ASD). In typically developing children, the ability to correctly identify, interpret, and produce social behaviors (Figure 28.1) is a key aspect for communication and is the basis of social cognition (Carpendale & Lewis, 2004). This process helps children to understand that other people have intentions, thoughts, and emotions and act as a trigger of empathy (Decety & Jackson, 2004; Narzisi et al., 2012). Social cognition includes the child's ability to spontaneously and correctly interpret verbal and nonverbal social and emotional cues (e.g., speech, facial and vocal expressions, posture and body movements, etc.); the ability to produce social and emotional information (e.g., initiating social contact or conversation); the ability to continuously adjust and synchronize behavior to others (i.e., parent, caregivers, peers); and the ability to make an adequate attribution about another's mental state (i.e., “theory of mind”).

Definitions and Treatments

ASDs are a group of behaviorally defined disorders with abnormalities or impaired development in two areas: (1) persistent deficits in social communication and social interaction and (2) restricted, repetitive patterns of behavior, interests, or activities. An individual with ASD has difficulty interacting with other people due to an inability to understand social cues as well as others’ behaviors and feelings. For example, children with ASD often have difficulty with cooperative play with other peers; they prefer to continue with their own repetitive activities (Baron-Cohen & Wheelwright, 1999). Persons with ASD evaluate both world and human behavior uniquely because they react in an abnormal way to input stimuli while there is problematic human engagement and inability to generalize the environment (Rajendran & Mitchell, 2000). Although ASD remains a devastating disorder with a poor outcome in adult life, there have been important improvements in treating ASD with the development of various therapeutic approaches (Cohen, 2012).

Successful autism “treatments” using educational interventions have been reported as recently as a decade ago (Murray, 1997). Since then, the literature devoted to the description and evaluation of interventions in ASD has become substantial over the last few years. From this literature, a number of conclusions can be drawn. First, there is increasing convergence between behavioral and developmental methods (Ospina et al., 2008).

Introduction

Recent years have seen a growing interest in the development of affective interfaces. At the heart of these systems is an ability to capture social signals and analyse them in a way that meets the requirements and characteristics of the application. There has been a concerted effort in devising a principled approach which could benefit from the interdisciplinary nature of the affective computing endeavour. One common strategy has been to seek conceptual models of emotion that could mediate between the social signals to be captured and the knowledge content of the application. Early systems have largely relied on the intrinsic, natural properties of emotional communication with, for instance, an emphasis on facial expressions both on the output and the input side, together with some theoretical foundation for the acceptance of computers as interaction partners (Reeves & Nass, 1996). Without downplaying the importance of these systems in the development of the field or the practical interest of the applications they entailed (Prendinger & Ishizuka, 2005), it soon appeared that not all interactions could be modelled after interhuman communication, in particular when considering interaction with more complex applications. This complexity can be described at two principal levels: the interaction with complex data, knowledge, or task elements and the nature of the emotions themselves (and their departing from universal or primitive emotions to reflect more sophisticated ones). On the other hand, part of the problem rests with various simplifications that were necessary to get early prototypes off the ground. A good introduction to the full complexity of an affective response can be found in Sander and Scherer (2014), in particular its description of the various levels and components to be considered. These have not always been transposed into affective computing systems; however, as is often the case, the original description frameworks may not be computational enough to support direct and complete transposition.

Dimensional models of emotions have been enthusiastically adopted in affective interfaces for their flexibility in terms of representation as well as the possibility of mapping input modalities onto their dimensions to provide a consistent representation of input.

Interpersonal interactions and relationships can be described as unfolding along two perpendicular dimensions: verticality (power, dominance, control; Burgoon & Hoobler, 2002; Hall, Coats, & LeBeau, 2005) and horizontality (affiliativeness, warmth, friendliness; Kiesler, 1983;Wiggins, 1979). The vertical dimension refers to how much control or influence people can exert, or believe they can exert, over others, as well as the status relations created by social class, celebrity, respect, or expertise. Numerous earlier authors have discussed variations and differences within the verticality concept (e.g., Burgoon & Dunbar, 2006; Burgoon, Johnson, & Koch, 1998; Ellyson & Dovidio, 1985; Keltner, Gruenfeld, & Anderson, 2003).

Social control aspects are prevalent in many social relationships and interactions, not only in formal hierarchies such as in the military or in organizations; there is also a difference in social control between parents and their children, and husbands and wives can have different degrees of power in their relationships. Even within groups of friends or peers, a hierarchy emerges regularly.

Verticality encompasses terms such as power, status, dominance, authority, or leadership. Although different concepts connote different aspects of the vertical dimension, their common denominator is that they are all indicative of the amount of social control or influence and thus of the vertical dimension. Structural power or formal authority describes the difference in social control or influence with respect to social or occupational functions or positions (Ellyson & Dovidio, 1985) (e.g., first officer). Status refers to the standing on the verticality dimension stemming from being a member of a specific social group (e.g., being a man versus a woman) (Pratto et al., 1994). Status also means being awarded a high position on the verticality dimension by others (e.g., emergent leader) (Berger, Conner, & Fisek, 1974). The term dominance (also authority) is used to describe a personality trait of striving for or of having high social control (Ellyson & Dovidio, 1985). Dominance is also used to denote behavior that is aimed at social control (Schmid Mast, 2010). Leadership is the influence on group members to achieve a common goal (Bass, 1960). In a given social situation, different verticality aspects can either converge or diverge. A company leader has high structural power but his or her interaction or leadership style can express more or less dominance.

Most of the world's people are now living in cities and urbanization is expected to keep increasing in the near future. The resulting challenges are complex, difficult to handle, and range from increasing dependence on energy, to the emergence of socio-spatial inequalities, to serious environmental and sustainability issues. Understanding and modeling the structure and evolution of cities is then more important than ever as policy makers are actively looking for new paradigms in urban and transport planning.

The recent advances obtained in the understanding of cities have generated increased attention to the potential implication of new theoretical models in agreement with data. Questions such as urban sprawl, effects of congestion, dominant mechanisms governing the spatial distribution of activities and residences, and the effect of new transportation infrastructures are fundamental questions that we need to understand if we want a harmonious development of cities in the future, from both social and economic points of view.

Cities were for a long time the subject of numerous studies in a large number of fields. Discussion of the ideal city can be traced back at least to the Renaissance, and more recently scientists have tried to describe quantitatively the formation and evolution of cities. Regional science and then quantitative geography addressed various problems such as the spatial organization of cities, the impact of infrastructures, and transport. It is remarkable to note that as early as the 1970s quantitative geographers realized the crucial importance of networks in these systems, and produced visionary studies about networks, their evolution, and the complexity of cities (Haggett et al. 1977).

These studies were further developed mathematically by economists who discussed the interplay between space and economic aspects in cities. Many important models find their origin in the seminal paper of Von Thunen and describe isolated, monocentric cities in terms of utility maximization subject to budget constraints. These models allowed spatial economics to get a grasp of the relations between space, income, and transportation; for example. Japanese economists Fujita and Ogawa discussed the impact of agglomeration effects between firms in a general model that deals with the location choice for individuals and companies.

In the previous chapter, we discussed the analysis and modeling of mobility patterns in cities. However, as cities expand, their transportation networks are also growing, with increasing interconnections between different transportation modes. In large cities, we can now choose the transportation mode to travel from one point to another, and this trip can even involve several different modes. This multimodality is a new aspect of large cities and brings new questions and problems. From the users, point of view, it becomes difficult to deal with the huge amount of information needed for describing the different transportation networks and their interconnections. From the transport agencies point of view, the managing task becomes harder because the different modes are usually run by separate agencies; this renders optimization difficult owing to the large number of aspects that have to be taken into account (Guo and Wilson 2011).

In particular, an important problem concerning multimodality is the synchronization between different modes. For example, on average in the UK, 23% of travel time is lost in connections for trips with more than one mode (Gallotti and Barthelemy, 2014). This lack of synchronization between modes induces differences between the theoretical quickest trip and the “time-respecting” path, which takes into account waiting times at interconnection nodes.

In order to address these problems and more generally to understand the impact of the coupling between modes, we need new tools in order to identify the main factors that govern their efficiency. The multilayer network approach seems to be the most convenient framework for studying these systems (Kivel ä et al. 2014; Boccaletti et al. 2014). In this framework, each layer represents a mode and intermodal connections are represented by inter-layer links. In this chapter we discuss some aspects of multimodality and present tools for measuring and characterizing these coupled networks and their efficiency as a whole.

A multilayer network view of urban navigation

Empirical observations of multimodality

We first describe empirical results obtained by Gallotti and Barthelemy (2014) from timetables for the whole UK and for all transport modes. We note that these results were not obtained for traffic data (that are usually difficult to get). Instead we used these timetable data and the assumption of uniform demand.

What is our “understanding”?

As we discussed in Chapter 1, “understanding” has many definitions, with a variable amount of quantitative input. For a physicist, understanding does not mean having a story consistent with reality only, but also having mathematical tools and models able to describe real phenomena and to predict the outcome of experiments. Even if a qualitative description of processes is somewhat satisfying, it is not enough for constructing a science of cities. Indeed, we would like to identify the most important parameters, not only to understand the past, but also to be able to construct a model that gives, with a reasonable confidence, the future evolution of a city and to test the impact of various policies.

At this point, we certainly have a number of pieces of the puzzle, and we have discussed some of them in this book. It doesn't however mean that we have solved the full puzzle. New data sources and large datasets allow us to get a precise idea of what is happening in cities. We are currently experiencing an exciting time during which we can challenge the purely theoretical developments made these last decades. In many empirical studies, the identification of relevant factors was essentially done statistically, and we can now hope to go beyond and to have a more mechanistic approach, where a model based on simple processes is able to reproduce empirical observations.

Concerning the spatial structure of cities, new data sources give us a real-time, high-resolution picture of mobility. The structure of mobility flows that come out from these datasets departs from the usual image of a monocentric city where flows converge towards the central business district. Instead, for large cities, the main flows appear to be far from the localization between centers of residence and activities, that we could have na ïvely expected. This massive amount of data also allows us to quantitatively assess the degree of polycentricity of an urban system. A simple model showed that congestion is a crucial factor in understanding the evolution of polycentricity and mobility patterns with the population size.