Hostname: page-component-cd9895bd7-7cvxr Total loading time: 0 Render date: 2024-12-26T15:41:33.543Z Has data issue: false hasContentIssue false
  • English
  • Français

Using automated computer vision and machine learning to code facial expressions of affect and arousal: Implications for emotion dysregulation research

Published online by Cambridge University Press:  28 March 2019

Nathaniel Haines ,
Ziv Bell ,
Sheila Crowell ,
Hunter Hahn ,
Dana Kamara ,
Heather McDonough-Caplan ,
Tiffany Shader  and
Theodore P. Beauchaine
Nathaniel Haines
Affiliation:
Department of Psychology, Ohio State University, Columbus, OH, USA
Ziv Bell
Affiliation:
Department of Psychology, Ohio State University, Columbus, OH, USA
Sheila Crowell
Affiliation:
Department of Psychology, University of Utah, Salt Lake City, UT, USA Department of Psychiatry, University of Utah, Salt Lake City, UT, USA
Hunter Hahn
Affiliation:
Department of Psychology, Ohio State University, Columbus, OH, USA
Dana Kamara
Affiliation:
Department of Psychology, Ohio State University, Columbus, OH, USA
Heather McDonough-Caplan
Affiliation:
Department of Psychology, Ohio State University, Columbus, OH, USA
Tiffany Shader
Affiliation:
Department of Psychology, Ohio State University, Columbus, OH, USA
Theodore P. Beauchaine*
Affiliation:
Department of Psychology, Ohio State University, Columbus, OH, USA
*
Author for Correspondence: Theodore P. Beauchaine, Department of Psychology, Ohio State University, 1835 Neil Avenue, Columbus, OH 43210; Email: [email protected].
Get access Rights & Permissions [Opens in a new window]

Abstract

As early as infancy, caregivers’ facial expressions shape children's behaviors, help them regulate their emotions, and encourage or dissuade their interpersonal agency. In childhood and adolescence, proficiencies in producing and decoding facial expressions promote social competence, whereas deficiencies characterize several forms of psychopathology. To date, however, studying facial expressions has been hampered by the labor-intensive, time-consuming nature of human coding. We describe a partial solution: automated facial expression coding (AFEC), which combines computer vision and machine learning to code facial expressions in real time. Although AFEC cannot capture the full complexity of human emotion, it codes positive affect, negative affect, and arousal—core Research Domain Criteria constructs—as accurately as humans, and it characterizes emotion dysregulation with greater specificity than other objective measures such as autonomic responding. We provide an example in which we use AFEC to evaluate emotion dynamics in mother–daughter dyads engaged in conflict. Among other findings, AFEC (a) shows convergent validity with a validated human coding scheme, (b) distinguishes among risk groups, and (c) detects developmental increases in positive dyadic affect correspondence as teen daughters age. Although more research is needed to realize the full potential of AFEC, findings demonstrate its current utility in research on emotion dysregulation.

Keywords

arousalemotion dysregulationfacial expressionnegative valence systempositive valence system
Type
Special Issue Articles
Information
Development and Psychopathology , Volume 31 , Special Issue 3: Emotion Dysregulation and Emerging Psychopathology , August 2019 , pp. 871 - 886
Copyright
Copyright © Cambridge University Press 2019 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Adams, A., & Robinson, P. (2015). Automated recognition of complex categorical emotions from facial expressions and head motions. Paper presented at the 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), Xi'an, China.Google Scholar
Baltrusaitis, T., Robinson, P., & Morency, L.-P. (2016). OpenFace: An open source facial behavior analysis toolkit. Paper presented at the 2016 IEEE Winter Conference on Applications of Computer Vision, Lake Placid, NY.Google Scholar
Barrett, L. F. (2009). The future of psychology: Connecting mind to brain. Perspectives on Psychological Science, 4, 326–339. doi:10.1111/j.1745-6924.2009.01134.xGoogle Scholar
Barsade, S. G. (2002). The ripple effect: Emotional contagion and its influence on group behavior. Administrative Science Quarterly, 47, 644–675. doi:10.2307/3094912Google Scholar
Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Measuring facial expressions by computer image analysis. Psychophysiology, 36, 253–263. doi:10.1017/S0048577299971664Google Scholar
Bartlett, M. S., Littlewort, G. C., Frank, M. G., & Lee, K. (2014). Automatic decoding of facial movements reveals deceptive pain expressions. Current Biology, 24, 738–743. doi:10.1016/j.cub.2014.02.009Google Scholar
Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48. doi:10.18637/jss.v067.i01Google Scholar
Beauchaine, T. P. (2015). Future directions in emotion dysregulation and youth psychopathology. Journal of Clinical Child and Adolescent Psychology, 44, 875–896. doi:10.1080/15374416.2015.1038827Google Scholar
Beauchaine, T. P., & Haines, N. (in press). Functionalist and constructionist perspectives on emotion. In Beauchaine, T. P. & Crowell, S. E. (Eds.), The Oxford handbook of emotion dysregulation. New York: Oxford University Press.Google Scholar
Beauchaine, T. P., & Webb, S. J. (2017). Developmental processes and psychophysiology. In Cacioppo, J. T., Tassinary, L. G., & Berntson, G. G. (Eds.), Handbook of psychophysiology (4th ed., pp. 495–510). New York: Cambridge University Press.Google Scholar
Beauchaine, T. P., & Zalewski, M. (2016). Physiological and developmental mechanisms of emotional lability in coercive relationships. In Dishion, T. J. & Snyder, J. J. (Eds.), The Oxford handbook of coercive relationship dynamics (pp. 39–52). New York: OXford University Press.Google Scholar
Beckes, L., & Edwards, W. L. (2018). Emotions as regulators of social behavior. In Beauchaine, T. P. & Crowell, S. E. (Eds.), The Oxford handbook of emotion dysregulation. New York: Oxford University Press.Google Scholar
Blair, R. (1995). A cognitive developmental approach to morality: Investigating the psychopath. Cognition, 57, 1–29. doi:10.1016/0010-0277(95)00676-PGoogle Scholar
Blair, R. J. R., & Coles, M. (2000). Expression recognition and behavioural problems in early adolescence. Cognitive Development, 15, 421–434. doi:10.1016/S0885-2014(01)00039-9Google Scholar
Bodner, N., Kuppens, P., Allen, N. B., Sheeber, L. B., & Ceulemans, E. (2018). Affective family interactions and their associations with adolescent depression: A dynamic network approach. Development and Psychopathology, 30, 1459–1473. doi:10.1017/S0954579417001699Google Scholar
Bonanno, G. A., & Burton, C. L. (2013). Regulatory flexibility: An individual differences perspective on coping and emotion regulation. Perspectives on Psychological Science, 8, 591–612. doi:10.1177/1745691613504116Google Scholar
Bonanno, G. A., Colak, D. M., Keltner, D., Shiota, M. N., Papa, A., Noll, J., … Trickett, P. K. (2007). Context matters: The benefits and costs of expressing positive emotion among survivors of childhood sexual abuse. Emotion, 7, 824–837. doi:10.1037/1528-3542.7.4.824Google Scholar
Bonanno, G. A., Papa, A., Lalande, K., Westphal, M., & Coifman, K. (2004). The importance of being flexible: The ability to both enhance and suppress emotional expression predicts long-term adjustment. Psychological Science, 15, 482–487. doi:10.1111/j.0956-7976.2004.00705.xGoogle Scholar
Buck, R. (1975). Nonverbal communication of affect in children. Journal of Personality and Social Psychology, 31, 644–653. doi:10.1037/h0077071Google Scholar
Buck, R. (1977). Nonverbal communication of affect in preschool children: Relationships with personality and skin conductance. Journal of Personality and Social Psychology, 35, 225–236. doi:10.1037/0022-3514.35.4.225Google Scholar
Campos, J. J., Mumme, D. L., Kermoian, R., & Campos, R. G. (1994). A functionalist perspective on the nature of emotion. Monographs of the Society for Research in Child Development, 59, 284–303. doi:10.1111=j.1540-5834.1994.tb01289.xGoogle Scholar
Camras, L. A., Grow, J. G., & Ribordy, S. C. (1983). Recognition of emotional expression by abused children. Journal of Clinical Child Psychology, 12, 325–328. doi:10.1207/s15374424jccp1203_16Google Scholar
Camras, L. A., Ribordy, S., Hill, J., Martino, S., Spaccarelli, S., & Stefani, R. (1988). Recognition and posing of emotional expressions by abused children and their mothers. Developmental Psychology, 24, 776–781. doi:10.1037//0012-1649.24.6.776Google Scholar
Carp, J. (2012). On the plurality of (methodological) worlds: Estimating the analytic flexibility of fMRI experiments. Frontiers in Neuroscience, 6, 149. doi:10.3389/fnins.2012.00149Google Scholar
Chen, S., Chen, T., & Bonanno, G. A. (2018). Expressive flexibility: Enhancement and suppression abilities differentially predict life satisfaction and psychopathology symptoms. Personality and Individual Differences, 126, 78–84. doi:10.1016/j.paid.2018.01.010Google Scholar
Chevalier-Skolnikoff, S. (1973). Facial expression of emotion in nonhuman primates. In Ekman, P. (Ed.), Darwin and facial expression (pp. 11–83). New York: Oxford University Press.Google Scholar
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159. doi:10.1037/0033-2909.112.1.155Google Scholar
Cohn, J. F. (2010). Advances in behavioral science using automated facial image analysis and synthesis [social sciences]. IEEE Signal Processing Magazine, 27, 128–133. doi:10.1109/MSP.2010.938102Google Scholar
Cohn, J. F., & De la Torre, F. (2014). Automated face analysis for affective. In Calvo, R. A., D'Mello, S., Gratch, J., & Kappas, A. (Eds.), The Oxford handbook of affective computing (pp. 131–150). New York: Oxford University Press.Google Scholar
Cohn, J. F., & Ekman, P. (2005). Measuring facial action. In Harrigan, J. A., Rosenthal, R., & Scherer, K. R. (Eds.), The new handbook of nonverbal behavior research methods in the affective sciences (pp. 9–64). New York: Oxford University Press.Google Scholar
Coifman, K., & Almahmoud, S. (2017). Emotion flexibility and psychological risk and resilience. In Kumar, U. (Ed.), The Routledge international handbook of psychosocial resilience (pp. 1–14). London: Routledge.Google Scholar
Coifman, K. G., Flynn, J. J., & Pinto, L. A. (2016). When context matters: Negative emotions predict psychological health and adjustment. Motivation and Emotion, 40, 602–624. doi:10.1007/s11031-016-9553-yGoogle Scholar
Crowell, S. E., Baucom, B. R., McCauley, E., Potapova, N. V., Fitelson, M., Barth, H., … Beauchaine, T. P. (2013). Mechanisms of contextual risk for adolescent self-injury: Invalidation and conflict escalation in mother–child interactions. Journal of Clinical Child and Adolescent Psychology, 42, 467–480. doi:10.1080/15374416.2013.785360Google Scholar
Crowell, S. E., Baucom, B. R., Yaptangco, M., Bride, D., Hsiao, R., McCauley, E., & Beauchaine, T. P. (2014). Emotion dysregulation and dyadic conflict in depressed and typical adolescents: Evaluating concordance across psychophysiological and observational measures. Biological Psychology, 98, 50–58. doi:10.1016/j.biopsycho.2014.02.009Google Scholar
Crowell, S. E., Beauchaine, T. P., Hsiao, R. C., Vasilev, C. A., Yaptangco, M., Linehan, M. M., & McCauley, E. (2012). Differentiating adolescent self-injury from adolescent depression: Possible implications for borderline personality development. Journal of Abnormal Child Psychology, 40, 45–57. doi:10.1007/s10802-011-9578-3Google Scholar
Crowell, S. E., Butner, J., Wiltshire, T. J., Munion, A. K., Yaptangco, M., & Beauchaine, T. P. (2017). Evaluating emotional and biological sensitivity to maternal behavior among depressed and self-injuring adolescent girls using nonlinear dynamics. Clinical Psychological Science, 5, 272–285. doi:10.1177/2167702617692861Google Scholar
Darwin, C. (1872). The expression of the emotions in man and animals. London: Murray.Google Scholar
Dibeklioğlu, H., Hammal, Z., Yang, Y., & Cohn, J. F. (2015). Multimodal detection of depression in clinical interviews. Paper presented at the 2015 ACM on International Conference on Multimodal Interaction (pp. 307–310), Seattle, WA.Google Scholar
Di Pierro, R., Sarno, I., Perego, S., Gallucci, M., & Madeddu, F. (2012). Adolescent nonsuicidal self-injury: The effects of personality traits, family relationships and maltreatment on the presence and severity of behaviours. European Child and Adolescent Psychiatry, 21, 511–520. doi:10.1007/s00787-012-0289-2Google Scholar
Domitrovich, C. E., Cortes, R. C., & Greenberg, M. T. (2007). Improving young children's social and emotional competence: A randomized trial of the preschool “PATHS” curriculum. Journal of Primary Prevention, 28, 67–91. doi:10.1007/s10935-007-0081-0Google Scholar
Du, S., Tao, Y., & Martinez, A. M. (2014). Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, 111, E1454–E1462. doi:10.1073/pnas.1322355111Google Scholar
Duchenne, G. (1990). The mechanism of human facial expression. Translated by Cuthbertson, R. A.. New York: Cambridge University Press. (Originally published as Mecanisme de la Physionomie Humaine in 1862.)Google Scholar
Dys, S. P., & Malti, T. (2016). It's a two-way street: Automatic and controlled processes in children's emotional responses to moral transgressions. Journal of Experimental Child Psychology, 152, 31–40. doi:10.1016/j.jecp.2016.06.011Google Scholar
Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion , 6 , 169–200. doi:10.1080/02699939208411068Google Scholar
Ekman, P. (1993). Facial expression and emotion. American Psychologist, 48, 384–392. doi:10.1037/0003-066X.48.4.384Google Scholar
Ekman, P., & Cordaro, D. (2011). What is meant by calling emotions basic. Emotion Review , 3 , 364–370. doi:10.1177/1754073911410740Google Scholar
Ekman, P., & Friesen, W. (1978). Facial action coding system: A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press.Google Scholar
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., … Ricci-Bitti, P. E. (1987). Universals and cultural differences in the judgments of facial expressions of emotion. Journal of Personality and Social Psychology, 53, 712–717. doi:10.1037/0022-3514.53.4.712Google Scholar
Ekman, P., & Oster, H. (1979). Facial expressions of emotion. Annual Review of Psychology, 30, 527–554. doi:0.1146/annurev.ps.30.020179.002523Google Scholar
Ekman, P., Roper, G., & Hager, J. C. (1980). Deliberate facial movement. Child Development, 51, 886–891. doi:10.2307/1129478Google Scholar
Ekman, P., & Rosenberg, E. L. (Eds.). (2005). What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (2nd ed.). New York: Oxford University Press.Google Scholar
Fairchild, G., Van Goozen, S. H. M., Calder, A. J., Stollery, S. J., & Goodyer, I. M. (2009). Deficits in facial expression recognition in male adolescents with early-onset or adolescence-onset conduct disorder. Journal of Child Psychology and Psychiatry, 50, 627–636. doi:10.1111/j.1469-7610.2008.02020.xGoogle Scholar
Fasel, B., & Luettin, J. (2003). Automatic facial expression analysis: A survey. Pattern Recognition, 36, 259–275. doi:10.1016/S0031-3203(02)00052-3Google Scholar
Field, T. M., & Walden, T. A. (1982). Production and discrimination of facial expressions by preschool children. Child Development, 53, 1299–1311. doi:10.2307/1129020Google Scholar
Field, T. M., Woodson, R., Greenberg, R., & Cohen, D. (1982). Discrimination and imitation of facial expression by neonates. Science, 218, 179–181. doi:10.1126/science.7123230Google Scholar
Gadea, M., Aliño, M., Espert, R., & Salvador, A. (2015). Deceit and facial expression in children: The enabling role of the “poker face” child and the dependent personality of the detector. Frontiers in Psychology, 6, 6. doi:10.3389/fpsyg.2015.01089Google Scholar
Gallup, A. C., O'Brien, D. T., & Wilson, D. S. (2010). The relationship between adolescent peer aggression and responses to a sequential Prisoner's Dilemma game during college: An explorative study. Journal of Social, Evolutionary, and Cultural Psychology, 4, 277–289. doi:10.1037/h0099283Google Scholar
Girard, J. M., & Cohn, J. F. (2015). Automated audiovisual depression analysis. Current Opinion in Psychology, 4, 75–79. doi:10.1016/j.copsyc.2014.12.010Google Scholar
Golan, O., Ashwin, E., Granader, Y., McClintock, S., Day, K., Leggett, V., & Baron-Cohen, S. (2009). Enhancing emotion recognition in children with autism spectrum conditions: An intervention using animated vehicles with real emotional faces. Journal of Autism and Developmental Disorders, 40, 269–279. doi:10.1007/s10803-009-0862-9Google Scholar
Golan, O., Baron-Cohen, S., & Hill, J. (2006). The Cambridge Mindreading (CAM) face-voice battery: Testing complex emotion recognition in adults with and without Asperger syndrome. Journal of Autism and Developmental Disorders, 36, 169–183. doi:10.1007/s10803-005-0057-yGoogle Scholar
Goodenough, F. L. (1932). Expression of the emotions in a blind-deaf child. Journal of Abnormal and Social Psychology, 27, 328–333. doi:10.1037/h0076099Google Scholar
Gordon, I., Pierce, M. D., Bartlett, M. S., & Tanaka, J. W. (2014). Training facial expression production in children on the autism spectrum. Journal of Autism and Developmental Disorders, 44, 2486–2498. doi:10.1007/s10803-014-2118-6Google Scholar
Gross, J. J., & Levenson, R. W. (1993). Emotional suppression: Physiology, self-report, and expressive behavior. Journal of Personality and Social Psychology, 64, 970–986. doi:10.1037/0022-3514.64.6.970Google Scholar
Gunes, H., & Pantic, M. (2010). Automatic, dimensional and continuous emotion recognition. International Journal of Synthetic Emotions, 1. doi:10.4018/jse.2010101605Google Scholar
Gunes, H., Schuller, B., Pantic, M., & Cowie, R. (2011). Emotion representation, analysis and synthesis in continuous space: A survey. Proceedings of the Automatic Face and Gesture Recognition Workshop, 2011 IEEE International Conference (pp. 827–834), Santa Barbara, CA.Google Scholar
Haines, N., Southward, M. W., Cheavens, J. S., Beauchaine, T. P., & Ahn, W.-Y. (2019). Using computer-vision and machine learning to automate facial coding of positive and negative affect intensity. PLOS ONE, 14, e0211735. doi:10.1371/journal.pone.0211735Google Scholar
Hamm, J., Kohler, C. G., Gur, R. C., & Verma, R. (2011). Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders. Journal of Neuroscience Methods, 200, 237–256. doi:10.1016/j.jneumeth.2011.06.023Google Scholar
Hammal, Z., Cohn, J. F., Heike, C., & Speltz, M. L. (2015). Automatic measurement of head and facial movement for analysis and detection of infants’ positive and negative affect. Frontiers in ICT, 2, 21. doi:10.3389/fict.2015.00021Google Scholar
Hipwell, A., Keenan, K., Kasza, K., Loeber, R., Stouthamer-Loeber, M., & Bean, T. (2008). Reciprocal influences between girls’ conduct problems and depression, and parental punishment and warmth: A six-year prospective analysis. Journal of Abnormal Child Psychology, 36, 663–677. doi:10.1007/s10802-007-9206-4Google Scholar
Hoffner, C., & Badzinski, D. M. (1989). Children's integration of facial and situational cues to emotion. Child Development, 60, 411–422. doi:10.2307/1130986Google Scholar
iMotions. (2018). Facial expression analysis: The definitive guide. Retrieved from https://imotions.com/facialexpression-guide-ebook/ on 3/17/2018Google Scholar
Isaacowitz, D. M., Löckenhoff, C. E., Lane, R. D., Wright, R., Sechrest, L., Riedel, R., & Costa, P. T. (2007). Age differences in recognition of emotion in lexical stimuli and facial expressions. Psychology and Aging, 22, 147–159. doi:10.1037/0882-7974.22.1.147Google Scholar
Izard, C. E., Fine, S., Schultz, D., Mostow, A., Ackerman, B., & Youngstrom, E. (2001). Emotion knowledge as a predictor of social behavior and academic competence in children at risk. Psychological Science, 12, 18–23. doi:10.1111/1467-9280.00304Google Scholar
Izard, C. E., Huebner, R. R., Risser, D., McGinnes, G., & Dougherty, L. (1980). The young infant's ability to produce discrete emotion expressions. Developmental Psychology, 16, 132–140. doi:10.1037/0012-1649.16.2.132Google Scholar
Izard, C. E., & Malatesta, C. Z. (1987). Perspectives on emotional development: I. Differential emotions theory of early emotional development. In Osofsky, J. D. (Ed.), Wiley series on personality processes: Handbook of infant development (pp. 494–554). Oxford: Wiley.Google Scholar
Jack, R. E., Garrod, O. G. B., Yu, H., Caldara, R., & Schyns, P. G. (2012). Facial expressions of emotion are not culturally universal. Proceedings of the National Academy of Sciences, 109, 7241–7244. doi:10.1073/pnas.1200155109Google Scholar
Kanluan, I., Grimm, M., & Kroschel, K. (2008). Audio-visual emotion recognition using an emotion space concept. Paper presented at the 16th European Signal Processing Conference, Lausanne, Switzerland.Google Scholar
Keltner, D., Moffitt, T. E., & Stouthamer-Loeber, M. (1995). Facial expressions of emotion and psychopathology in adolescent boys. Journal of Abnormal Psychology, 104, 644–652. doi:10.1037/0021-843X.104.4.644Google Scholar
Korb, S., With, S., Niedenthal, P., Kaiser, S., & Grandjean, D. (2014). The perception and mimicry of facial movements predict judgments of smile authenticity. PLOS ONE, 9, e99194. doi:10.1371/journal.pone.0099194Google Scholar
Kotsia, I., & Pitas, I. (2007). Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Transactions on Image Processing, 16, 172–187. doi:10.1109/TIP.2006.884954Google Scholar
Krumhuber, E. G., Skora, L., Küster, D., & Fou, L. (2016). A review of dynamic datasets for facial expression research. Emotion Review, 9, 280–292. doi:10.1177/1754073916670022Google Scholar
Lewinski, P., den Uyl, T. M., & Butler, C. (2014). Automated facial coding: Validation of basic emotions and FACS AUs in FaceReader. Journal of Neuroscience, Psychology, and Economics, 7, 227–236. doi:10.1037/npe0000028Google Scholar
Li, J. J., & Lansford, J. E. (2018). A smartphone-based ecological momentary assessment of parental behavioral consistency: Associations with parental stress and child ADHD symptoms. Developmental Psychology, 54, 1086–1098. doi:10.1037/dev0000516Google Scholar
Littlewort, G., Whitehill, J., Wu, T., & Fasel, I. (2011). The computer expression recognition toolbox (CERT). Paper presented at the IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), Santa Barbara, CA.Google Scholar
Lopes, A. T., de Aguiar, E., De Souza, A. F., & Oliveira-Santos, T. (2017). Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order. Pattern Recognition, 61, 610–628. doi:10.1016/j.patcog.2016.07.026Google Scholar
Luterek, J. A., Orsillo, S. M., & Marx, B. P. (2005). An experimental examination of emotional experience, expression, and disclosure in women reporting a history of childhood sexual abuse. Journal of Traumatic Stress, 18, 237–244. doi:10.1002/jts.20027Google Scholar
Marsh, P., Beauchaine, T. P., & Williams, B. (2008). Dissociation of sad facial expressions and autonomic nervous system responding in boys with disruptive behavior disorders. Psychophysiology, 45, 100–110. doi:10.1111/j.1469-8986.2007.00603.xGoogle Scholar
Mattson, W. I., Cohn, J. F., Mahoor, M. H., Gangi, D. N., & Messinger, D. S. (2013). Darwin's Duchenne: Eye constriction during infant joy and distress. PLOS ONE, 8, e80161. doi:10.1371/journal.pone.0080161Google Scholar
McDuff, D., El Kaliouby, R., Kassam, K., & Picard, R. (2010). Affect valence inference from facial action unit spectrograms. Paper presented at the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), San Francisco, CA.Google Scholar
Meltzoff, A. N., & Moore, M. K. (1977). Imitation of facial and manual gestures by human neonates. Science, 198, 75–78. doi:10.1126/science.198.4312.75Google Scholar
Messinger, D. S., Mahoor, M., Chow, S.-M., & Cohn, J. (2009). Automated measurement of facial expression in infant-mother interaction: A pilot study. Infancy, 14, 285–305. doi:10.1080/15250000902839963Google Scholar
Messinger, D. S., Mattson, W. I., Mahoor, M. H., & Cohn, J. F. (2012). The eyes have it: Making positive expressions more positive and negative expressions more negative. Emotion, 12, 430–436. doi:10.1037/a0026498Google Scholar
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2017). AffectNet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 1. doi:10.1109/TAFFC.2017.2740923Google Scholar
Muehlenkamp, J. J., Kerr, P. L., Bradley, A. R., & Adams Larsen, M. (2010). Abuse subtypes and nonsuicidal self-injury: Preliminary evidence of complex emotion regulation patterns. Journal of Nervous and Mental Disease, 198, 258–263. doi:10.1097/NMD.0b013e3181d612abGoogle Scholar
National Advisory Mental Health Council Workgroup on Tasks and Measures for Research Domain Criteria. (2016). Behavioral assessment methods for RDoC constructs. Bethesda, MD: National Institute of Mental Health.Google Scholar
Nelson, C. A. (1987). The recognition of facial expressions in the first two years of life: Mechanisms of development. Child Development, 58, 889–909. doi:10.2307/1130530Google Scholar
Nicolaou, M. A., Gunes, H., & Pantic, M. (2011). Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Transactions on Affective Computing, 2, 92–105. doi:10.1109/T-AFFC.2011.9Google Scholar
Noldus. (2018). FaceReader: Tool for automatic analysis of facial expression: Version 6.0 [Software]. Wageningen, Netherlands: Noldus Information Technology.Google Scholar
Odom, R. D., & Lemond, C. M. (1972). Developmental differences in the perception and production of facial expressions. Child Development, 43, 359–369. doi:10.2307/1127541Google Scholar
Parrott, W. G. (2000). Emotions in social psychology: Key readings. Philadelphia, PA: Psychology Press.Google Scholar
Pessoa, L. (2008). On the relationship between emotion and cognition. Nature Reviews Neuroscience, 9, 148–158. doi:10.1038/nrn2317Google Scholar
Pollak, S. D., Cicchetti, D., Hornung, K., & Reed, A. (2000). Recognizing emotion in faces: Developmental effects of child abuse and neglect. Developmental Psychology, 36, 679–688. doi:10.1037/0012-1649.36.5.679Google Scholar
Pollak, S. D., & Sinha, P. (2002). Effects of early experience on children's recognition of facial displays of emotion. Developmental Psychology, 38, 784–791. doi:10.1037/0012-1649.38.5.784Google Scholar
Porter, S., & ten Brinke, L. (2010). The truth about lies: What works in detecting high-stakes deception? Legal and Criminological Psychology, 15, 57–75. doi:10.1348/135532509X433151Google Scholar
Preuschoft, S., & van Hooff, J. A. (1995). Homologizing primate facial displays: A critical review of methods. Folia Primatologica, 65, 121–137. doi:10.1159/000156878Google Scholar
Prinz, R. J., Foster, S., Kent, R. N., & O'Leary, K. D. (1979). Multivariate assessment of conflict in distressed and nondistressed mother-adolescent dyads. Journal of Applied Behavior Analysis, 12, 691–700. doi:10.1901/jaba.1979.12-691Google Scholar
Ramsook, K. A., Cole, P. M., & Fields-Olivieri, M. A. (2019). What emotion dysregulation looks like: Inferences from behavioral observations. In Beauchaine, T. P. & Crowell, S. E. (Eds.), The Oxford handbook of emotion dysregulation. New York: Oxford University Press.Google Scholar
Reed, L. I., DeScioli, P., & Pinker, S. A. (2014). The commitment function of angry facial expressions. Psychological Science, 25, 1511–1517. doi:10.1177/0956797614531027Google Scholar
Ringeval, F., Schuller, B., Valstar, M., Jaiswal, S., Marchi, E., Lalanne, D., … Pantic, M. (2015). AV+EC 2015: The first affect recognition challenge bridging across audio, video, and physiological data. Paper presented at the 5th International Workshop on Audio/Visual Emotion Challenge, Brisbane, Australia.Google Scholar
Robin, M., Pham-Scottez, A., Curt, F., Bigre, C. D.-L., Speranza, M., Sapinho, D., … Kedia, B. S. (2012). Decreased sensitivity to facial emotions in adolescents with borderline personality disorder. Psychiatry Research, 200, 417–421. doi:10.1016/j.psychres.2012.03.032Google Scholar
Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1161–1178. doi:10.1037/h0077714Google Scholar
Russell, J. A., & Bullock, M. (1985). Multidimensional scaling of emotional facial expressions: Similarity from preschoolers to adults. Journal of Personality and Social Psychology, 48, 1290–1298. doi:10.1037//0022-3514.48.5.1290Google Scholar
Schlosberg, H. (1941). A scale for the judgment of facial expressions. Journal of Experimental Psychology, 29, 497–510. doi:10.1037/h0061489Google Scholar
Schlosberg, H. (1952). The description of facial expressions in terms of two dimensions. Journal of Experimental Psychology, 44, 229–237. doi:10.1037/h0055778Google Scholar
Schlosberg, H. (1954). Three dimensions of emotion. Psychological Review, 61, 81–88. doi:10.1037/h0054570Google Scholar
Sikka, K., Ahmed, A. A., Diaz, D., Goodwin, M. S., Craig, K. D., Bartlett, M. S., & Huang, J. S. (2015). Automated assessment of children's postoperative pain using computer vision. Pediatrics, 136, e124–e131. doi:10.1542/peds.2015-0029Google Scholar
Smith, C. A., & Ellsworth, P. C. (1985). Patterns of cognitive appraisal in emotion. Journal of Personality and Social Psychology, 48, 813–838. doi:10.1037/0022-3514.48.4.813Google Scholar
Smith, E. R., & Mackie, D. M. (2015). Dynamics of group-based emotions: Insights from intergroup emotions theory. Emotion Review, 7, 349–354. doi:10.1177/1754073915590614Google Scholar
Smith, M., & Walden, T. (1998). Developmental trends in emotion understanding among a diverse sample of African-American preschool children. Journal of Applied Developmental Psychology, 19, 177–197. doi:10.1016/S0193-3973(99)80035-5Google Scholar
Smith, S. (2003). Digital signal processing: A practical guide for engineers and scientists. Burlington, MA: Elsevier Science.Google Scholar
Snyder, J., Schrepferman, L., & St Peter, C. (1997). Origins of antisocial behavior: Negative reinforcement and affect dysregulation of behavior as socialization mechanisms in family interaction. Behavior Modification, 21, 187–215. doi:10.1177/01454455970212004Google Scholar
Song, R., Over, H., & Carpenter, M. (2016). Young children discriminate genuine from fake smiles and expect people displaying genuine smiles to be more prosocial. Evolution and Human Behavior, 37, 490–501. doi:10.1016/j.evolhumbehav.2016.05.002Google Scholar
Sorce, J. F., Emde, R. N., Campos, J., & Klinnert, M. (1985). Maternal emotional signaling: Its effect on the visual cliff behavior of 1-year-olds. Developmental Psychology, 21, 195–200. doi:10.1037/0012-1649.21.1.195Google Scholar
Stöckli, S., Schulte-Mecklenbeck, M., Borer, S., & Samson, A. C. (2017). Facial expression analysis with AFFDEX and FACET: A validation study. Behavior Research Methods, 26, 1–15. doi:10.3758/s13428-017-0996-1Google Scholar
Stubbs, J., Crosby, L., Forgatch, M. S., & Capaldi, D. M. (1998). Family and peer process code: A synthesis of three Oregon Social Learning Center behavior codes [Training manual]. Eugene, OR: Oregon Social Learning Center.Google Scholar
Thomas, L. A., De Bellis, M. D., Graham, R., & LaBar, K. S. (2007). Development of emotional facial recognition in late childhood and adolescence. Developmental Science, 10, 547–558. doi:10.1111/j.1467-7687.2007.00614.xGoogle Scholar
Thompson, J. (1941). Development of facial expression of emotion in blind and seeing children. Archives of Psychology (Columbia University), 264, 47.Google Scholar
Thompson, R. A. (1990). Emotion and self-regulation. In Thompson, R. A. (Ed.), Nebraska Symposium on Motivation: Vol. 36. Socioemotional development (pp. 383–483). Lincoln, NE: University of Nebraska Press.Google Scholar
Tian, Y. I., Kanade, T., & Cohn, J. F. (2001). Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23, 97–115. doi:10.1109/34.908962Google Scholar
Troster, H., & Brambring, M. (1992). Early social-emotional development in blind infants. Child: Care, Health and Development, 18, 207–227. doi:10.1111/j.1365-2214.1992.tb00355.xGoogle Scholar
Valstar, M., Gratch, J., Schuller, B., Ringeval, F., Lalanne, D., Torres, M. T., … Pantic, M. (2016). AVEC 2016: Depression, mood, and emotion recognition workshop and challenge. Paper presented at the Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, Amsterdam, Netherlands.Google Scholar
Watson, D., & Tellegen, A. (1985). Toward a consensual structure of mood. Psychological Bulletin, 98, 219–235. doi:10.1037/0033-2909.98.2.219Google Scholar
Westphal, M., Seivert, N. H., & Bonanno, G. A. (2010). Expressive flexibility. Emotion, 10, 92–100. doi:10.1037/a0018420Google Scholar
Widen, S. C., & Russell, J. (2003). A closer look at preschoolers' freely produced labels for facial expressions. Developmental Psychology, 39, 114–128. doi:10.1037/0012-1649.39.1.114Google Scholar
Yarczower, M., Kilbride, J. E., & Hill, L. (1979). Imitation and inhibition of facial expression. Developmental Psychology, 15, 453–454. doi:10.1037/0012-1649.15.4.453Google Scholar
Yoshimura, S., Sato, W., Uono, S., & Toichi, M. (2014). Impaired overt facial mimicry in response to dynamic facial expressions in high-functioning autism spectrum disorders. Journal of Autism and Developmental Disorders, 45, 1318–1328. doi:10.1007/s10803-014-2291-7Google Scholar
Zhao, X., & Zhang, S. (2016). A review on facial expression recognition: Feature extraction and classification. IETE Technical Review, 33, 505–517. doi:10.1080/02564602.2015.1117403Google Scholar
Zisner, A., & Beauchaine, T. P. (2016). Psychophysiological methods and developmental psychopathology. In Cicchetti, D. (Ed.), Developmental psychopathology: Vol. 2. Developmental neuroscience (3rd ed., pp. 832–884). Hoboken, NJ: Wiley.Google Scholar
Zivin, G. (1977). On becoming subtle: Age and social rank changes in the use of a facial gesture. Child Development, 48, 1314–1321. doi:10.2307/1128489Google Scholar