In recent years, artificial intelligence (AI) systems powered by machine learning have demonstrated human-level capabilities in a variety of games (Brown et al., Reference Brown and Sandholm2018; Jaderberg et al., Reference Jaderberg, Czarnecki, Dunning, Marris, Lever, Castañeda and Graepel2019; Moravčík et al., Reference Moravčík, Schmid, Burch, Lisý, Morrill, Bard and Bowling2017; Silver et al., Reference Strouse, McKee, Botvinick, Hughes and Everett2018), and are increasingly finding applications in the real world (Grigorescu et al., Reference Grigorescu, Trasnea, Cocias and Macesanu2020; Hwangbo et al., Reference Hwangbo, Lee, Dosovitskiy, Bellicoso, Tsounis, Koltun and Hutter2020; Mandhane et al., Reference Mandhane, Zhernov, Rauh, Gu, Wang, Xue and Mann2022). Despite this progress, AI remains specialists, lacking the breadth of competence across diverse tasks, which is characteristic of human intelligence (Chollet, Reference Chollet2019; Hutter, Reference Hutter2000; Legg et al., Reference Legg and Hutter2007). Training large models with large, diverse datasets of interactive behavior appears to be a promising direction for increased generality, both in the language domain (Brown et al., Reference Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal and Amodei2020) and in 3D simulated worlds (Abramson et al., Reference Abramson, Ahuja, Barr, Brussee, Carnevale, Cassin and Zhu2020; Baker et al., Reference Baker, Kanitscheider, Markov, Wu, Powell, McGrew and Mordatch2019; Team, Open Ended Learning et al., Reference Tolstaya, Mahjourian, Downey, Vadarajan, Sapp and Anguelov2021). On the other hand, it is unclear whether a purely data-driven approach can scale toward open-ended intelligence. This motivates interest in algorithms designed for general learning from scratch, such as in emergent communication (Foerster et al., Reference Foerster, Assael, De Freitas and Whiteson2016; Lazaridou et al., Reference Lazaridou, Peysakhovich and Baroni2016) or never-ending learning (Mitchell et al., Reference Mitchell, Cohen, Hruschka, Talukdar, Yang, Betteridge and Welling2018).
The versatility of human social interaction provides a powerful lens through which to study the general capabilities of AI, dating back at least as far as Turing (Reference Turing1950). To partner with diverse individuals across a wide range of tasks necessarily requires flexible modes of expression, adapted on the fly with new conventions and commitments (Bard et al., Reference Bard, Foerster, Chandar, Burch, Lanctot, Song and Bowling2020; Dafoe et al., Reference Dafoe, Hughes, Bachrach, Collins, McKee, Leibo and Graepel2020). Indeed, domain-agnostic social intelligence may even be a sufficient iterative bootstrap to reach individual general intelligence, via cultural evolution (Henrich, Reference Henrich2015; Team, Cultural General Intelligence et al., Reference Bhoopchand, Brownfield, Collister, Lago, Edwards and Zhang2022). The question of how to unleash expression in AI is therefore timely and relevant. Research in this direction could even provide new insights into the evolutionary psychology of language, echoing recent links between AI and neuroscience (Macpherson et al., Reference Macpherson, Churchland, Sejnowski, DiCarlo, Kamitani, Takahashi and Hikida2021; Savage, Reference Scott-Phillips, Kirby and Ritchie2019).
We argue that the target article can be interpreted as a minimal set of socio-cognitive biases that may lead to improved versatility in AI, particularly in interaction with humans. Following the model of reinforcement learning algorithms (Sutton et al., Reference Bhoopchand, Brownfield, Collister, Lago, Edwards and Zhang2018), we identify desirable properties of the environment and of the agent, inspired by the co-evolutionary ecology of human communication. We relate these perspectives to existing approaches in AI, showing that they are relatively underrepresented, and thus provide valuable inspiration for future research.
In the social environment, partner choice ecology is perhaps the main driver for the evolution of ostension and inference. These capabilities underlie all human communication and expression because they enable humans to influence and decode the intentions of others. Partner choice social ecology develops these capabilities whenever humans can select their teammates. In harmony with these observations, partner choice catalyzes artificial learning agents to find the tit-for-tat solution to Prisoner's Dilemma, a strategy that not only plays cooperatively, but also encourages others to cooperate (Anastassacos et al., Reference Anastassacos, Hailes and Musolesi2020). Human feedback can itself be seen as a form of partner choice, when humans choose which AI models they would prefer. Indeed, social interaction with humans in the loop promotes generalizable and robust AI (Carroll et al., Reference Carroll, Shah, Ho, Griffiths, Seshia, Abbeel and Dragan2019; Jaques et al., Reference Jaques, McCleary, Engel, Ha, Bertsch, Eck and Picard2018). These works are a proof of concept that partner choice is important for generally expressive AI, and there is much yet to explore.
The paradigm of emergent communication has shown great promise in training artificial agents, both in situations where incentives are aligned, and in settings requiring negotiation or partial competition (Lazaridou et al., Reference Lazaridou and Baroni2020). Typically, the symbols used for communication do not have any pre-existing semantics. Rather their meaning emerges during training, leading to “code-model” communication (Scott-Phillips, Reference Scott-Phillips2014). Various studies (Bouchacourt et al., Reference Bouchacourt and Baroni2019; Kottur et al., Reference Kottur, Moura, Lee and Batra2017; Resnick et al., Reference Resnick, Gupta, Foerster, Dai and Cho2020) have found that the resulting protocols are not human-interpretable and do not share the structural features of human language. On the other hand, humans are capable of devising generalizable protocols in a zero-shot or few-shot manner (Kirby et al., Reference Kirby, Cornish and Smith2008; Scott-Phillips et al., Reference Silver, Hubert, Schrittwieser, Antonoglou, Lai, Guez and Hassabis2009).
We argue that this lacuna can be resolved if Gricean pragmatics is viewed as a fundamental objective in the design of agent algorithms for emergent communication. There is already promising work in this direction (Eccles et al., Reference Eccles, Bachrach, Lever, Lazaridou and Graepel2019; Kang et al., Reference Kang, Wang and de Melo2020; Pandia et al., Reference Pandia, Cong and Ettinger2021), but pragmatic reasoning is still often regarded as a supplementary bolt-on. Inverting this viewpoint would put inference and ostension at the heart of AI learning algorithms. For instance, an agent with an inverse model of its own policy may use this to infer the communicative intention of others on the fly, a simulation of simulation theory (Gordon, Reference Gordon1986; Heal, Reference Heal1986). Alternatively, one might hope such a model is constructed implicitly during the course of meta-learning across a population of partners (Gupta et al., Reference Gupta, Lanctot and Lazaridou2021; Strouse et al., Reference Sutton and Barto2021). Furthermore, such approaches can easily be combined with data-driven language models (Lowe et al., Reference Lowe, Gupta, Foerster, Kiela and Pineau2020).
There is a close relationship between pragmatic capabilities and theory of mind, a topic that has received some attention in the AI literature (Moreno et al., Reference Moreno, Hughes, McKee, Pires and Weber2021; Rabinowitz et al., Reference Rabinowitz, Perbet, Song, Zhang, Eslami and Botvinick2018). The ability to infer the beliefs of others has been shown to aid convention-building, leading to more generalizable conventions across diverse agents (Foerster et al., Reference Foerster, Song, Hughes, Burch, Dunning, Whiteson and Bowling2019; Hu et al., Reference Hu and Foerster2019, Reference Hu, Lerer, Cui, Pineda, Brown and Foerster2021). Moreover, when agents are incentivized to manipulate the learning of others, they achieve greater success across a variety of games, including when communication is useful (Foerster et al., Reference Foerster, Chen, Al-Shedivat, Whiteson, Abbeel and Mordatch2017; Jaques et al., Reference Jaques, Lazaridou, Hughes, Gülçehre, Ortega, Strouse and Freitas2019; Yang et al., Reference Yang, Li, Farajtabar, Sunehag, Hughes and Zha2020). As they strive for more general and versatile agents, algorithm designers could benefit greatly from understanding the cognitive bases for punishment and teaching in humans.
We conclude with an example of real-world importance. Autonomous vehicles (AVs) have the potential to make transportation safer and more convenient. To optimize for safety in interactive situations, the AV must both predict other road users, and be predictable to other road users. In other words, AVs require ostension and inference capabilities (Dolgov, Reference Dolgov2021). Existing AV systems use data from human drivers to generate human-like plans and to predict other road users' behavior (Sadigh et al., Reference Savage2020; Tolstaya et al., Reference Yang, Li, Farajtabar, Sunehag, Hughes and Zha2021). However, such data may not be enough if the behavior of others is strongly influenced by the autonomous car itself, particularly in previously unseen scenarios. Hence, just as in the target article, we come to the need for a metarepresentational framework, a means of reasoning over the representations that an AV induces in other road users. Already, there exists communicative hardware for AVs (Habibovic et al., Reference Habibovic, Lundgren, Andersson, Klingegård, Lagström, Sirkka and Larsson2020), alongside plans to elicit online human feedback as a guide (Team, Open Ended Learning et al., Reference Tolstaya, Mahjourian, Downey, Vadarajan, Sapp and Anguelov2021). Ostensive communication may be a key ingredient for safe autonomous driving in highly interactive urban environments.
You have
Access
In recent years, artificial intelligence (AI) systems powered by machine learning have demonstrated human-level capabilities in a variety of games (Brown et al., Reference Brown and Sandholm2018; Jaderberg et al., Reference Jaderberg, Czarnecki, Dunning, Marris, Lever, Castañeda and Graepel2019; Moravčík et al., Reference Moravčík, Schmid, Burch, Lisý, Morrill, Bard and Bowling2017; Silver et al., Reference Strouse, McKee, Botvinick, Hughes and Everett2018), and are increasingly finding applications in the real world (Grigorescu et al., Reference Grigorescu, Trasnea, Cocias and Macesanu2020; Hwangbo et al., Reference Hwangbo, Lee, Dosovitskiy, Bellicoso, Tsounis, Koltun and Hutter2020; Mandhane et al., Reference Mandhane, Zhernov, Rauh, Gu, Wang, Xue and Mann2022). Despite this progress, AI remains specialists, lacking the breadth of competence across diverse tasks, which is characteristic of human intelligence (Chollet, Reference Chollet2019; Hutter, Reference Hutter2000; Legg et al., Reference Legg and Hutter2007). Training large models with large, diverse datasets of interactive behavior appears to be a promising direction for increased generality, both in the language domain (Brown et al., Reference Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal and Amodei2020) and in 3D simulated worlds (Abramson et al., Reference Abramson, Ahuja, Barr, Brussee, Carnevale, Cassin and Zhu2020; Baker et al., Reference Baker, Kanitscheider, Markov, Wu, Powell, McGrew and Mordatch2019; Team, Open Ended Learning et al., Reference Tolstaya, Mahjourian, Downey, Vadarajan, Sapp and Anguelov2021). On the other hand, it is unclear whether a purely data-driven approach can scale toward open-ended intelligence. This motivates interest in algorithms designed for general learning from scratch, such as in emergent communication (Foerster et al., Reference Foerster, Assael, De Freitas and Whiteson2016; Lazaridou et al., Reference Lazaridou, Peysakhovich and Baroni2016) or never-ending learning (Mitchell et al., Reference Mitchell, Cohen, Hruschka, Talukdar, Yang, Betteridge and Welling2018).
The versatility of human social interaction provides a powerful lens through which to study the general capabilities of AI, dating back at least as far as Turing (Reference Turing1950). To partner with diverse individuals across a wide range of tasks necessarily requires flexible modes of expression, adapted on the fly with new conventions and commitments (Bard et al., Reference Bard, Foerster, Chandar, Burch, Lanctot, Song and Bowling2020; Dafoe et al., Reference Dafoe, Hughes, Bachrach, Collins, McKee, Leibo and Graepel2020). Indeed, domain-agnostic social intelligence may even be a sufficient iterative bootstrap to reach individual general intelligence, via cultural evolution (Henrich, Reference Henrich2015; Team, Cultural General Intelligence et al., Reference Bhoopchand, Brownfield, Collister, Lago, Edwards and Zhang2022). The question of how to unleash expression in AI is therefore timely and relevant. Research in this direction could even provide new insights into the evolutionary psychology of language, echoing recent links between AI and neuroscience (Macpherson et al., Reference Macpherson, Churchland, Sejnowski, DiCarlo, Kamitani, Takahashi and Hikida2021; Savage, Reference Scott-Phillips, Kirby and Ritchie2019).
We argue that the target article can be interpreted as a minimal set of socio-cognitive biases that may lead to improved versatility in AI, particularly in interaction with humans. Following the model of reinforcement learning algorithms (Sutton et al., Reference Bhoopchand, Brownfield, Collister, Lago, Edwards and Zhang2018), we identify desirable properties of the environment and of the agent, inspired by the co-evolutionary ecology of human communication. We relate these perspectives to existing approaches in AI, showing that they are relatively underrepresented, and thus provide valuable inspiration for future research.
In the social environment, partner choice ecology is perhaps the main driver for the evolution of ostension and inference. These capabilities underlie all human communication and expression because they enable humans to influence and decode the intentions of others. Partner choice social ecology develops these capabilities whenever humans can select their teammates. In harmony with these observations, partner choice catalyzes artificial learning agents to find the tit-for-tat solution to Prisoner's Dilemma, a strategy that not only plays cooperatively, but also encourages others to cooperate (Anastassacos et al., Reference Anastassacos, Hailes and Musolesi2020). Human feedback can itself be seen as a form of partner choice, when humans choose which AI models they would prefer. Indeed, social interaction with humans in the loop promotes generalizable and robust AI (Carroll et al., Reference Carroll, Shah, Ho, Griffiths, Seshia, Abbeel and Dragan2019; Jaques et al., Reference Jaques, McCleary, Engel, Ha, Bertsch, Eck and Picard2018). These works are a proof of concept that partner choice is important for generally expressive AI, and there is much yet to explore.
The paradigm of emergent communication has shown great promise in training artificial agents, both in situations where incentives are aligned, and in settings requiring negotiation or partial competition (Lazaridou et al., Reference Lazaridou and Baroni2020). Typically, the symbols used for communication do not have any pre-existing semantics. Rather their meaning emerges during training, leading to “code-model” communication (Scott-Phillips, Reference Scott-Phillips2014). Various studies (Bouchacourt et al., Reference Bouchacourt and Baroni2019; Kottur et al., Reference Kottur, Moura, Lee and Batra2017; Resnick et al., Reference Resnick, Gupta, Foerster, Dai and Cho2020) have found that the resulting protocols are not human-interpretable and do not share the structural features of human language. On the other hand, humans are capable of devising generalizable protocols in a zero-shot or few-shot manner (Kirby et al., Reference Kirby, Cornish and Smith2008; Scott-Phillips et al., Reference Silver, Hubert, Schrittwieser, Antonoglou, Lai, Guez and Hassabis2009).
We argue that this lacuna can be resolved if Gricean pragmatics is viewed as a fundamental objective in the design of agent algorithms for emergent communication. There is already promising work in this direction (Eccles et al., Reference Eccles, Bachrach, Lever, Lazaridou and Graepel2019; Kang et al., Reference Kang, Wang and de Melo2020; Pandia et al., Reference Pandia, Cong and Ettinger2021), but pragmatic reasoning is still often regarded as a supplementary bolt-on. Inverting this viewpoint would put inference and ostension at the heart of AI learning algorithms. For instance, an agent with an inverse model of its own policy may use this to infer the communicative intention of others on the fly, a simulation of simulation theory (Gordon, Reference Gordon1986; Heal, Reference Heal1986). Alternatively, one might hope such a model is constructed implicitly during the course of meta-learning across a population of partners (Gupta et al., Reference Gupta, Lanctot and Lazaridou2021; Strouse et al., Reference Sutton and Barto2021). Furthermore, such approaches can easily be combined with data-driven language models (Lowe et al., Reference Lowe, Gupta, Foerster, Kiela and Pineau2020).
There is a close relationship between pragmatic capabilities and theory of mind, a topic that has received some attention in the AI literature (Moreno et al., Reference Moreno, Hughes, McKee, Pires and Weber2021; Rabinowitz et al., Reference Rabinowitz, Perbet, Song, Zhang, Eslami and Botvinick2018). The ability to infer the beliefs of others has been shown to aid convention-building, leading to more generalizable conventions across diverse agents (Foerster et al., Reference Foerster, Song, Hughes, Burch, Dunning, Whiteson and Bowling2019; Hu et al., Reference Hu and Foerster2019, Reference Hu, Lerer, Cui, Pineda, Brown and Foerster2021). Moreover, when agents are incentivized to manipulate the learning of others, they achieve greater success across a variety of games, including when communication is useful (Foerster et al., Reference Foerster, Chen, Al-Shedivat, Whiteson, Abbeel and Mordatch2017; Jaques et al., Reference Jaques, Lazaridou, Hughes, Gülçehre, Ortega, Strouse and Freitas2019; Yang et al., Reference Yang, Li, Farajtabar, Sunehag, Hughes and Zha2020). As they strive for more general and versatile agents, algorithm designers could benefit greatly from understanding the cognitive bases for punishment and teaching in humans.
We conclude with an example of real-world importance. Autonomous vehicles (AVs) have the potential to make transportation safer and more convenient. To optimize for safety in interactive situations, the AV must both predict other road users, and be predictable to other road users. In other words, AVs require ostension and inference capabilities (Dolgov, Reference Dolgov2021). Existing AV systems use data from human drivers to generate human-like plans and to predict other road users' behavior (Sadigh et al., Reference Savage2020; Tolstaya et al., Reference Yang, Li, Farajtabar, Sunehag, Hughes and Zha2021). However, such data may not be enough if the behavior of others is strongly influenced by the autonomous car itself, particularly in previously unseen scenarios. Hence, just as in the target article, we come to the need for a metarepresentational framework, a means of reasoning over the representations that an AV induces in other road users. Already, there exists communicative hardware for AVs (Habibovic et al., Reference Habibovic, Lundgren, Andersson, Klingegård, Lagström, Sirkka and Larsson2020), alongside plans to elicit online human feedback as a guide (Team, Open Ended Learning et al., Reference Tolstaya, Mahjourian, Downey, Vadarajan, Sapp and Anguelov2021). Ostensive communication may be a key ingredient for safe autonomous driving in highly interactive urban environments.
Financial support
This work was supported by DeepMind, Waymo, and MILA.
Conflict of interest
Edward Hughes is employed at DeepMind and owns shares in Alphabet. Ekaterina Tolstaya is employed at Waymo and will be granted shares in Waymo. Abhinav Gupta is a student at MILA.