Computational models of intrinsic motivation for curiosity and creativity

Abstract

We link Ivancovsky et al.'s novelty-seeking model (NSM) to computational models of intrinsically motivated behavior and learning. We argue that dissociating different forms of curiosity, creativity, and memory based on the involvement of distinct intrinsic motivations (e.g., surprise and novelty) is essential to empirically test the conceptual claims of the NSM.

Human and animal behavior is driven not only by extrinsically available rewards like food and money but also by various intrinsic motivations, such as the desire to experience novelty or surprise (Gottlieb & Oudeyer, 2018; Modirshanechi et al., 2023b). Curiosity and creativity are two modes of cognitive processing where such intrinsic motivations have a significant influence. Ivancovsky et al.'s novelty-seeking model (NSM) creates a valuable conceptual link between these intuitively related modes, and divides the shared cognitive processes underlying curiosity and creativity into four phases (Ivancovsky et al.). However, the model's high-level conceptual nature makes it challenging to give quantitative explanations and derive experimentally testable hypotheses. To address this problem, we relate each of the four phases of the NSM to computational models of intrinsically motivated behavior and learning. We discuss (i) in which ways computational models support or contradict the NSM's core claims, and illustrate (ii) how computational models make the conceptual explanations and predictions of the NSM empirically testable.

First, the NSM posits that curiosity and creativity share brain networks and mechanisms to detect “novelty,” either in the external space of sensory stimuli (curiosity) or in the internal space of associations (creativity). Second, these shared mechanisms initiate downstream processing of the “novel” stimulus or association (Ivancovsky et al.). However, although Ivancovsky et al. use “novelty” as a general notion, distinct intrinsic motivations contributing to curiosity (e.g., novelty, surprise, information gain) are mathematically well-defined (Barto et al., 2013; Modirshanechi et al., 2022), have different neural signatures (Akiti et al., 2022; Morrens et al., 2020; Xu et al., 2021; Zhang et al., 2022), and are triggered by different statistical regularities of the task or environment (Maheu et al., 2019) (see Modirshanechi et al., 2023a, for a review). For example, novelty signals are triggered by unfamiliar stimuli and situations, both when the unfamiliarity is expected and when it is unexpected (Homann et al., 2022). Surprise signals, on the contrary, arise in the face of unexpected stimuli, both familiar and unfamiliar ones (Zhang et al., 2022). In line with that, different neuromodulatory signals are thought to communicate expected versus unexpected novelty or uncertainty (Schomaker & Meeter, 2015; Yu & Dayan, 2005); and computational models suggest different network mechanisms for the detection of novelty and surprise (Barry & Gerstner, 2024; Schulz et al., 2021). Despite the partial overlap in the processing of novelty and surprise (Zhang et al., 2022), we can thus not simply speak of “novelty” detection as a homogeneous process as assumed in the NSM. When empirically testing shared neural mechanisms of curiosity- and creativity-related signal detection and downstream processing, we should therefore consider how the neural correlates of curiosity and creativity may vary across environments and experimental tasks.

Third, the NSM proposes that both curiosity and creativity require a balance of exploratory and exploitatory states of mind (SoM), and that this balance is mediated by cognitive control processes. This NSM prediction agrees with reinforcement learning-based (RL) models that arbitrate between intrinsic motivations (curiosity/exploratory SoM) and extrinsic motivations (reward/exploitatory SoM) (Modirshanechi et al., 2022; Puigdomènech Badia et al., 2020). Importantly, these RL models quantify the respective contributions of exploration and exploitation to behavior, and allow us to test which mechanisms regulate the trade-off between the exploratory and exploitatory states. For example, a recent model that arbitrates exploration and exploitation based on the agent's reward optimism (Modirshanechi et al., 2022) provides a concrete computational implementation of Ivancovsky et al.'s conceptual links between curiosity and the SoM dimension of openness to experience. We propose that this modeling approach is a useful tool to experimentally validate links between curiosity/creativity and different SoM dimensions as suggested by the NSM.

Lastly, a central component of the NSM is the bidirectional link between memory and curiosity/creativity (Ivancovsky et al.). However, there are different forms of memory and distinct synaptic learning rules that are influenced by intrinsic motivational signals (three-factor learning rules; Gerstner et al., 2018; Lisman et al., 2011). Although we agree with the bidirectional link between curiosity/creativity and memory systems, we propose that the respective memory system with which curiosity and creativity engage could differ (e.g., episodic vs. recognition memory). More importantly, distinct forms of curiosity and creativity may link to different learning rules and roles of memory. For example, novelty is particularly important for initial memory formation (Duszkiewicz et al., 2019; Priestley et al., 2022), whereas surprise, triggered by the violation of known rules and expectations (Barto et al., 2013; Xu et al., 2021; Zhang et al., 2022), might be more important for targeted memory updates (Gershman et al., 2017). Another relevant distinction that the NSM is currently abstracting is between (i) memory systems that support the detection of intrinsic motivational signals and (ii) memory systems that are downstream targets of curiosity/creativity-related signals. These memory systems may – but do not have to – be identical. For example, novelty detection relies on state representations in sensory areas and recognition memory (Bogacz & Brown, 2003; Homann et al., 2022), but downstream novelty signals are also involved in updating semantic or episodic memories (Duszkiewicz et al., 2019; Priestley et al., 2022; Wittmann et al., 2007). To empirically determine how memory is shared by curiosity and creativity, it is necessary to experimentally test how different memory systems are involved at each stage and in each type of curiosity/creativity-related processing.

To conclude, we illustrated how the high-level cognitive NSM framework relates to concrete computational models of intrinsically motivated behavior and learning. Although computational models and the NSM align on the general structure of curiosity- and creativity-related processing, computational models suggest important distinctions within each phase of the NSM. In particular, different forms of curiosity and creativity arising from the contribution of distinct intrinsic motivational signals, like novelty and surprise, could differ in the specifics of how they are detected, signaled to downstream targets, and interacting with memory systems. Linking the NSM to computational models is thus a necessary step to empirically test the NSM's conceptual predictions and gain insights into the neural correlates and network mechanisms underlying curiosity and creativity.