For human-like models, train on human-like tasks

Abstract

Bowers et al. express skepticism about deep neural networks (DNNs) as models of human vision due to DNNs' failures to account for results from psychological research. We argue that to fairly assess DNNs, we must first train them on more human-like tasks which we hypothesize will induce more human-like behaviors and representations.

We agree with Bowers et al. that accounting for results from behavioral experiments should serve as a North Star as we develop models of human vision. But what is a promising path to finding models that perform well on experimental benchmarks? In this commentary, we focus on the role of the task(s) on which models are trained. Zhang, Bengio, Hardt, Recht, and Vinyals (2017) have shown that modern deep neural networks (DNNs) are more than expressive enough to overfit to any classification task on which they are trained. In particular, the authors show that DNNs can learn to classify ImageNet images (Deng et al., 2009) with arbitrarily shuffled labels, demonstrating maximal flexibility with respect to this training set. To introduce a metaphor, our models are like sponges, capable of absorbing whatever information we teach them through the training tasks we present. Thus, when we ask about a model's behavior, we should ask, first, what it was trained to do. Although Bowers et al. take failures of ImageNet-trained models to behave in human-like ways as support for abandoning DNN architectures, we argue that we should instead consider alternative training tasks for DNNs.

Recent work has shown that pushing DNNs to perform well on ImageNet may not, in general, push them to be more human-like. Very high ImageNet performance becomes inversely related to primate neural predictivity (Schrimpf et al., 2018; Schrimpf, 2022), and there is a tradeoff between perceptual scores from human judgments (Kumar, Houlsby, Kalchbrenner, & Cubuk, 2022) and ImageNet performance, and between shape bias and ImageNet performance, when shape bias is modulated by data augmentation (Hermann, Chen, & Kornblith, 2020).

Certainly, humans can categorize the objects they see, but categorization is only a small part of how we process the visual world. Mostly, we use our visual systems to interact with the objects around us, in a closed loop comprising perception, inference, decision making, and action. There are several reasons to believe that training models on similarly embodied and active learning tasks may bring their behavior and representations closer to humans'. First, physically interacting with objects requires detailed perception of their global spatial properties (shape, position, motor affordances, etc.). Arguably, several of the most famous divergences between models and people stem from models' failures to weigh exactly this kind of information. For example, unlike people (Kucker et al., 2019; Landau, Smith, & Jones, 1988), many standard DNNs seem to rely on texture information more than shape (Baker, Lu, Erlikhman, & Kellman, 2018; Geirhos et al., 2019; Hermann et al., 2020). While, empirically, texture seems to be sufficient for good performance on ImageNet, it is unlikely to suffice for embodied navigation or manipulation tasks. In determining how to position oneself to sit in a chair, the shape and position of the chair are far more important than its color or upholstery texture. Similarly, adversarial examples (Nguyen, Yosinski, & Clune, 2015; Szegedy et al., 2013), another often-cited separator of humans and DNNs, arguably arise from models' over-reliance on local pixel patterns at the expense of the global configural information required for embodied interaction. Overall, we hypothesize that existing DNN architectures, if trained to navigate the world and interact with objects in the way that humans do, would be more likely to display human-like visual behavior and representations than they do under current training methods.

Another implication of the Zhang et al. (2017) work is that modern networks are sufficiently large that training them on a 1,000-way classification task on a million images is insufficient to exhaust their capacity, leaving important degrees of freedom governing their generalization performance underconstrained, which allows for deviant phenomena such as adversarial examples of the kind and severity currently observed. As another example of flexibility in how DNNs can learn a classification task, models often learn spurious/shortcut features (Arjovsky, Bottou, Gulrajani, & Lopez-Paz, 2019; Geirhos et al., 2020; McCoy, Pavlick, & Linzen, 2020), for example, using image backgrounds rather than foreground objects (Beery, Van Horn, & Perona, 2018; Xiao, Engstrom, Ilyas, & Madry, 2021), or single diagnostic pixels rather than other image content (Malhotra, Evans, & Bowers, 2020). This brings us to a second argument in favor of embodied training tasks. A dataset of similar size to ImageNet but with a richer, more ecological output space – for example, choosing a physical action and its control parameters, or predicting subsequent frames – would contain a vastly larger amount of information, perhaps more fully constraining the model's behavior.

Existing work validates the impact of training tasks on model behavior and representations. Even when restricted to training on ImageNet images, the training objective and/or data augmentation can affect how well models match human similarity judgments of images (Muttenthaler, Dippel, Linhardt, Vandermeulen, & Kornblith, 2023), categorization patterns (Geirhos et al., 2021), performance on real-time and life-long learning benchmarks (Zhuang et al., 2022), and feature preferences (Hermann et al., 2020), and also how well they predict primate physiology (Zhuang et al., 2021) and human fMRI (Konkle & Alvarez, 2022) data. Still, it is possible to enrich DNN training tasks much further, even for object categorization (Sun, Shrivastava, Singh, & Gupta, 2017).

We have discussed the promise of training embodied, interactive agents in rich, ethologically relevant environments. What efforts have already been made in this direction, and what might they look like in the future? Past work situating a vision system within a simulated agent navigating and interacting with its environment gives promising initial indications that human-like visual behaviors can emerge in this setting (Haber, Mrowca, Wang, Fei-Fei, & Yamins, 2018; Hill et al., 2020; Nayebi et al., 2021; Weihs et al., 2021). The continued development of new, more naturalistic training environments (Gan et al., 2021; Greff et al., 2022; Puig et al., 2018; Savva et al., 2019; Xiang et al., 2020) should support pushing this research program still further toward human-like learning. In addition, state-of-the-art large language models provide a new means of communicating richer tasks to models (Chen et al., 2023), and a new reservoir of human-like knowledge for models to draw on (Brohan et al., 2023). We predict further work in these directions will address shortcomings Bowers et al. identify and yield improved DNN accounts of human vision.