Toward biologically plausible artificial vision

Abstract

Quilty-Dunn et al. argue that deep convolutional neural networks (DCNNs) optimized for image classification exemplify structural disanalogies to human vision. A different kind of artificial vision – found in reinforcement-learning agents navigating artificial three-dimensional environments – can be expected to be more human-like. Recent work suggests that language-like representations substantially improves these agents’ performance, lending some indirect support to the language-of-thought hypothesis (LoTH).

Image classifiers implemented with deep convolutional neural networks (DCNNs) have been taken by many to tell against language-of-thought (LoT) architectures. Quilty-Dunn et al. argue that this is a mistake. These image classifiers exhibit deep structural disanalogies to human vision, so, whether or not they implement LoT architectures tells us little about human vision. This is perhaps unsurprising, because biological vision is plausibly not optimized solely for image classification (Bowers et al., 2022, p. 10). Would training artificial vision under more ecologically realistic conditions produce a more realistic model of human vision? To make progress on this question, I describe some reinforcement-learning (RL) agents trained to navigate artificial three-dimensional environments on the basis of how things appear from their perspective, and explain why we might expect their vision to be more human-like. Interestingly, language-like representations seem to be especially helpful to these agents. They explore more effectively, more quickly learn novel tasks, and are even facilitated in downstream image classification. These models arguably provide some indirect evidence for the language-of-thought hypothesis (LoTH) about human vision, and may offer some clues as to why LoT architectures arose evolutionarily.

What is biological vision optimized for, and what would artificial vision that was similarly optimized be like? One answer to the first question is that biological vision is optimized for an agent's success in their environment. Success requires a number of competences that vision must contribute to simultaneously. Agents need to effectively explore, learn new behaviors, and act to achieve their goals, all while the environment changes in often surprising ways.

Recent work in RL arguably more closely approximates the optimization problem facing biological agents. Artificial RL agents can learn to do many complex tasks, across a variety of environments – most interestingly, in this context, exploring and pursuing goals in artificial three-dimensional environments like Habitat (Savva et al., 2019), Matterport3D (Chang et al., 2017), Gibson Env (Xia et al., 2018), Franka Kitchen (Gupta et al., 2019), VizDoom (Kempka et al., 2016), Playroom (Tam et al., 2022), and City (Tam et al., 2022). One way of accomplishing this – especially in environments where environmental reward is sparse – is by making novelty intrinsically rewarding. These “curious agents” can learn, without supervision, representations that enable them to perform navigation tasks, interact with objects, and also perform better than baseline in image recognition tasks (Du, Gan, & Isola, 2021). As the authors put it, their agents are “learning a task-agnostic representation for different downstream interactive tasks” (Du et al., 2021, p. 10409).

One challenge these researchers face is how to characterize novelty. Superficial differences in viewing angle or pixel distribution can easily be rated as highly novel, leading to low-level exploration that does not serve learning conducive to achieving goals. A recent innovation is to equip RL agents with “prior knowledge, in the form of abstractions derived from large vision-language models” (Tam et al., 2022, p. 2). Doing so enables the state space over which novelty is defined to be characterized by abstract, semantic categories, such that novelty is defined in task-relevant ways (Mu et al., 2022). This method has been shown to substantially improve performance across a variety of tasks and environments, compared to nonlinguistic ways of characterizing the state space (Mu et al., 2022; Schwartz et al., 2019; Tam et al., 2022). The improvements are especially pronounced for tasks involving relations between objects, for example, “Put an OBJECT on a {bed, tray}” (Tam et al., 2022, p. 2), reminiscent of work on relations reviewed in the target article (Hafri & Firestone, 2021). As the authors note, their training on vision–language representations that encode “objects and relationship” instead of on ImageNet – optimized for classification – should be expected to be more successful (Tam et al., 2022, p. 10).

Why would linguistic categories facilitate performance? One possibility is that language compresses the state space in ways that facilitate successful actions. The semantic categories enshrined in natural language tend to abstract from action-irrelevant variation, and respect action-relevant variation. So, visual processing optimized relative to natural language categories is de facto optimized for action-relevant distinctions. The LoT architecture characteristic of object files and visual working memory seems well-suited to serving this function (though LoT plausibly is importantly different from natural languages; Green, 2020; Mandelbaum et al., 2022). Predicating abstract properties of individual objects in a LoT is poised to guide action, because abstract semantic categories often determine the action affordances available for some individual object, independent of nuisance variation associated with, for example, viewing angle (though viewing angle is plausibly relevant for more fine-grained control tasks; Parisi et al., 2022, p. 6). Such abstract, task-agnostic representations are also able to transfer to new tasks or environments, in which familiar kinds take on novel relevance for action.

These recent innovations in RL arguably offer indirect support for the LoTH as applied to humans. Of course, similar performance can be achieved by distinct underlying competence, and we should not exaggerate how similar even artificial RL agents’ performance actually is to humans at present. Nevertheless, language-like structures appear especially helpful for artificial agents when faced with rather more biologically plausible optimization problems than the one that faces image classifiers. Perhaps an LoT served our ancestors similarly in an evolutionary context. Language-like structures enabled creatures to encode abstract properties in a task-agnostic way, which nevertheless facilitated downstream performance on a wide variety of tasks, as the environment changed. It's not hard to imagine why evolution might see to it that such a system stuck around.