A deep new look at color

Abstract

Bowers et al. counter deep neural networks (DNNs) as good models of human visual perception. From our color perspective we feel their view is based on three misconceptions: A misrepresentation of the state-of-the-art of color perception; the type of model required to move the field forward; and the attribution of shortcomings to DNN research that are already being resolved.

One of the main arguments that Bowers et al. put forth is that deep neural networks (DNNs) classify objects in a fundamentally different manner from humans. However, what Bowers et al. promote as the state-of-the-art in terms of color processing, namely a strict segregation of visual streams for color and shape (Livingstone & Hubel, 1987), is outdated and has repeatedly been rejected (see Garg, Li, Rashid, & Callaway, 2019; Gegenfurtner & Kiper, 2003; Shapley & Hawken, 2011). The fact that line drawings can be recognized quickly does not imply that object processing in humans does not rely on color. On the contrary, boundaries defined by color appear essential for image segmentation in humans (Hansen & Gegenfurtner, 2009, 2017; Shapley & Hawken, 2011). Moreover, the view on how color is represented in the brain has evolved from one of having a single-color center, to one where color-biased regions are found throughout the ventral stream (e.g., Conway, 2018; Gegenfurtner, 2003). While classical algebraic models of color vision have been highly successful in explaining the processing in the cones and color-opponent stages in the eye, higher level cortical processing is still not well understood (for a recent review, see Siuda-Krzywicka & Bartolomeo, 2020). All the evidence points toward an integral role for color in extracting objects, and this perfectly matches the emphasis that DNNs place on objects rather than isolated features.

Bowers et al. emphasize the lack of experimental rigor in testing DNNs compared to testing humans. While we largely agree, it is also important to consider the limitations of a myopic drive to constrain experiments to single-feature manipulations. Reductionist experimental methodology in human research typically diverges greatly from our natural experiences. It biases research toward investigating cognitive functions not necessarily at the core of how our system operates in daily life (e.g., Shamay-Tsoory & Mendelsohn, 2019). While Biederman's (1987) geons may be sufficient for recognizing isolated objects in stereotypical configurations, in daily scenes objects appear in countless varying states (e.g., a cat curled up in a ball on the couch) and other features will gain importance. To avoid a reductionist bias, neuroscientific models need to be grounded in behavior natural to the organism (Krakauer, Ghazanfar, Gomez-Marin, MacIver, & Poeppel, 2017). DNNs may seem misplaced within a classical approach of model-based hypothesis testing, where strongly reductive process models are defined explicitly to test a single hypothesis. However, DNNs are highly suitable for studying how behavior shapes underlying mechanisms. By observing emerging properties in the context of learning specific tasks and manipulating input we can develop improved hypotheses on why colors are represented the way that they are in the human visual system. Subsequently, as DNNs allow for comparisons at many levels, from single trials in psychophysics or electrophysiology experiments, up to derived mental representations in a human observer, they make it possible to study how these mechanisms may be implemented.

Naturally, without considerable overlap with the human visual system we would not consider DNNs adequate models of human vision. However, Bowers et al. focus strongly on discrepancies between humans and DNNs, but neglect important overlap. For color, properties of artificial neurons show great overlap with those in primate visual cortex: Many neurons exhibit double-opponent receptive fields (Flachot & Gegenfurtner, 2018, 2021; Rafegas, Vanrell, Alexandre, & Arias, 2020; Rafegas & Vanrell, 2018), and a moderate functional segregation between color and achromatic information was found at the early stages, corresponding to retino-geniculate processing (Flachot & Gegenfurtner, 2018). On a higher level, DNNs were also shown to outperform classical models in identifying regions of the objects that are highly predictive of human behavioral patterns when discriminating color of naturalistic objects (Ponting, Morimoto, & Smithson, 2023). Moreover, qualitative similarities have been uncovered between DNNs and human participants in color constancy experiments where individual cues known to affect human color constancy were manipulated (Flachot et al., 2022). Finally, in our efforts to uncover why humans adopt a categorical representation of color, we found that DNNs trained specifically for object recognition incorporate a categorical representation of color that is highly similar to that of humans (de Vries, Akbarinia, Flachot, & Gegenfurtner, 2022).

In that study, we strongly focused on translating psychophysical methods to DNNs. We created a match-to-sample task inspired by work on color categorization in pigeons (Wright & Cumming, 1971) using controlled-stimuli and, in secondary experiments, validated their use in the DNN. We also translated the concept of categorical color perception (where colors from different categories are distinguished faster than those from the same category) to the DNN. Our study on color constancy, mentioned above, also included the typical manipulations found in psychophysical studies on the issue. Finally, the studies on neural color tuning translated methods from single-cell recordings in nonhuman primates to DNNs. Together, this introduces important tools to move beyond purely correlational human–DNN comparisons and to investigate where the DNN is similar to the human visual system and where it deviates. Carefully designed experiments allow for collecting response patterns from DNNs through which richer human–DNN comparisons are possible. Notably, our findings are not purely correlational in nature. For example, to establish whether object recognition was important to finding a categorical representation of color, we contrasted an object-trained DNN with the one trained to distinguish artificial from man-made scenes. Importantly, human-like color categories were only found for the object task, indicating that object learning may be crucial in shaping human color categories.

Color and object processing are intricately connected (e.g., Conway, 2018; Witzel & Gegenfurtner, 2018) and understanding color perception will require a model that takes objects into account. DNNs enable us to investigate under what circumstances color phenomena arise and to inspect how they are implemented. Naturally, shortcomings, such as a reliance on correlation-based comparisons and strong divergences from human object processing should be addressed. However, we believe these shortcomings will prove predominantly temporary in nature, as they are already being taken into account in several recent studies. As such, where Bowers et al. take issue with using object-trained DNNs, we see opportunity.