While color vision mediated by rod photoreceptors in dim light is possible (Kelber & Roth, 2006), most animals, including humans, do not see in color at night. This is because their retinas contain only a single class of rod photoreceptors. Many of these same animals have daylight color vision, mediated by multiple classes of cone photoreceptors. We develop a general formulation, based on Bayesian decision theory, to evaluate the efficacy of various retinal photoreceptor mosaics. The formulation evaluates each mosaic under the assumption that its output is processed to optimally estimate the image. It also explicitly takes into account the statistics of the environmental image ensemble. Using the general formulation, we consider the trade-off between monochromatic and dichromatic retinal designs as a function of overall illuminant intensity. We are able to demonstrate a set of assumptions under which the prevalent biological pattern represents optimal processing. These assumptions include an image ensemble characterized by high correlations between image intensities at nearby locations, as well as high correlations between intensities in different wavelength bands. They also include a constraint on receptor photopigment biophysics and/or the information carried by different wavelengths that produces an asymmetry in the signal-to-noise ratio of the output of different receptor classes. Our results thus provide an optimality explanation for the evolution of color vision for daylight conditions and monochromatic vision for nighttime conditions. An additional result from our calculations is that regular spatial interleaving of two receptor classes in a dichromatic retina yields performance superior to that of a retina where receptors of the same class are clumped together.

Efficient coding has been proposed to play an essential role in early visual processing. While several approaches used an objective function to optimize a particular aspect of efficient coding, such as the minimization of mutual information or the maximization of sparseness, we here explore how different estimates of efficient coding in a model with nonlinear dynamics and Hebbian learning determine the similarity of model receptive fields to V1 data with respect to spatial tuning. Our simulation results indicate that most measures of efficient coding correlate with the similarity of model receptive field data to V1 data, that is, optimizing the estimate of efficient coding increases the similarity of the model data to experimental data. However, the degree of the correlation varies with the different estimates of efficient coding, and in particular, the variance in the firing pattern of each cell does not predict a similarity of model and experimental data.

Form vision is traditionally regarded as processing primarily achromatic information. Previous investigations into the statistics of color and luminance in natural scenes have claimed that luminance and chromatic edges are not independent of each other and that any chromatic edge most likely occurs together with a luminance edge of similar strength. Here we computed the joint statistics of luminance and chromatic edges in over 700 calibrated color images from natural scenes. We found that isoluminant edges exist in natural scenes and were not rarer than pure luminance edges. Most edges combined luminance and chromatic information but to varying degrees such that luminance and chromatic edges were statistically independent of each other. Independence increased along successive stages of visual processing from cones via postreceptoral color-opponent channels to edges. The results show that chromatic edge contrast is an independent source of information that can be linearly combined with other cues for the proper segmentation of objects in natural and artificial vision systems. Color vision may have evolved in response to the natural scene statistics to gain access to this independent information.

Historically, the principal function of vision has been to provide the information needed to support action. Visually mediated actions rely on three systems: the gaze system responsible for locating and fixating task-relevant objects, the motor system of the limbs to carry out the task, and the visual system to supply information to the other two. All three systems are under the control of a fourth system, the schema system, which specifies the current task and plans the overall sequence of actions. These four systems have separate but interconnected cortical representations. The way these systems interact in time and space is discussed here in relation to two studies of the gaze changes and manipulations made during two ordinary food preparation tasks. The main conclusions are that complex action sequences consist of a succession of individual object-related actions, each of which typically involve a turn toward the object (if needed), followed by fixation and finally manipulation monitored by vision. Gaze often moves on to the next object just before manipulation is complete. Task-irrelevant objects are hardly ever fixated, implying that the control of fixation comes principally from top-down instructions from the schema system, not bottom-up salience. Single fixations have identifiable functions (locating, directing, guiding, and checking) related to the action to be taken. Several variants of the basic object-related action scheme are discussed, including single-action events in ball sports involving only one anticipatory gaze shift, continuous production loops in text and music reading, and storage–action alternation in copying tasks such as portrait sketching.

Visual cue integration strategies are known to depend on cue reliability and how rapidly the visual system processes incoming information. We investigated whether these strategies also depend on differences in the information demands for different natural tasks. Using two common goal-oriented tasks, prehension and object placement, we determined whether monocular and binocular information influence estimates of three-dimensional (3D) orientation differently depending on task demands. Both tasks rely on accurate 3D orientation estimates, but 3D position is potentially more important for grasping. Subjects placed an object on or picked up a disc in a virtual environment. On some trials, the monocular cues (aspect ratio and texture compression) and binocular cues (e.g., binocular disparity) suggested slightly different 3D orientations for the disc; these conflicts either were present upon initial stimulus presentation or were introduced after movement initiation, which allowed us to quantify how information from the cues accumulated over time. We analyzed the time-varying orientations of subjects’ fingers in the grasping task and those of the object in the object placement task to quantify how different visual cues influenced motor control. In the first experiment, different subjects performed each task, and those performing the grasping task relied on binocular information more when orienting their hands than those performing the object placement task. When subjects in the second experiment performed both tasks in interleaved sessions, binocular cues were still more influential during grasping than object placement, and the different cue integration strategies observed for each task in isolation were maintained. In both experiments, the temporal analyses showed that subjects processed binocular information faster than monocular information, but task demands did not affect the time course of cue processing. How one uses visual cues for motor control depends on the task being performed, although how quickly the information is processed appears to be task invariant.

Despite the extensive investigation of binocular and stereoscopic vision, relatively little is known about its importance in natural visually guided behavior. In this paper, we explored the role of binocular vision when walking over and around obstacles. We monitored eye position during the task as an indicator of the difference between monocular and binocular performances. We found that binocular vision clearly facilitates walking performance. Walkers were slowed by about 10% in monocular vision and raised their foot higher when stepping over obstacles. Although the location and sequence of the fixations did not change in monocular vision, the timing of the fixations relative to the actions was different. Subjects spent proportionately more time fixating the obstacles and fixated longer while guiding foot placement near an obstacle. The data are consistent with greater uncertainty in monocular vision, leading to a greater reliance on feedback in the control of the movements.

Theories of efficient sensory processing have considered the regularities of image properties due to the structure of the environment in order to explain properties of neuronal representations of the visual world. The regularities imposed on the input to the visual system due to the regularities of the active selection process mediated by the voluntary movements of the eyes have been considered to a much lesser degree. This is surprising, given that the active nature of vision is well established. The present article investigates statistics of image features at the center of gaze of human subjects navigating through a virtual environment and avoiding and approaching different objects. The analysis shows that contrast can be significantly higher or lower at fixation location compared to random locations, depending on whether subjects avoid or approach targets. Similarly, significant differences in the distribution of responses of model simple and complex cells between horizontal and vertical orientations are found over timescales of tens of seconds. By clustering the model simple cell responses, it is established that gaze was directed toward three distinct features of intermediate complexity the vast majority of time. Thus, this study demonstrates and quantifies how the visuomotor tasks of approaching and avoiding objects during navigation determine feature statistics of the input to the visual system through the combined influence on body and eye movements.

The neural mechanisms driving perception and saccades during search use information about the target but are also based on an inhibitory surround not present in the target luminance profile (e.g., Eckstein et al., 2007). Here, we ask whether these inhibitory surrounds might reflect a strategy that the brain has adapted to optimize the search for targets in natural scenes. To test this hypothesis, we sought to estimate the best linear template (behavioral receptive field), built from linear combinations of Gabor channels representing V1 simple cells in search for an additive Gaussian target embedded in natural images. Statistically nonstationary and non-Gaussian properties of natural scenes preclude calculation of the best linear template from analytic expressions and require an iterative optimization method such as a virtual evolution via a genetic algorithm. Evolved linear receptive fields built from linear combinations of Gabor functions include substantial inhibitory surround, larger than those found in humans performing target search in white noise. The inhibitory surrounds were robust to changes in the contrast of the signal, generalized to a larger calibrated natural image data set, and tasks in which the signal occluded other objects in the image. We show that channel nonlinearities can have strong effects on the observed linear behavioral receptive field but preserve the inhibitory surrounds. Together, the results suggest that the apparent suboptimality of inhibitory surrounds in human behavioral receptive fields when searching for a target in white noise might reflect a strategy to optimize detection of signals in natural scenes. Finally, we contend that optimized linear detection of spatially compact signals in natural images might be a new possible hypothesis, distinct from decorrelation of visual input and sparse representations (e.g., Graham et al., 2006), to explain the evolution of center–surround organization of receptive fields in early vision.

Correctly interpreting a natural image requires dealing properly with the effects of occlusion, and hence, contour grouping across occlusions is a major component of many natural visual tasks. To better understand the mechanisms of contour grouping across occlusions, we (a) measured the pair-wise statistics of edge elements from contours in natural images, as a function of edge element geometry and contrast polarity, (b) derived the ideal Bayesian observer for a contour occlusion task where the stimuli were extracted directly from natural images, and then (c) measured human performance in the same contour occlusion task. In addition to discovering new statistical properties of natural contours, we found that naïve human observers closely parallel ideal performance in our contour occlusion task. In fact, there was no region of the four-dimensional stimulus space (three geometry dimensions and one contrast dimension) where humans did not closely parallel the performance of the ideal observer (i.e., efficiency was approximately constant over the entire space). These results reject many other contour grouping hypotheses and strongly suggest that the neural mechanisms of contour grouping are tightly related to the statistical properties of contours in natural images.

The human visual system is remarkably tolerant to degradation in image resolution: human performance in scene categorization remains high no matter whether low-resolution images or multimegapixel images are used. This observation raises the question of how many pixels are required to form a meaningful representation of an image and identify the objects it contains. In this article, we show that very small thumbnail images at the spatial resolution of 32 × 32 color pixels provide enough information to identify the semantic category of real-world scenes. Most strikingly, this low resolution permits observers to report, with 80% accuracy, four to five of the objects that the scene contains, despite the fact that some of these objects are unrecognizable in isolation. The robustness of the information available at very low resolution for describing semantic content of natural images could be an important asset to explain the speed and efficiently at which the human brain comprehends the gist of visual scenes.

Modern accounts of color appearance differ in whether they assume that the perceptual primaries (e.g., white and the unique hues of red, green, blue, and yellow) correspond to unique states determined by the spectral sensitivities of the observer or by the spectral statistics of the environment. We examined the interaction between observers and their environments by asking how color perception should vary if appearance depends on fixed responses in a set of color channels, when the sensitivities of these channels are adapted in plausible ways to different environments. Adaptation was modeled as gain changes in the cones and in multiple postreceptoral channels tuned to different directions in color–luminance space. Gains were adjusted so that the average channel responses were equated across two environments or for the same environment during different seasons, based on sets of natural outdoor scenes (Webster et al., 2007). Because of adaptation, even observers with a shared underlying physiology should perceive color in significantly and systematically different ways when they are exposed to and thus adapted by different contexts. These include differences in achromatic settings (owing to variations in the average chromaticity of locations) and differences in perceived hue (because of differences in scene contrasts). Modeling these changes provides a way of simulating how colors might be experienced by individuals in different color environments and provides a measure of how much color appearance might be modulated for a given observer by variations in the environment.

Bayesian decision theory (BDT) is a mathematical framework that allows the experimenter to model ideal performance in a wide variety of visuomotor tasks. The experimenter can use BDT to compute benchmarks for ideal performance in such tasks and compare human performance to ideal. Recently, researchers have asked whether BDT can also be treated as a process model of visuomotor processing. It is unclear what sorts of experiments are appropriate to testing such claims and whether such claims are even meaningful. Any such claim presupposes that observers’ performance is close to ideal, and typical experimental tests involve comparison of human performance to ideal. We argue that this experimental criterion, while necessary, is weak. We illustrate how to achieve near-optimal performance in combining perceptual cues with a process model bearing little resemblance to BDT. We then propose experimental criteria termed transfer criteria that constitute more powerful tests of BDT as a model of perception and action. We describe how recent work in motor control can be viewed as tests of transfer properties of BDT. The transfer properties discussed here comprise the beginning of an operationalization (Bridgman, 1927) of what it means to claim that perception is or is not Bayesian inference (Knill & Richards, 1996). They are particularly relevant to research concerning natural scenes since they assess the ability of the organism to rapidly adapt to novel tasks in familiar environments or carry out familiar tasks in novel environments without learning.