Although the study of visual perception has made more progress in the past 40 years than any other area of cognitive science, there remain major disagreements as to how closely vision is tied to cognition. This target article sets out some of the arguments for both sides (arguments from computer vision, neuroscience, psychophysics, perceptual learning, and other areas of vision science) and defends the position that an important part of visual perception, corresponding to what some people have called early vision, is prohibited from accessing relevant expectations, knowledge, and utilities in determining the function it computes – in other words, it is cognitively impenetrable. That part of vision is complex and involves top-down interactions that are internal to the early vision system. Its function is to provide a structured representation of the 3-D surfaces of objects sufficient to serve as an index into memory, with somewhat different outputs being made available to other systems such as those dealing with motor control. The paper also addresses certain conceptual and methodological issues raised by this claim, such as whether signal detection theory and event-related potentials can be used to assess cognitive penetration of vision.

A distinction is made among several stages in visual processing, including, in addition to the inflexible early-vision stage, a pre-perceptual attention-allocation stage and a post-perceptual evaluation, selection, and inference stage, which accesses long-term memory. These two stages provide the primary ways in which cognition can affect the outcome of visual perception. The paper discusses arguments from computer vision and psychology showing that vision is “intelligent” and involves elements of “problem solving.” The cases of apparently intelligent interpretation sometimes cited in support of this claim do not show cognitive penetration; rather, they show that certain natural constraints on interpretation, concerned primarily with optical and geometrical properties of the world, have been compiled into the visual system. The paper also examines a number of examples where instructions and “hints” are alleged to affect what is seen. In each case it is concluded that the evidence is more readily assimilated to the view that when cognitive effects are found, they have a locus outside early vision, in such processes as the allocation of focal attention and the identification of the stimulus.

Cognitive impenetrability (CI) of a large part of visual perception is taken for granted by those of us in the field of computational vision who attempt to recover descriptions of space using geometry and statistics as tools. These tools clearly point out, however, that CI cannot extend to the level of structured descriptions of object surfaces, as Pylyshyn suggests. The reason is that visual space – the description of the world inside our heads – is a nonEuclidean curved space. As a consequence, the only alternative for a vision system is to develop several descriptions of space–time; these are representations of reduced intricacy and capture partial aspects of objective reality. As such, they make sense in the context of a class of tasks/actions/plans/purposes, and thus cannot be cognitively impenetrable.

This commentary discusses Pylyshyn's model of perceptual processing in the light of the philosophical distinction between the conceptual and the nonconceptual content of perception. Pylyshyn's processing distinction maps onto an important distinction in the phenomenology of visual perception.

According to Pylyshyn, the early visual system is able to categorize perceptual inputs into shape classes based on visual similarity criteria; it is also suggested that written words may be categorized within early vision. This speculation is contradicted by the fact that visually unrelated exemplars of a given letter (e.g., a/A) or word (e.g., read/READ) map onto common visual categories.

Neuroscience studies show many examples of very early modulation of visual cortex responses. It is argued that such early routing is essential for a rapid processing of information by the visual system.

We amplify possible complications to the tidy division between early vision and later categorisation which arise when we consider the perception of human faces. Although a primitive face-detecting system, used for social attention, may indeed be integral to “early vision,” the relationship between this and diverse other uses made of information from faces is far from clear.

Cognitive impenetrability is really two assertions: (1) perception and cognition have access to different knowledge bases; and (2) perception does not use cognitive-style processes. The first leads to the unusual corollary that cognition is itself cognitively impenetrable. The second fails when it is seen to be the claim that reasoning is available only in conscious processing.

Pylyshyn is willing to assume that attention can influence feature integration. We argue that he concedes too much. Feature integration occurs preattentively, except in the case of certain “perverse” displays, such as those used in feature-conjunction searches.

We question the usefulness of Pylyshyn's dichotomy between cognitively penetrable and cognitively impenetrable mechanisms as the basis for his distinction between cognition and early vision. This dichotomy is comparable to others that have been proposed in psychology prompting disputes that by their very nature could not be resolved. This fate is inevitable for Pylyshyn's thesis because of its reliance on internal representations and their interpretation. What is more fruitful in relation to this issue is not a difficult dichotomy, but a different look at perception such as proposed by Gibson (1979).

Pylyshyn uses constraints to solve many of the problems associated with the inverse problem in vision. We are sympathetic to such an approach, and indeed, we think that in many cases constraints allow tract-able solutions to otherwise insoluble problems. We argue, however, that Pylyshyn has been too quick to assume that certain perceptual phenomena can be explained by appealing to constraints embodied in the visual machinery. For several more complex perceptual phenomena it is not clear how one proceeds to look for constraints once simple constraints like rigidity have been abandoned.

Pylyshyn argues that many of the methods used to study perception are too coarse to detect the distinction between perceptual and cognitive processing. We suggest that the reason for this is that the theories used to guide research in perception are at fault. More powerful theories – for instance, computer simulations – will be required to identify where perception ends and where cognition begins.

Two of the premises of Pylyshyn's target article – surface reconstruction as the goal of early vision and inaccessibility of intermediate stages in the process presumably leading to such reconstruction – are questioned and found wanting.

The heuristic value of Pylyshyn's cognitive impenetrability theory is questioned in this commentary, mainly because, as it stands, the key argument cannot be challenged empirically. Pylyshyn requires unambiguous evidence for an effect of cognitive states on early perceptual mechanisms, which is impossible to provide because we can only infer what might happen at these earlier levels of processing on the basis of evidence collected at the post-perceptual stage. Furthermore, the theory that early visual processes cannot be modified by cognitive states implies that it is totally pointless to try to investigate interactions between consciousness and neurosensory processes.

Pylyshyn's argument is very similar to one made in the 1960s to the effect that vision may be influenced by spatial selective attention being directed to distinctive stimulus features, but not by mental set for meaning or membership in an ill-defined category. More recent work points to a special role for spatial attention in determining the contents of perception.

Perception and cognition can be understood either as conscious experience, thought, and behaviour or as bodily functions executed at the level of information processing. Whether or not they are cognitively penetrable depends on the level referred to. Selective attention is the mechanism by which cognition affects perception, theory influences observation and observational reports, culture biases experience, and current knowledge determines what inferences are made. Seeing must be distinguished from seeing as.

A further step in Pylyshyn's discontinuity thesis is to examine the penetrability of haptic (tactual-kinesthetic) perception. The study of the perception of orientation and the “oblique effect” (lower performance in oblique orientations than in vertical–horizontal orientations) in the visual and haptic modalities allows this question to be discussed. We suggest that part of the visual process generating the visual oblique effect is cognitively impenetrable, whereas all haptic processes generating the haptic oblique effect are cognitively penetrable.

Short-term memory, nonattentional task effects and nonspatial extraretinal representations in the visual system are signs of cognitive penetration. All of these have been found physiologically, arguing against the cognitive impenetrability of vision as a whole. Instead, parallel subcircuits in the brain, each subserving a different competency including sensory and cognitive (and in some cases motor) aspects, may have cognitively impenetrable components.

Pylyshyn defends a distinction between early visual perception and cognitive processing. But exactly where should the line between vision and cognition be drawn? Our research on object identification suggests that the construction of an object's visual description is isolated from contextually derived expectations. Moreover, the matching of constructed descriptions to stored descriptions appears to be similarly isolated.

We argue that cognitive penetration in non-early vision extends beyond the special situations considered by Pylyshyn. Many situations which do not involve difficult stimuli or require expert skills nevertheless load on high-level cognitive processes. School learning effects illustrate this point: they provide a way to observe task dissociations which support the discontinuity hypothesis, but they show that the scope of visual cognition in our visual experience is often underestimated.

Pylyshyn could have strengthened his case by avoiding side issues and by taking a sterner, firmer line on the unresolved (and perhaps unresolvable) problems plaguing the sensitivity (d') measure of top-down, cognitive effects, as well as the general (nearly utter!) lack of convincing evidence provided by proponents of the cognitive penetrability of visual perception.