Many of our daily activities are informed by vision, from reading and watching television to driving and sports. At the same time, visual perception during these activities is influenced by our knowledge, expectations, and goals. Visual cognitive neuroscience is the field of research that investigates the neural basis of the bidirectional interaction between visual perception and cognition, between seeing and thinking. Researchers in this field ask how the brain extracts meaningful information from visual input, how this information is represented in the visual cortex, and how visual cortex activity interfaces with cognitive functions such as memory and attention. While these questions have long been of interest to neuroscientists, advances in noninvasive brain imaging techniques have made it possible to study neural processes at the nexus of perception and cognition in humans and across the whole brain. This research has revealed cognitive influences on the functional organization of the visual cortex, has shown how visual cortex activity is modulated by cognitive states, and has started to shed light on the various ways in which the visual cortex supports cognitive tasks even without visual input, such as during visual imagery and visual working memory.

History

Much of our knowledge of the visual system originates from neurophysiological work in animals. Hubel and Wiesel recorded activity from single neurons in the visual cortex of anesthetized cats and monkeys (for a review, see Hubel & Wiesel, 1979). Subsequent neurophysiological investigations revealed organizing principles of the primate visual system, including a hierarchical organization: from neurons with small receptive fields (the portion of the visual field that a neuron responds to) that respond to specific visual features, to neurons with large receptive fields responding to more complex shapes, up to specific categories such as faces and hands (for a review, see Gross, 1994). While the possibility to use cognitive tasks in animals is limited and requires extensive animal training, some neurophysiological studies have also revealed the effects of attention and visual working memory on visual cortex activity (for a review, see Desimone & Duncan, 1995).

In parallel, neuropsychological studies have investigated the neural basis of visual cognition in patients with acquired brain damage, including from strokes or traumatic brain injuries. Patients with selective deficits in visual recognition tasks provided evidence for neural specialization. For example, patients with a selective impairment in recognizing faces after damage to the visual cortex (prosopagnosia; Bodamer, 1947) gave evidence that there were specialized neural mechanisms for face perception that had been compromised by the patients’ injuries. Another influential study reported a patient who was able to accurately grasp but not perceive objects (Goodale et al., 1991), supporting the neural distinction between vision for action (in the dorsal stream—a pathway running from the primary visual cortex to the parietal lobe) and vision for perception (in the ventral stream—a pathway running from the primary visual cortex to the temporal lobe; Goodale & Milner, 1992). Finally, studies have shown that brain damage can lead to impaired visual imagery with intact visual perception and vice versa, informing debates about the shared versus distinct neural mechanisms supporting imagery and perception (Behrmann et al., 1994; Farah, 1988).

Core concepts

Functional specialization in the visual cortex

Human neuroimaging studies have revealed particular circumscribed regions in the visual cortex that respond selectively to specific classes of visual stimuli, including faces [see Face Perception], bodies, scenes, tools, and words (Figure 1; Konkle & Caramazza, 2013; Kanwisher, 2010). These regions have been considered as nodes of domain-specific networks supporting core behaviors such as social cognition (faces, bodies), navigation (scenes), tool use (tools), and reading (words; Peelen & Downing, 2017). The reliable and selective response of these category-selective regions has enabled researchers to test cognitive models of attention, memory, expectation, and visual awareness (Downing et al., 2001). The discovery of category selectivity in the visual cortex has triggered many debates, including about the category specificity of the regions’ functions (Tarr & Gauthier, 2000; Ritchie et al., 2024), their innate versus acquired origins (Arcaro & Livingstone, 2021), the contributions of feedforward and feedback processes (Price & Devlin, 2011), and, more generally, the modularity of cognition (Coltheart, 1999).

Cognitive influences on visual cortex activity

Neural activity in the visual cortex is shaped by cognitive processes, including attention, motivation, emotion, and expectation. Attention, motivation, and emotion enhance visual processing in favor of behaviorally relevant, previously rewarded, or emotionally salient visual stimuli (Kastner & Ungerleider, 2000; Serences, 2008; Vuilleumier, 2005) [see Attention]. By contrast, expectation reduces visual cortex activity to expected stimuli (Summerfield & Egner, 2009). According to predictive processing theories, visual perception is an inference process, in which internal models (informed by knowledge, context, and experience) are generated to match and “explain away” visual input (Friston, 2005; Hohwy, 2016) [see Free Energy Principle, Bayesian Models of Cognition]. On this account, feedforward visual processing reflects the unexplained (i.e., unexpected) visual input, which is used to update the internal model and thereby improve subsequent predictions. Together, the modulatory effects of attention, motivation, emotion, and expectation align visual cortex activity with the observer’s interpretation and subjective perception of the visual environment.

Role of the visual cortex in cognition

The visual cortex can be selectively activated in the absence of visual input, for example, when listening to the names of objects (Martin, 2007). In some visual cortex regions, such internally driven activity can also be observed in people who are born blind, raising the possibility that representations in these areas are not exclusively visual (Bi et al., 2016). Internally-driven visual cortex activity has also been observed during visual mental imagery, visual working memory, retrieval of autobiographical memory, and language comprehension (Pearson, 2019). These findings indicate that the visual cortex may play a broader role in cognition, although its specific and causal contribution to these processes is still debated (Binder & Desai, 2011; Xu, 2017).

Questions, controversies, and new developments

A prominent new development is the modeling of the visual cortex using deep neural networks (DNNs). Research has shown correspondence between the feedforward representations in DNNs and representations in visual cortex (Yamins & DiCarlo, 2016). However, differences remain (Bowers et al., 2022), and DNNs are not yet able to accurately model higher-level cognitive influences on visual cortex activity or tailor visual representations to the tasks humans perform in daily life (e.g., navigation, reading).

Other new developments include relating individual differences in visual cognition to individual differences in brain anatomy and function (Dubois & Adolphs, 2016) and using recent advances in neuroimaging methods to study visual cognition, for example, using high-resolution imaging to test the cortical layer specificity of cognitive influences on visual processing (Lawrence et al., 2019). Finally, researchers have started to use more naturalistic visual stimuli and tasks to make findings more generalizable to daily-life environments (Snow & Culham, 2021; Willems & Peelen, 2021).

Broader connections

Because vision is the dominant input modality for many types of behaviors, the field of visual cognitive neuroscience is closely connected to other areas of cognitive neuroscience, including sensorimotor neuroscience, the neuroscience of language, and social cognitive neuroscience. Furthermore, findings in the field of visual cognitive neuroscience, such as the effects of attention and expectation, may similarly apply to other sensory modalities (e.g., audition). Finally, internally driven activity in the visual cortex has been implicated in multiple cognitive processes, including memory, language, and the representation of concepts [see Concepts]. Thereby, research in visual cognitive neuroscience informs broader philosophical debates about the grounded nature of cognition (Barsalou, 2008) and the distinction between perception and cognition (Pylyshyn, 1999).

Further reading

• Bracci, S., & Op de Beeck, H. P. (2023). Understanding human object vision: A picture is worth a thousand representations. Annual Review of Psychology, 74(1), 113–135. https://doi.org/10.1146/annurev-psych-032720-041031

• Pearson, J. (2019). The human imagination: The cognitive neuroscience of visual mental imagery. Nature Reviews Neuroscience, 20(10), 624–634. https://doi.org/10.1038/s41583-019-0202-9

• Peelen, M. V., Berlot, E., & de Lange, F. P. (2024). Predictive processing of scenes and objects. Nature Reviews Psychology, 3(1), 13–26. https://doi.org/10.1038/s44159-023-00254-0

• Storm, J. F., Klink, P. C., Aru, J., Senn, W., Goebel, R., Pigorini, A., Avanzini, P., Vanduffel, W., Roelfsema, P. R., Massimini, M., Larkum, M. E., & Pennartz, C. M. A. (2024). An integrative, multiscale view on neural theories of consciousness. Neuron, 112(10), 1531–1552. https://doi.org/10.1016/j.neuron.2024.02.004