Trial-to-trial, uncertainty-based adjustment of decision boundaries in visual categorization

A. T. Qamar and R. J. Cotton and R. G. George and J. M. Beck and E. Prezhdo and A. Laudano and A. S. Tolias and W. J. Ma

Proc Natl Acad Sci U S A  110  20332-7  (2013)

Categorization is a cornerstone of perception and cognition. Computationally, categorization amounts to applying decision boundaries in the space of stimulus features. We designed a visual categorization task in which optimal performance requires observers to incorporate trial-to-trial knowledge of the level of sensory uncertainty when setting their decision boundaries. We found that humans and monkeys did adjust their decision boundaries from trial to trial as the level of sensory noise varied, with some subjects performing near optimally. We constructed a neural network that implements uncertainty-based, near-optimal adjustment of decision boundaries. Divisive normalization emerges automatically as a key neural operation in this network. Our results offer an integrated computational and mechanistic framework for categorization under uncertainty.

Characterizing the impact of category uncertainty on human auditory categorization behavior

A. M. Gifford and Y. E. Cohen and A. A. Stocker

PLoS Comput Biol  10  e1003715  (2014)

Categorization is an important cognitive process. However, the correct categorization of a stimulus is often challenging because categories can have overlapping boundaries. Whereas perceptual categorization has been extensively studied in vision, the analogous phenomenon in audition has yet to be systematically explored. Here, we test whether and how human subjects learn to use category distributions and prior probabilities, as well as whether subjects employ an optimal decision strategy when making auditory-category decisions. We asked subjects to classify the frequency of a tone burst into one of two overlapping, uniform categories according to the perceived tone frequency. We systematically varied the prior probability of presenting a tone burst with a frequency originating from one versus the other category. Most subjects learned these changes in prior probabilities early in testing and used this information to influence categorization. We also measured each subject's frequency-discrimination thresholds (i.e., their sensory uncertainty levels). We tested each subject's average behavior against variations of a Bayesian model that either led to optimal or sub-optimal decision behavior (i.e. probability matching). In both predicting and fitting each subject's average behavior, we found that probability matching provided a better account of human decision behavior. The model fits confirmed that subjects were able to learn category prior probabilities and approximate forms of the category distributions. Finally, we systematically explored the potential ways that additional noise sources could influence categorization behavior. We found that an optimal decision strategy can produce probability-matching behavior if it utilized non-stationary category distributions and prior probabilities formed over a short stimulus history. Our work extends previous findings into the auditory domain and reformulates the issue of categorization in a manner that can help to interpret the results of previous research within a generative framework.

Categorical speech representation in human superior temporal gyrus

E. F. Chang and J. W. Rieger and K. Johnson and M. S. Berger and N. M. Barbaro and R. T. Knight

Nat Neurosci  13  1428-32  (2010)

Speech perception requires the rapid and effortless extraction of meaningful phonetic information from a highly variable acoustic signal. A powerful example of this phenomenon is categorical speech perception, in which a continuum of acoustically varying sounds is transformed into perceptually distinct phoneme categories. We found that the neural representation of speech sounds is categorically organized in the human posterior superior temporal gyrus. Using intracranial high-density cortical surface arrays, we found that listening to synthesized speech stimuli varying in small and acoustically equal steps evoked distinct and invariant cortical population response patterns that were organized by their sensitivities to critical acoustic features. Phonetic category boundaries were similar between neurometric and psychometric functions. Although speech-sound responses were distributed, spatially discrete cortical loci were found to underlie specific phonetic discrimination. Our results provide direct evidence for acoustic-to-higher order phonetic level encoding of speech sounds in human language receptive cortex.

Frontal eye field neurons signal changes in decision criteria

V. P. Ferrera and M. Yanike and C. Cassanello

Nat Neurosci  12  1458-62  (2009)

Flexible links between sensory stimuli and behavioral responses underlie many cognitive processes. One process that contributes to flexible decision-making is categorization. Some categories are innate or overlearned, but, in many cases, category boundaries represent flexible decision criteria that can shift on the fly to adapt to changes in the environment. The ability to shift category boundaries allows decision-making to adapt to changing circumstances. We found that monkeys were able to switch rapidly between two category boundaries when classifying the speed of a moving dot pattern and that neurons in monkey frontal eye field (FEF) changed their activity when the boundary changed. The responses of a subpopulation of FEF neurons that were sensitive to both stimulus and boundary speed were used to classify the stimuli as accurately as the monkeys' performance.

Neural correlates of categorical perception in learned vocal communication

J. F. Prather and S. Nowicki and R. C. Anderson and S. Peters and R. Mooney

Nat Neurosci  12  221-8  (2009)

The division of continuously variable acoustic signals into discrete perceptual categories is a fundamental feature of vocal communication, including human speech. Despite the importance of categorical perception to learned vocal communication, the neural correlates underlying this phenomenon await identification. We found that individual sensorimotor neurons in freely behaving swamp sparrows expressed categorical auditory responses to changes in note duration, a learned feature of their songs, and that the neural response boundary accurately predicted the categorical perceptual boundary measured in field studies of the same sparrow population. Furthermore, swamp sparrow populations that learned different song dialects showed different categorical perceptual boundaries that were consistent with the boundary being learned. Our results extend the analysis of the neural basis of perceptual categorization into the realm of vocal communication and advance the learned vocalizations of songbirds as a model for investigating how experience shapes categorical perception and the activity of categorically responsive neurons.

Forming classes by stimulus frequency: behavior and theory

O. Rosenthal and S. Fusi and S. Hochstein

Proc Natl Acad Sci U S A  98  4265-70  (2001)

Visual classification is the way we relate to different images in our environment as if they were the same, while relating differently to other collections of stimuli (e.g., human vs. animal faces). It is still not clear, however, how the brain forms such classes, especially when introduced with new or changing environments. To isolate a perception-based mechanism underlying class representation, we studied unsupervised classification of an incoming stream of simple images. Classification patterns were clearly affected by stimulus frequency distribution, although subjects were unaware of this distribution. There was a common bias to locate class centers near the most frequent stimuli and their boundaries near the least frequent stimuli. Responses were also faster for more frequent stimuli. Using a minimal, biologically based neural-network model, we demonstrate that a simple, self-organizing representation mechanism based on overlapping tuning curves and slow Hebbian learning suffices to ensure classification. Combined behavioral and theoretical results predict large tuning overlap, implicating posterior infero-temporal cortex as a possible site of classification.

Increases in functional connectivity between prefrontal cortex and striatum during category learning

E. G. Antzoulatos and E. K. Miller

Neuron  83  216-25  (2014)

Functional connectivity between the prefrontal cortex (PFC) and striatum (STR) is thought critical for cognition and has been linked to conditions like autism and schizophrenia. We recorded from multiple electrodes in PFC and STR while monkeys acquired new categories. Category learning was accompanied by an increase in beta band synchronization of LFPs between, but not within, the PFC and STR. After learning, different pairs of PFC-STR electrodes showed stronger synchrony for one or the other category, suggesting category-specific functional circuits. This category-specific synchrony was also seen between PFC spikes and STR LFPs, but not the reverse, reflecting the direct monosynaptic connections from the PFC to STR. However, causal connectivity analyses suggested that the polysynaptic connections from STR to the PFC exerted a stronger overall influence. This supports models positing that the basal ganglia "train" the PFC. Category learning may depend on the formation of functional circuits between the PFC and STR.

Differences between neural activity in prefrontal cortex and striatum during learning of novel abstract categories

E. G. Antzoulatos and E. K. Miller

Neuron  71  243-9  (2011)

Learning to classify diverse experiences into meaningful groups, like categories, is fundamental to normal cognition. To understand its neural basis, we simultaneously recorded from multiple electrodes in lateral prefrontal cortex and dorsal striatum, two interconnected brain structures critical for learning. Each day, monkeys learned to associate novel abstract, dot-based categories with a right versus left saccade. Early on, when they could acquire specific stimulus-response associations, striatum activity was an earlier predictor of the corresponding saccade. However, as the number of exemplars increased and monkeys had to learn to classify them, PFC activity began to predict the saccade associated with each category before the striatum. While monkeys were categorizing novel exemplars at a high rate, PFC activity was a strong predictor of their corresponding saccade early in the trial before the striatal neurons. These results suggest that striatum plays a greater role in stimulus-response association and PFC in abstraction of categories.

PFC neurons reflect categorical decisions about ambiguous stimuli

J. E. Roy and T. J. Buschman and E. K. Miller

J Cogn Neurosci  26  1283-91  (2014)

We examined whether PFC neuron activity reflects categorical decisions in monkeys categorizing ambiguous stimuli. A morphing system was used to systematically vary stimulus shape and precisely define category boundaries. Ambiguous stimuli were centered on a category boundary, that is, they were a mix of 50% of two prototypes and therefore had no category information, so monkeys guessed at their category membership. We found that the monkeys' trial-by-trial decision about the category membership of an ambiguous image was reflected in PFC activity. Activity to the same ambiguous image differed significantly, depending on which category the monkey had assigned it to. This effect only occurred when that scheme was behaviorally relevant. These indicate that PFC activity reflects categorical decisions.

A comparison of lateral and medial intraparietal areas during a visual categorization task

S. K. Swaminathan and N. Y. Masse and D. J. Freedman

J Neurosci  33  13157-70  (2013)

Categorization is essential for interpreting sensory stimuli and guiding our actions. Recent studies have revealed robust neuronal category representations in the lateral intraparietal area (LIP). Here, we examine the specialization of LIP for categorization and the roles of other parietal areas by comparing LIP and the medial intraparietal area (MIP) during a visual categorization task. MIP is involved in goal-directed arm movements and visuomotor coordination but has not been implicated in non-motor cognitive functions, such as categorization. As expected, we found strong category encoding in LIP. Interestingly, we also observed category signals in MIP. However, category signals were stronger and appeared with a shorter latency in LIP than MIP. In this task, monkeys indicated whether a test stimulus was a category match to a previous sample with a manual response. Test-period activity in LIP showed category encoding and distinguished between matches and non-matches. In contrast, MIP primarily reflected the match/non-match status of test stimuli, with a strong preference for matches (which required a motor response). This suggests that, although category representations are distributed across parietal cortex, LIP and MIP play distinct roles: LIP appears more involved in the categorization process itself, whereas MIP is more closely tied to decision-related motor actions.

A proposed common neural mechanism for categorization and perceptual decisions

D. J. Freedman and J. A. Assad

Nat Neurosci  14  143-6  (2011)

One of the most fascinating issues in neuroscience is how the brain makes decisions. Recent evidence points to the parietal cortex as an important locus for certain kinds of decisions. Because parietal neurons are also involved in movements, it has been proposed that decisions are encoded in an intentional, action-based framework based on the movements used to report decisions. An alternative or complementary view is that decisions represent more abstract information not linked to movements per se. Parallel experiments on categorization suggest that parietal neurons can indeed represent abstract categorical outcomes that are not linked to movements. This could provide a unified or complementary view of how the brain decides and categorizes.

Global neural pattern similarity as a common basis for categorization and recognition memory

T. Davis and G. Xue and B. C. Love and A. R. Preston and R. A. Poldrack

J Neurosci  34  7472-84  (2014)

Familiarity, or memory strength, is a central construct in models of cognition. In previous categorization and long-term memory research, correlations have been found between psychological measures of memory strength and activation in the medial temporal lobes (MTLs), which suggests a common neural locus for memory strength. However, activation alone is insufficient for determining whether the same mechanisms underlie neural function across domains. Guided by mathematical models of categorization and long-term memory, we develop a theory and a method to test whether memory strength arises from the global similarity among neural representations. In human subjects, we find significant correlations between global similarity among activation patterns in the MTLs and both subsequent memory confidence in a recognition memory task and model-based measures of memory strength in a category learning task. Our work bridges formal cognitive theories and neuroscientific models by illustrating that the same global similarity computations underlie processing in multiple cognitive domains. Moreover, by establishing a link between neural similarity and psychological memory strength, our findings suggest that there may be an isomorphism between psychological and neural representational spaces that can be exploited to test cognitive theories at both the neural and behavioral levels.

Supervised learning with decision margins in pools of spiking neurons

C. Le Mouel and K. D. Harris and P. Yger

J Comput Neurosci      (2014)

Learning to categorise sensory inputs by generalising from a few examples whose category is precisely known is a crucial step for the brain to produce appropriate behavioural responses. At the neuronal level, this may be performed by adaptation of synaptic weights under the influence of a training signal, in order to group spiking patterns impinging on the neuron. Here we describe a framework that allows spiking neurons to perform such "supervised learning", using principles similar to the Support Vector Machine, a well-established and robust classifier. Using a hinge-loss error function, we show that requesting a margin similar to that of the SVM improves performance on linearly non-separable problems. Moreover, we show that using pools of neurons to discriminate categories can also increase the performance by sharing the load among neurons.

Learning selective top-down control enhances performance in a visual categorization task

M. Pannunzi and G. Gigante and M. Mattia and G. Deco and S. Fusi and P. Del Giudice

J Neurophysiol  108  3124-37  (2012)

We model the putative neuronal and synaptic mechanisms involved in learning a visual categorization task, taking inspiration from single-cell recordings in inferior temporal cortex (ITC). Our working hypothesis is that learning the categorization task involves both bottom-up, ITC to prefrontal cortex (PFC), and top-down (PFC to ITC) synaptic plasticity and that the latter enhances the selectivity of the ITC neurons encoding the task-relevant features of the stimuli, thereby improving the signal-to-noise ratio. We test this hypothesis by modeling both areas and their connections with spiking neurons and plastic synapses, ITC acting as a feature-selective layer and PFC as a category coding layer. This minimal model gives interesting clues as to properties and function of the selective feedback signal from PFC to ITC that help solving a categorization task. In particular, we show that, when the stimuli are very noisy because of a large number of nonrelevant features, the feedback structure helps getting better categorization performance and decreasing the reaction time. It also affects the speed and stability of the learning process and sharpens tuning curves of ITC neurons. Furthermore, the model predicts a modulation of neural activities during error trials, by which the differential selectivity of ITC neurons to task-relevant and task-irrelevant features diminishes or is even reversed, and modulations in the time course of neural activities that appear when, after learning, corrupted versions of the stimuli are input to the network.

Reward modulates the neural dynamics of early visual category processing

T. Apitz and N. Bunzeck

Neuroimage  63  1614-22  (2012)

Converging evidence suggests that visual brain regions are part of a widespread network that signals forthcoming reward. However, the precise temporal dynamics underlying the interaction between reward and visual information processing remain unclear. To further investigate this issue, we used magnetoencephalography (MEG) in combination with two versions of a face/scene discrimination task followed by a recognition memory test. In experiment 1, the distinction between faces and scenes was associated with monetary reward prospect, whereas in experiment 2 subjects distinguished between both categories in the absence of reward. In both experiments characteristic neural category effects (i.e., differences between faces and scenes) were observed in the event-related magnetic fields (ERF) at ~100 ms (M100) and ~170 ms (M170) after stimulus onset. Importantly, both ERF components (M100 and M170) were amplified in the context of reward (i.e., experiment 1) and this interaction could be source localized to the lateral occipital cortex (~100 ms) and fusiform gyrus (~170 ms). Furthermore, neural effects of reward prediction emerged over frontal sensors at ~300 ms after stimulus onset which reliably correlated with subsequent recognition memory performance. These results demonstrate that reward motivation can modulate early neural computations of complex visual information, possibly by tuning sensory neurons within the visual cortex.

Category representation and generalization in the prefrontal cortex

X. Pan and M. Sakagami

Eur J Neurosci  35  1083-91  (2012)

Categorization is a function of the brain that serves to group together items and events in our environments. Here we review the following important issues related to category representation and generalization: namely, where categories are presented in the brain, and how the brain utilizes categorical membership to generate new information. Accumulated experimental evidence shows that the prefrontal cortex (PFC) plays a critical role in category formation and generalization. We propose that prefrontal neurons abstract the commonality beyond individual stimuli, and categorize these based on their common meaning by ignoring their physical properties and learning to represent the boundaries between behaviorally significant categories. We also claim that a subgroup of prefrontal neurons simultaneously receives the category-related information and specific property information (e.g. reward) associated with an exemplar, to form a category-based representation of that property, and propagates it among stimuli of the same category, possibly reflecting a neural basis for category generalization in the PFC. These results suggest that the PFC is involved in representing abstract rules, and generating new information on the basis of previously acquired knowledge.

Behavioral and anatomical consequences of early versus late symbol training in macaques

K. Srihasam and J. B. Mandeville and I. A. Morocz and K. J. Sullivan and M. S. Livingstone

Neuron  73  608-19  (2012)

Distinct brain regions, reproducible from one person to the next, are specialized for processing different kinds of human expertise, such as face recognition and reading. Here, we explore the relationship between age of learning, learning ability, and specialized brain structures. Specifically, we ask whether the existence of reproducible cortical domains necessarily means that certain abilities are innate, or innately easily learned, or whether reproducible domains can be formed, or refined, by interactions between genetic programs and common early experience. Functional MRI showed that intensive early, but not late, experience caused the formation of category-selective regions in macaque temporal lobe for stimuli never naturally encountered by monkeys. And behaviorally, early training produced more fluent processing of these stimuli than the same training in adults. One explanation for these results is that in higher cortical areas, as in early sensory areas, experience drives functional clustering and functional clustering determines how that information is processed.

Perceptual classification in a rapidly changing environment

C. Summerfield and T. E. Behrens and E. Koechlin

Neuron  71  725-36  (2011)

Humans and monkeys can learn to classify perceptual information in a statistically optimal fashion if the functional groupings remain stable over many hundreds of trials, but little is known about categorization when the environment changes rapidly. Here, we used a combination of computational modeling and functional neuroimaging to understand how humans classify visual stimuli drawn from categories whose mean and variance jumped unpredictably. Models based on optimal learning (Bayesian model) and a cognitive strategy (working memory model) both explained unique variance in choice, reaction time, and brain activity. However, the working memory model was the best predictor of performance in volatile environments, whereas statistically optimal performance emerged in periods of relative stability. Bayesian and working memory models predicted decision-related activity in distinct regions of the prefrontal cortex and midbrain. These findings suggest that perceptual category judgments, like value-guided choices, may be guided by multiple controllers.

Monkeys quickly learn and generalize visual categories without lateral prefrontal cortex

T. Minamimoto and R. C. Saunders and B. J. Richmond

Neuron  66  501-7  (2010)

Categorization is a basic mental process that helps individuals distinguish among groups of negative and positive objects, e.g., poisons and nutrients, or predators and prey. Monkey experiments have suggested that lateral prefrontal cortex (LPFC) participates in learning and processing visual categories. However, in humans category specific visual agnosia follows inferior temporal cortex but not LPFC damage. Here, we use a new behavioral approach to show that both normal monkeys and those with bilateral removal of LPFC learn and generalize perceptual categories of related visual stimuli rapidly without explicit instruction. These results strongly indicate that visual categorization occurs at some earlier stage of feed-forward processing, presumably in temporal cortex, without top-down information from LPFC.

Prefrontal cortex activity during flexible categorization

J. E. Roy and M. Riesenhuber and T. Poggio and E. K. Miller

J Neurosci  30  8519-28  (2010)

Items are categorized differently depending on the behavioral context. For instance, a lion can be categorized as an African animal or a type of cat. We recorded lateral prefrontal cortex (PFC) neural activity while monkeys switched between categorizing the same image set along two different category schemes with orthogonal boundaries. We found that each category scheme was largely represented by independent PFC neuronal populations and that activity reflecting a category distinction was weaker, but not absent, when that category was irrelevant. We suggest that the PFC represents competing category representations independently to reduce interference between them.

Representation of multiple, independent categories in the primate prefrontal cortex

J. A. Cromer and J. E. Roy and E. K. Miller

Neuron  66  796-807  (2010)

Neural correlates of visual categories have been previously identified in the prefrontal cortex (PFC). However, whether individual neurons can represent multiple categories is unknown. Varying degrees of generalization versus specialization of neurons in the PFC have been theorized. We recorded from lateral PFC neural activity while monkeys switched between two different and independent categorical distinctions (Cats versus Dogs, Sports Cars versus Sedans). We found that many PFC neurons reflected both categorical distinctions. In fact, these multitasking neurons had the strongest category effects. This stands in contrast to our lab's recent report that monkeys switching between competing categorical distinctions (applied to the same stimulus set) showed independent representations. We suggest that cognitive demands determine whether PFC neurons function as category "multitaskers."

Learning shapes the representation of behavioral choice in the human brain

S. Li and S. D. Mayhew and Z. Kourtzi

Neuron  62  441-52  (2009)

Making successful decisions under uncertainty due to noisy sensory signals is thought to benefit from previous experience. However, the human brain mechanisms that mediate flexible decisions through learning remain largely unknown. Comparing behavioral choices of human observers with those of a pattern classifier based on multivoxel single-trial fMRI signals, we show that category learning shapes processes related to decision variables in frontal and higher occipitotemporal regions rather than signal detection or response execution in primary visual or motor areas. In particular, fMRI signals in prefrontal regions reflect the observers' behavioral choice according to the learned decision criterion only in the context of the categorization task. In contrast, higher occipitotemporal areas show learning-dependent changes in the representation of perceived categories that are sustained after training independent of the task. These findings demonstrate that learning shapes selective representations of sensory readout signals in accordance with the decision criterion to support flexible decisions.

Reward prediction based on stimulus categorization in primate lateral prefrontal cortex

X. Pan and K. Sawa and I. Tsuda and M. Tsukada and M. Sakagami

Nat Neurosci  11  703-12  (2008)

To adapt to changeable or unfamiliar environments, it is important that animals develop strategies for goal-directed behaviors that meet the new challenges. We used a sequential paired-association task with asymmetric reward schedule to investigate how prefrontal neurons integrate multiple already-acquired associations to predict reward. Two types of reward-related neurons were observed in the lateral prefrontal cortex: one type predicted reward independent of physical properties of visual stimuli and the other encoded the reward value specific to a category of stimuli defined by the task requirements. Neurons of the latter type were able to predict reward on the basis of stimuli that had not yet been associated with reward, provided that another stimulus from the same category was paired with reward. The results suggest that prefrontal neurons can represent reward information on the basis of category and propagate this information to category members that have not been linked directly with any experience of reward.

Decoding the brain's algorithm for categorization from its neural implementation

M. L. Mack and A. R. Preston and B. C. Love

Curr Biol  23  2023-7  (2013)

Acts of cognition can be described at different levels of analysis: what behavior should characterize the act, what algorithms and representations underlie the behavior, and how the algorithms are physically realized in neural activity [1]. Theories that bridge levels of analysis offer more complete explanations by leveraging the constraints present at each level [2-4]. Despite the great potential for theoretical advances, few studies of cognition bridge levels of analysis. For example, formal cognitive models of category decisions accurately predict human decision making [5, 6], but whether model algorithms and representations supporting category decisions are consistent with underlying neural implementation remains unknown. This uncertainty is largely due to the hurdle of forging links between theory and brain [7-9]. Here, we tackle this critical problem by using brain response to characterize the nature of mental computations that support category decisions to evaluate two dominant, and opposing, models of categorization. We found that brain states during category decisions were significantly more consistent with latent model representations from exemplar [5] rather than prototype theory [10, 11]. Representations of individual experiences, not the abstraction of experiences, are critical for category decision making. Holding models accountable for behavior and neural implementation provides a means for advancing more complete descriptions of the algorithms of cognition.

Experience-dependent sharpening of visual shape selectivity in inferior temporal cortex.

D. J. Freedman and M. Riesenhuber and T. Poggio and E. K. Miller

Cereb Cortex  16  1631--1644  (2006)

Whereas much is known about the visual shape selectivity of neurons in the inferior temporal cortex (ITC), less is known about the role of visual learning in the development and refinement of ITC shape selectivity. To address this, we trained monkeys to perform a visual categorization task with a parametric set of highly familiar stimuli. During training, the stimuli were always presented at the same orientation. In this experiment, we recorded from ITC neurons while monkeys viewed the trained stimuli in addition to image-plane rotated versions of those stimuli. We found that, concomitant with the monkeys' behavioral performance, neuronal stimulus selectivity was stronger for stimuli presented at the trained orientation than for rotated versions of the same stimuli. We also recorded from ITC neurons while monkeys viewed sets of novel and familiar (but not explicitly trained) randomly chosen complex stimuli. We again found that ITC stimulus selectivity was sharper for familiar than novel stimuli, suggesting that enhanced shape tuning in ITC can arise for both passively experienced and explicitly trained stimuli.

Effects of task demands on the responses of color-selective neurons in the inferior temporal cortex.

K. Koida and H. Komatsu

Nat Neurosci  10  108-16  (2007)


Categorization and fine discrimination are two different functions in visual perception, and we can switch between these two functions depending on the situation or task demands. To explore how visual cortical neurons behave in such situations, we recorded the activities of color-selective neurons in the inferior temporal (IT) cortex of two monkeys trained to perform a color categorization task, a color discrimination task and a simple fixation task. Many IT neurons changed their activity depending upon the task, although color selectivity was well conserved. A majority of neurons showed stronger responses during the categorization task. Moreover, for the population of IT neurons as a whole, signals contributing to performing the categorization task were enhanced. These results imply that judgment of color category by color-selective IT neurons is facilitated during the categorization task and suppressed during the discrimination task as a consequence of task-dependent modulation of their activities.

Object recognition and segmentation by a fragment-based hierarchy.

S. Ullman

Trends Cogn Sci  11  58-64  (2007)


How do we learn to recognize visual categories, such as dogs and cats? Somehow, the brain uses limited variable examples to extract the essential characteristics of new visual categories. Here, I describe an approach to category learning and recognition that is based on recent computational advances. In this approach, objects are represented by a hierarchy of fragments that are extracted during learning from observed examples. The fragments are class-specific features and are selected to deliver a high amount of information for categorization. The same fragments hierarchy is then used for general categorization, individual object recognition and object-parts identification. Recognition is also combined with object segmentation, using stored fragments, to provide a top-down process that delineates object boundaries in complex cluttered scenes. The approach is computationally effective and provides a possible framework for categorization, recognition and segmentation in human vision.

Synaptic computation underlying probabilistic inference

A. Soltani and X.-J. Wang

Nat Neurosci  13  112-9  (2010)


We propose that synapses may be the workhorse of the neuronal computations that underlie probabilistic reasoning. We built a neural circuit model for probabilistic inference in which information provided by different sensory cues must be integrated and the predictive powers of individual cues about an outcome are deduced through experience. We found that bounded synapses naturally compute, through reward-dependent plasticity, the posterior probability that a choice alternative is correct given that a cue is presented. Furthermore, a decision circuit endowed with such synapses makes choices on the basis of the summed log posterior odds and performs near-optimal cue combination. The model was validated by reproducing salient observations of, and provides insights into, a monkey experiment using a categorization task. Our model thus suggests a biophysical instantiation of the Bayesian decision rule, while predicting important deviations from it similar to the 'base-rate neglect' observed in human studies when alternatives have unequal prior probabilities.

Two modulatory effects of attention that mediate object categorization in human cortex.

G. Rees and R. Frackowiak and C. Frith

Science  275  835-8  (1997)


Attentional modulation of cortical activity was examined by varying the rate of visual stimuli in object categorization tasks according to single and conjoined features. Activation of dorsolateral frontal cortex was independent of the stimulus presentation rate and elicited by the participant's attention to conjoined compared with single features. Several cortical regions showed attentionally modulated activity. In inferior temporal cortex, modulation was due to an additional bias signal underlying normal rate-correlated activity. In two other regions (premotor cortex and cerebellum), attention modified the correlation of activity and the stimulus presentation rate. Attentional effects in the human cortex are expressed by at least two physiologically distinct mechanisms acting on spatially distributed areas.

Role of primary somatic sensory cortex in the categorization of tactile stimuli: effects of lesions.

A. Zainos and H. Merchant and A. Hernandez and E. Salinas and R. Romo

Exp Brain Res  115  357-60  (1997)


We lesioned the right primary somatic sensory (S1) cortex in two monkeys trained to categorize the speed of moving tactile stimuli. Animals performed the task by pressing with the right hand one of two target switches to indicate whether the speed of a probe moving across the glabrous skin of the left hand was low or high. Sensory performance was evaluated with psychometric techniques and motor behavior was monitored by measuring the reaction (RT) and movement (MT) times before the experiment and throughout the 60 days after the ablation of SI cortex. After the lesion, there was a slight increase in the RTs but no change in the MTs, indicating that removal of SI cortex did not affect the animals' capacity to detect the stimuli. However, monkeys lost their ability to categorize the stimulus speeds. This effect was observed from the 1st day after the lesion until the end of the study. We conclude that somatosensory areas outside SI can by themselves process tactile information in a limited way and that the extraction of higher-order features that takes place during the categorization task requires the intervention of SI cortex.

The representation of shape in the context of visual object categorization tasks.

H. Op de Beeck and E. Beatse and J. Wagemans and S. Sunaert and P. Van Hecke

Neuroimage  12  28-40  (2000)


To investigate the role of human fusiform gyrus in shape processing, we determined the effect of shape degradation on BOLD contrast in this region with fMRI during three tasks requiring subjects to determine either whether two successively presented nonsense shapes had the same global orientation (OR task); whether two successively presented meaningful objects belonged to the same basic level category (CAT task); or whether two successively presented objects represented the same exemplar of a category (EX task). On the behavioral level, shape degradation by locally shifting the pixels constituting the lines of stimuli had no effect on performance in the OR task, while it was detrimental to performance in the CAT and EX tasks. In comparison to the OR task, both the CAT and EX tasks were associated with activations in the occipitotemporal and parietal cortex. When shape degradation was applied, activation in the middle fusiform gyrus was reduced in all tasks. The occurrence of this effect in the OR task indicates that it is independent of memory representations. The persistence of the effect in both tasks that showed a behavioral effect of degradation suggests that it does not reflect the amount of shape processing performed on the stimuli, but rather the specificity of the final perceptual representation that can be built from the shape information that is available. Other studies have shown effects of stimulus familiarity and task requirements in the fusiform gyrus, suggesting that there is no need to assume different modules for perceptual representation and representation in memory.

Visual categorization shapes feature selectivity in the primate temporal cortex.

N. Sigala and N. K. Logothetis

Nature  415  318-20  (2002)


The way that we perceive and interact with objects depends on our previous experience with them. For example, a bird expert is more likely to recognize a bird as a sparrow, a sandpiper or a cockatiel than a non-expert. Neurons in the inferior temporal cortex have been shown to be important in the representation of visual objects; however, it is unknown which object features are represented and how these representations are affected by categorization training. Here we show that feature selectivity in the macaque inferior temporal cortex is shaped by categorization of objects on the basis of their visual features. Specifically, we recorded from single neurons while monkeys performed a categorization task with two sets of parametric stimuli. Each stimulus set consisted of four varying features, but only two of the four were important for the categorization task (diagnostic features). We found enhanced neuronal representation of the diagnostic features relative to the non-diagnostic ones. These findings demonstrate that stimulus features important for categorization are instantiated in the activity of single units (neurons) in the primate inferior temporal cortex.

A feedforward architecture accounts for rapid categorization.

T. Serre and A. Oliva and T. Poggio

Proc Natl Acad Sci U S A  104  6424--6429  (2007)

Primates are remarkably good at recognizing objects. The level of performance of their visual system and its robustness to image degradations still surpasses the best computer vision systems despite decades of engineering effort. In particular, the high accuracy of primates in ultra rapid object categorization and rapid serial visual presentation tasks is remarkable. Given the number of processing stages involved and typical neural latencies, such rapid visual processing is likely to be mostly feedforward. Here we show that a specific implementation of a class of feedforward theories of object recognition (that extend the Hubel and Wiesel simple-to-complex cell hierarchy and account for many anatomical and physiological constraints) can predict the level and the pattern of performance achieved by humans on a rapid masked animal vs. non-animal categorization task.

Distinct encoding of spatial and nonspatial visual information in parietal cortex

D. J. Freedman and J. A. Assad

J Neurosci  29  5671-80  (2009)

It is well established that the primate parietal cortex plays an important role in visuospatial processing. Parietal cortex damage in both humans and monkeys can lead to behavioral deficits in spatial processing, and many parietal neurons, such as in the macaque lateral intraparietal area (LIP), are strongly influenced by visual-spatial factors. Several recent studies have shown that LIP neurons can also convey robust signals related to nonspatial factors, such as color, shape, and the behavioral context or rule that is relevant for solving the task at hand. But what is the relationship between the encoding of spatial factors and more abstract, nonspatial, influences in LIP? To examine this, we trained monkeys to group visual motion patterns into two arbitrary categories, and recorded the activity of LIP neurons while monkeys performed a categorization task in which stimuli were presented either inside each neuron's receptive field (RF) or at a location in the opposite visual field. While the activity of nearly all LIP neurons showed strong spatial dependence (i.e., greater responses when stimuli were presented within neurons' RFs), we also found that many LIP neurons also showed reliable encoding of the category membership of stimuli even when the stimuli were presented away from neurons' RFs. This suggests that both spatial and nonspatial information can be encoded by individual LIP neurons, and that parietal cortex may be a nexus for the integration of visuospatial signals and more abstract task-dependent information during complex visually based behaviors.

Inferotemporal cortex subserves three-dimensional structure categorization

B.-E. Verhoef and R. Vogels and P. Janssen

Neuron  73  171-82  (2012)

We perceive real-world objects as three-dimensional (3D), yet it is unknown which brain area underlies our ability to perceive objects in this way. The macaque inferotemporal (IT) cortex contains neurons that respond selectively to 3D structures defined by binocular disparity. To examine the causal role of IT in the categorization of 3D structures, we electrically stimulated clusters of IT neurons with a similar 3D-structure preference while monkeys performed a 3D-structure categorization task. Microstimulation of 3D-structure-selective IT clusters caused monkeys to choose the preferred structure of the 3D-structure-selective neurons considerably more often. Microstimulation in IT also accelerated the monkeys' choice for the preferred structure, while delaying choices corresponding to the nonpreferred structure of a given site. These findings reveal that 3D-structure-selective neurons in IT contribute to the categorization of 3D objects.

Visual categorization and the primate prefrontal cortex: neurophysiology and behavior.

D. J. Freedman and M. Riesenhuber and T. Poggio and E. K. Miller

J Neurophysiol  88  929-41  (2002)


The ability to group stimuli into meaningful categories is a fundamental cognitive process. To explore its neuronal basis, we trained monkeys to categorize computer-generated stimuli as 'cats' and 'dogs.' A morphing system was used to systematically vary stimulus shape and precisely define a category boundary. Psychophysical testing and analysis of eye movements suggest that the monkeys categorized the stimuli by attending to multiple stimulus features. Neuronal activity in the lateral prefrontal cortex reflected the category of visual stimuli and changed with learning when a monkey was retrained with the same stimuli assigned to new categories. Further, many neurons showed activity that appeared to reflect the monkey's decision about whether two stimuli were from the same category or not. These results suggest that the lateral prefrontal cortex is an important part of the neuronal circuitry underlying category learning and category-based behaviors.

Experience-dependent modulation of category-related cortical activity.

L. L. Chao and J. Weisberg and A. Martin

Cereb Cortex  12  545-51  (2002)


Naming pictures of objects from different categories (e.g. animals or tools) evokes maximal responses in different brain regions. However, these 'category-specific' regions typically respond to other object categories as well. Here we used stimulus familiarity to further investigate category representation. Naming pictures of animals and tools elicited category-related activity in a number of previously identified regions. This activity was reduced for familiar relative to novel stimuli. Reduced activation occurred in all object responsive areas in the ventral occipito-temporal cortex, regardless of which category initially produced the maximal response. This suggests that object representations in the ventral occipito-temporal cortex are not limited to a discrete area, but rather are widespread and overlapping. In other regions (e.g. the lateral temporal and left premotor cortices), experience-dependent reductions were category specific. Together, these findings suggest that category-related activations reflect the retrieval of information about category-specific features and attributes.

Categorical representation of visual stimuli in the primate prefrontal cortex.

D. Freedman and M. Riesenhuber and T. Poggio and E. Miller

Science  291  312-6  (2001)


The ability to group stimuli into meaningful categories is a fundamental cognitive process. To explore its neural basis, we trained monkeys to categorize computer-generated stimuli as 'cats' and 'dogs.' A morphing system was used to systematically vary stimulus shape and precisely define the category boundary. Neural activity in the lateral prefrontal cortex reflected the category of visual stimuli, even when a monkey was retrained with the stimuli assigned to new categories.

A comparison of primate prefrontal and inferior temporal cortices during visual categorization.

D. J. Freedman and M. Riesenhuber and T. Poggio and E. K. Miller

J Neurosci  23  5235-46  (2003)


Previous studies have suggested that both the prefrontal cortex (PFC) and inferior temporal cortex (ITC) are involved in high-level visual processing and categorization, but their respective roles are not known. To address this, we trained monkeys to categorize a continuous set of visual stimuli into two categories, 'cats' and 'dogs.' The stimuli were parametrically generated using a computer graphics morphing system (Sheltonelton, 2000) that allowed precise control over stimulus shape. After training, we recorded neural activity from the PFC and the ITC of monkeys while they performed a category-matching task. We found that the PFC and the ITC play distinct roles in category-based behaviors: the ITC seems more involved in the analysis of currently viewed shapes, whereas the PFC showed stronger category signals, memory effects, and a greater tendency to encode information in terms of its behavioral meaning.

Enhanced category tuning revealed by intracranial electroencephalograms in high-order human visual areas

E. Privman and Y. Nir and U. Kramer and S. Kipervasser and F. Andelman and M. Y. Neufeld and R. Mukamel and Y. Yeshurun and I. Fried and R. Malach

J Neurosci  27  6234-42  (2007)


The functional organization of human sensory cortex was studied by comparing intracranial EEG (iEEG) recordings of local field potentials in neurosurgical patients with functional magnetic resonance imaging (fMRI) obtained in healthy subjects. Using naturalistic movie stimuli, we found a tight correlation between these two measures throughout the human sensory cortex. Importantly, the correlation between the iEEG and fMRI signals was site-specific, exhibiting neuroanatomically specific coupling. In several cortical sites the iEEG activity was confined strictly to one object category. This site selectivity was not limited to faces but included other object categories such as houses and tools. The selectivity of the iEEG signals to images of different object categories was remarkably higher when compared with the selectivity of the corresponding fMRI signals. A plausible interpretation of the fMRI and iEEG results concerns cortical organization in which object categories are organized in a mosaic of narrowly tuned object-selective clusters.

Matching categorical object representations in inferior temporal cortex of man and monkey

N. Kriegeskorte and M. Mur and D. A. Ruff and R. Kiani and J. Bodurka and H. Esteky and K. Tanaka and P. A. Bandettini

Neuron  60  1126-41  (2008)


Inferior temporal (IT) object representations have been intensively studied in monkeys and humans, but representations of the same particular objects have never been compared between the species. Moreover, IT's role in categorization is not well understood. Here, we presented monkeys and humans with the same images of real-world objects and measured the IT response pattern elicited by each image. In order to relate the representations between the species and to computational models, we compare response-pattern dissimilarity matrices. IT response patterns form category clusters, which match between man and monkey. The clusters correspond to animate and inanimate objects; within the animate objects, faces and bodies form subclusters. Within each category, IT distinguishes individual exemplars, and the within-category exemplar similarities also match between the species. Our findings suggest that primate IT across species may host a common code, which combines a categorical and a continuous representation of objects.