Visual Crowding: a fundamental limit on conscious perception and object recognition

Dispelling the illusion

With regular flicks of the eye, we establish and maintain the illusion of a continuous high-resolution representation of our visual environment. This compelling illusion is easy to dispel by trying to describe the details of objects in your peripheral visual field— scrutinizing or trying to count objects in the visual periphery is impossible. This partly reflects the well-known decline in visual acuity in peripheral vision. However, the most widespread impediment to reading and object recognition in the periphery is the mysterious process known as crowding—the deleterious effect of clutter on peripheral object recognition. Objects that can be easily identified in isolation seem indistinct and jumbled in clutter (Fig. 1).

Crowding is an essential bottleneck, setting limits on object perception, eye and hand movements, visual search, reading and perhaps other functions in peripheral, amblyopic and developing vision. Crowding impairs not only discrimination of object features and contours but the ability to recognize and respond appropriately to objects in clutter. Thus, studying crowding may lead to a better understanding of the processes involved in object recognition. Crowding also has important clinical implications for patients with macular degeneration, amblyopia and dyslexia.

Interest in crowding has significantly increased in the past few years, yielding a more sophisticated understanding of the phenomenon itself as well as of the processes involved in object recognition and reading. Two reviews [1, 2] provide overviews of much of the relevant literature published at the time. Levi[1] concluded that “Crowding is an enigma wrapped in a paradox and shrouded in a conundrum. Despite a great deal of new (and old) work, we do not yet have a full understanding of crowding.” Since then, new approaches, models and findings have provided new insights into the mysteries of crowding, suggesting that crowding occurs at multiple stages in the visual hierarchy.

Operationally defining crowding

The significance of crowding is clear from phenomenological demonstrations of its power and ubiquity in natural scenes (Fig. 1). Ultimately, however, characterizing and understanding the mechanism(s) of crowding requires more than a phenomenological description. Recent work has established that there are several diagnostic criteria for crowding, and using these as converging evidence can help studies individuate and distinguish crowding from other effects, such as masking, lateral interaction and surround suppression. All of these share the characteristic of making a target more difficult to see, but each is distinct and most likely reflects different neural processes.

Crowding and appearance

As discussed earlier, crowded objects do not simply disappear. On the contrary, crowding changes the appearance of the crowded zone, which is important, because it can help distinguish among the main models for crowding (masking, pooling and substitution—discussed below). As examples, Tyler & Likova noted their impression of a crowded letter as a “gray, or inchoate, smudge between the two outer letters, including the inner parts of those letters” [22] (see Figs. 1 and 2). Greenwood, Bex & Dakin[23] elegantly showed that crowded objects appear to take on characteristics of the flankers, a finding consistent with the “jumbled” percept that accompanies crowding. They conclude, in agreement with Levi & Carney [24] that crowding is a regularization process that simplifies the appearance of the peripheral array by promoting consistent appearance among adjacent objects.

Greenwood et al’s [23] results are consistent with a number of earlier studies suggesting that information about crowded objects is not lost. In particular, Parkes et al. [25] found that the orientation signals from the target and flankers in a cluttered peripheral display were pooled rather than being lost through masking. They concluded that crowding reflects compulsory averaging of signals (but see Refs[26-28]) and that crowding is the term we use to define texture perception “when we do not wish it to occur”.

Under conditions of crowding, orientation perception is characterized by strong perceptual assimilation (e.g., the flanker orientation captures the target) near the target and perceptual repulsion (i.e. “anti-crowding”) farther from the target[27, 29-31]. Assimilation could regularize perception of the peripheral array, whereas repulsion could highlight salient differences among visual signals (making different stimuli “pop out”). Assimilation and repulsion reflect opponent influences on orientation perception, and a recent study [30] suggests that the switch from assimilation (crowding) to repulsion depends on cortical distance. Whether this is specific to orientation, or more general is not yet known.

What information survives crowding?

The type of information that gets through to conscious perception under conditions of crowding provides important clues about the nature of crowding.

What information breaks crowding?

Under certain circumstances, crowding may be reduced or released completely.

1. Ungrouping of target and flankers. In peripheral vision, there is a predilection to perceptually group features into a Gestalt.

   Target-flanker grouping. When targets and flankers are similar they are likely to group, and when they are dissimilar they ungroup and the target pops out (Fig. 4). Thus, crowding is reduced when targets and flankers are dissimilar in shape and size [41, 42], orientation [4, 43, 44], polarity[16, 41], spatial frequency [45], depth [41] color [41, 46-48], synesthetic color to some degree [49], motion [50] and “order” (i.e., first-order vs. second order[51]). Temporal grouping also modulates crowding. Crowding is maximal when targets and flankers are presented nearly simultaneously; presenting targets before or after the flankers (by ~150 ms) is sufficient to break crowding [17, 18].

   Flanker-flanker grouping. In multi-element flankers, when the flankers group separately from the target, crowding may be reduced [24, 28, 52-56]. Thus when target and flanker look like a regular texture, it is difficult to make judgments about the target and crowding is strong, whereas when the target appears distinct from the flankers, crowding is weak or absent[56].

2. Object-centered or “holistic” crowding (Fig 4D-E). Crowding can occur between configural, high-level representations of objects. Inverting face flankers can release face crowding. Specifically, it is harder to recognize an upright target face when it is surrounded by a crowd of nearby upright faces than by a crowd of inverted faces[13, 57] (Fig. 4). The inversion effect in crowding also occurs between Mooney (two-tone) faces[13], stimuli that require holistic processing[58]. These object-centered crowding effects adhere to all of the diagnostic criteria for crowding, and are not due to masking, similarity effects, or grouping of low-level features. Likewise, holistic crowding—crowding between upright face representations—can be distinguished from within-face or facial feature crowding[13, 59]. In fact, these object-centered crowding effects demonstrate compound crowding within the same stimulus—crowding between the whole upright faces, and between the low level features that comprise each face; this suggests that crowding operates at multiple stages. Recent work demonstrates that object-centered crowding effects also occur for letter-like stimuli[35], raising the possibility that object-centered, holistic crowding might occur independently at different levels of visual processing.

3. Attention. Several recent studies have demonstrated that cueing a crowded target location reduces the effects of crowding [21, 39, 60, 61]. Dakin and colleagues[62] also found that devoting attention to the target region ameliorated the effect of crowding. The only explanation for these findings, as well as Cavanagh & Holcombe’s demonstration of attentionally-gated crowding effects (Vision Sciences Society Annual Meeting, 2007), is that attention can modulate the critical spacing in crowded arrays (see also Box 1).

4. Masked flankers. Crowding may be “released” when the flankers are masked. However, Chakavarthy & Cavanagh[63] showed that this release only occurred with noise and metacontrast masks but not with object substitution masks. They argue that noise and metacontrast masks act early in the visual processing cascade, degrading the features, whereas object substitution masks do not interfere with feature encoding but act much later, by replacing the representation of the stimulus.

5. Suppression of flankers from visual awareness. Wallis & Bex[64] used “adaptation-induced blindness” to render flankers perceptually invisible and used a dual report paradigm to obtain a trial-by-trial assessment of awareness and crowding. Target identification was dependent on the number of flanking letters perceived on a given trial, independent of the number that were physically present, and they concluded that crowding is “released” when the flankers are suppressed from visual awareness.

Where in the brain does crowding take place?

There has been a great deal more psychophysical than neurophysiological work on crowding. More than anything, the neurophysiological studies of crowding have proven that it is surprisingly difficult to isolate the neural mechanism(s) of crowding per se (Box 2). Nevertheless, the psychophysical studies help narrow down the level(s) at which crowding occurs, and will help guide the design of more stringent future neurophysiological experiments.

Although there is great diversity in the stimuli employed (ranging from oriented bars, Gabor patches and shapes to letters, words, and faces among others—for reviews, see [1, 2]), most studies of crowding implicitly (if not explicitly) argue that crowding is a unitary phenomenon, occurring at a single circumscribed level of visual processing, or perhaps in a particular visual area. Early work demonstrated that crowding works dichoptically (target to one eye, distractors to the other eye) [65, 66], suggesting that crowding arises in the cortex. Since then, various researchers have suggested that the site of crowding might be V1 (e.g., [67]), V2 (e.g., Freeman and Simoncelli (Vision Sciences Society Annual Meeting, 2010), V3 (e.g., [22, 68]), V4 (e.g., [10, 69] but c.f., Ref [70]), or even later in visual processing[32, 57]. The evidence that crowding occurs in each of these and other visual areas is mixed, and difficulties comparing across studies is compounded by the fact that crowding can occur selectively between different kinds of stimuli (see Box 1 and the preceding section). Even within a stimulus type (e.g., orientation, motion, or faces), crowding is modulated by stimulus similarity, context, and attention (Fig. 4 and Box 1), as discussed earlier. Concluding that any single visual area could explain this range of effects is therefore tenuous, at best. Neurophysiological studies have provided relatively little additional evidence about the neural mechanism(s) of visual crowding (Box 2).

Models of crowding

There is no shortage of ideas about crowding, but few are computational or make specific quantitative predictions. The large number of different models may be distilled down to three basic classes: i) masking, ii) pooling (either pooling of low level features or pooling by attention) and iii) substitution. Within each class many different architectures and algorithms have been proposed. These extant models are largely descriptive, and have been reviewed in detail elsewhere[1]. There are few quantitative models of crowding, and most are quite recent. Wilkinson et al[71] proposed a model in which complex cells and simple cells interact by mutual inhibition. In this model, isolated visual contours are processed by simple cells, which suppress weak complex cell responses. However, in the presence of nearby similarly oriented flanking contours in a small area, complex cells respond vigorously because of spatial pooling, and they then suppress simple cell activity within their receptive field area. This texture model nicely predicts several aspects of their data; however, the pooling parameter was based on the simulations that best fit the data, rather than on physiology or some other principled approach.

There are three more recent approaches to modeling crowding. Van den Berg and colleagues[72] propose a quantitative model for spatial integration of orientation signals, based on the principles of population coding. Their model nicely predicts several properties of crowding, including critical spacing, “compulsory averaging”, and the inner/outer asymmetry. However, in its current form it fails to predict the effect of target flank similarity [41, 42], the configuration effects [24, 28, 52], and object-centered or holistic crowding [13, 35, 57]. Dayan & Solomon[73] take a very different approach, in which spatial selection of a target among flankers emerges through a process of Bayesian inference, in a computational form. “Interference” (also known as crowding) in this model results from the spatial uncertainty inherent in large receptive fields, and receptive field size is assumed to increase with eccentricity according to the cortical magnification factor. The model was developed to explain the Eriksen flanker task. It remains to be seen how well it accounts for many of the key features of crowding reviewed here.

None of the models naturally accounts for the radial/tangential anisotropy in a principled way. Van den Berg et al [72] simply use different parameters to define the radial and tangential integration fields. In contrast, Nandy and Tjan (Vision Sciences Society Annual Meeting, 2010) begin with a model of cortical area V1 and its geometry and lateral connections, quite similar to the model of Neri & Levi [74], combined with the important role of natural image statistics[75]. However, the novel insight and advance in their model is the idea that image statistics are acquired primarily at attended spatial locations via a gating mechanism, and that spatial attention and any subsequent eye movement that it elicits overlap in time. Nandy & Tjan further argue that learning image statistics during development leads to the formation of lateral connections that distort the true image statistics in the peripheral field, leading in turn to the radial-tangential anisotropy of crowding. Whether this model will be able to account for grouping, similarity, feature/object specific crowding, remains to be seen.

Any successful computational model of crowding needs to account for each of the characteristics above, including the diagnostic criteria and the factors that modulate crowding. It is not sufficient for a model of crowding to simply mimic or reproduce the phenomenological “jumble” that is representative of crowding [75]; the model must hold to the diagnostic criteria for crowding and it must account for what escapes crowding, what breaks crowding, and what modulates crowding.

Multiple levels of crowding

Although there is great heterogeneity in the results on crowding, there is sufficient evidence to cast doubt on the idea that crowding is a unitary effect due to a single stage of processing, though this is implicitly assumed in most studies of crowding (reviewed in Refs [1, 2]); rather, the collective work suggests that crowding happens independently at several stages of visual processing. In support of this view are the observations that crowding is specific to the similarity between, and the configuration of, target and flanks (discussed above and in Box 1), and the fact that there is “compound” crowding: in a given scene, crowding occurs selectively between features [1], object parts [59], and whole objects [13, 57]. These make an ‘all-convergent’ crowding stage unlikely. If crowding occurs at multiple levels of visual analysis, or if different ‘channels’ (chromatic, spatial frequency, object, etc.) each possess their own unique crowding bottleneck, then one would expect that the gradient of crowding as a function of eccentricity might be channel or stimulus specific. There is intriguing evidence building in favor of this view (Box 1), but much work remains (Box 3).

In natural scenes, crowding may be ubiquitous, but it may occur in layers, with location, content, and attention-dependence. One of the major challenges in future work on crowding will be to develop a parsimonious model that can account for the diversity of findings. Individual models that ignore content, category boundaries, similarity, and attention are not likely to succeed.

The crowded future

Our discussion of crowding has been confined exclusively to visual perception, but the implications of crowding are far and wide and will likely expand rapidly in the near future (see Box 3). For example, crowding may exert a fundamental limit on visually guided actions in naturally cluttered scenes. Although the impact of crowding, per se, on visually guided action has rarely been investigated[14, 76, 77], several studies suggest that clutter impairs action (e.g., Refs[78, 79]). More intriguingly, there is evidence that crowded visual information is differentially used by perceptual and visuomotor systems [76]. Such dissociations between the perceptual and motor responses to crowded scenes may help address the paradoxical but unanswered question of why eye (and hand) movements are not more random than they are in natural scenes—if peripheral object identities are crowded and unrecognizable, how do we make accurate, non-random eye and hand movements to those objects? The practical consequences of visuomotor crowding, divorced from perceptual crowding, would be extensive, ranging from clinical settings to ergonomics and human factors; therefore, the search should be on for dissociations between perception and action in crowded scenes (Box 3).

Because crowding limits visually guided action and it defines the resolution of conscious visual perception, another important question is how crowding develops from infancy to old age. Although little work has been done specifically on crowding in older populations (c.f., Ref[80]), studies with infants as young as 6 months[81], toddlers[82], and adolescents from 8 to 11 years old[83] have reported that children are much more impaired by crowding than adults, even when their acuity is fully developed[83]. Given this protracted development of crowding, an important question is what sorts of knock-on effects might occur later in life if crowding developed abnormally in infancy. For example, crowding may be uniquely and differentially impacted in neurodevelopmental, visual, and cognitive disorders including autism[84], dyslexia[85], amblyopia[86], and macular degeneration[1], among others. Therefore, the clinical and practical implications of crowding, as well as its training and possible rehabilitation, are widespread.