Visual indexing theory

Theoretical context

Visual indexing theory was created partly in response to what Pylyshyn viewed as limitations of traditional theories of perception and cognition — in particular, the spotlight model of attention, and the descriptive view of visual representation.

Spotlight and zoom-lens models of attention

The traditional view of visual perception holds that attention is fundamental to visual processing. In terms of an analogy offered by Posner, Snyder and Davidson (1980): "Attention can be likened to a spotlight that enhances the efficiency of detection of events within its beam". This spotlight can be controlled volitionally, or drawn involuntarily to salient elements of a scene, but a key characteristic is that it can only be deployed to one location at a time. In 1986, Eriksen and St. James conducted a series of experiments which suggested that the spotlight of attention comes equipped with a zoom-lens. The zoom-lens allows the size of the area of attentional focus to be expanded (but due to a fixed limit on available attentional resources, only at the expense of processing efficiency).

According to Pylyshyn, the spotlight/zoom-lens model cannot tell the complete story of visual perception. He argues that a pre-attentive mechanism is needed to individuate objects upon which a spotlight of attention could be directed in the first place. Furthermore, results of multiple object tracking studies (discussed below) are "incompatible with the proposal that items are accessed by moving around a single spotlight of attention." Visual indexing theory addresses these limitations.

Descriptive view of visual representation

According to the classical view of mental representation, we perceive objects according to the conceptual descriptions they fall under. It is these descriptions, and not the raw content of our visual perceptions, that allow us to construct meaningful representations of the world around us, and determine appropriate courses of action. In Pylyshyn's words, "it is not the bright spot in the sky that determines which way we set out when we are lost, but the fact that we see it (or represent it) as the North Star". The method by which we come to match a percept to its appropriate description has been the subject of ongoing investigation (for example the way in which parts of objects are combined to represent their whole), but there is a general consensus that descriptions are fundamental in this way to visual perception.

Like the spotlight model of attention, Pylyshyn takes the descriptive model of visual representation to be incomplete. One issue is that the theory does not account for demonstrative, or indexical references. "For example, in the presence of a visual stimulus, we can think thoughts such as `that is red' where the term `that' refers to something we have picked out in our field of view without reference to what category it falls under or what properties it may have." Relatedly, the theory has problems accounting for how we are able to pick out a single token among several objects of the same type. For example, I may refer to a particular can of soup on a supermarket shelf sitting among a number of identical cans that answer to the same description. In both cases, a spatiotemporal reference is required in order to pick out the object within the scene, independently of any description that object may fall under. FINSTs, Pylyshyn suggests, provide just such a reference.

A deeper problem for this view, according to Pylyshyn, is that it cannot account for objects' continuity over time. "An individual remains the same individual when it moves about or when it changes any (or even all) of its visible properties." If we refer to objects solely in terms of their conceptual descriptions, it is not clear how the visual system maintains an object's identity when those descriptions change. "The visual system needs to be able to pick out a particular individual regardless of what properties the individual happens to have at any instant of time." Pylyshyn argues that FINSTs' detachment from the descriptions of the objects they reference overcomes this problem.

Experimental evidence

Three main types of experiments provide data that support visual indexing theory. Multiple tracking studies demonstrate that more than one object can be tracked within the visual field simultaneously, subitizing studies suggest the existence of a mechanism that allows small numbers of objects to be efficiently enumerated, and subset selection studies show that certain elements of a visual scene can be processed independently of other items. In all three cases, FINSTs provide an explanation of the phenomenon observed.

Multiple object tracking studies

Multiple object tracking describes the ability of human subjects to simultaneously track the movement of up to five target objects as they move across the visual field, usually in the presence of identical moving distractor objects of equal or greater number. The phenomenon was first demonstrated by Pylyshyn and Storm in 1988, and their results have been widely replicated (see Pylyshyn, 2007 for a summary.)

Experimental setup

In a typical experiment, a number of identical objects (up to 10) are initially shown on a screen. Some subset of these objects (up to five) are then designated as targets — usually by flashing or changing colour momentarily — before returning to being indistinguishable from the non-target objects. All of the objects then proceed to move randomly around the screen for between 7 and 15 seconds. The subject's task it to identify, once the objects have stopped moving, which objects were the targets. Successful completion of the task thus requires subjects to continually track each of the target objects as they move, and ignore the distractors.

Results

Under such experimental conditions, it has been repeatedly found that subjects can track multiple moving objects simultaneously. In addition to consistently observing a high rate of successful target tracking, researchers have shown that subjects can:

• track objects that have undergone occlusion, even when they have changed trajectory whilst occluded,
• track objects across changes in viewpoint,
• track 'holes' as effectively as they can track 'objects',
• track objects moving at high speed, and into the periphery of the visual field,
• become better at tracking multiple objects with relevant practice/expertise.

Two defining properties of FINSTs are their plurality, and their capacity to track indexed objects as they move around a visually cluttered scene. "Thus multiple-item tracking studies provide strong support for one of the more counterintuitive predictions of FINST theory — namely, that the identity of items can be maintained by the visual system even when the items are visually indiscriminable from their neighbors and when their locations are constantly changing."

Subitizing studies

Subitizing refers to the rapid and accurate enumeration of small numbers of items. Numerous studies (dating back to Jevons in 1871) have demonstrated that subjects can very quickly and accurately report the quantity of objects randomly presented on a display, when they number fewer than around five. While larger quantities require subjects to count or estimate — at great expense of time and accuracy — it seems that a different enumeration method is employed in these low-quantity cases. In 1949, Kaufman, Lord, Reese and Volkmann coined the term 'subitizing' to describe the phenomenon.

In 2023 a study of single neuron recordings in the medial temporal lobe of neurosurgical patients judging numbers reported evidence of two separate neural mechanisms with a boundary in neuronal coding around number 4 that correlates with the behavioural transition from subitizing to estimation, supporting the old observation of Jevons.

Experimental setup

In a typical experiment, subjects are briefly shown (for around 100ms) a screen containing a number of randomly arranged objects. The subjects' task is to report the number of items shown, which can range between one and several hundred per trial.

Results

When the number of items to be enumerated is within the subitizing range, each additional item on the display adds around 40–120ms to the total response time. Beyond the subitizing range, each additional item adds 250–350ms to the total response time (so that when the number of items presented is plotted against reaction time, an 'elbow' shaped curve results.) Researchers generally take this to be evidence of there being (at least) two different enumeration methods at work — one for small numbers, and another for larger numbers.

Trick and Pylyshyn (1993) argue that "subitizing can be explained only by virtue of a limited-capacity mechanism that operates after the spatially parallel processes of feature detection and grouping but before the serial processes of spatial attention." In other words, by a mechanism such as a FINST.

Subset selection studies

A key assumption of visual indexing theory is that once an item entering the visual field has been indexed, that index provides the subject with rapid subsequent access to the object, which bypasses any higher level cognitive processes. In order to test this hypothesis, Burkell and Pylyshyn (1997) designed a series of experiments to see whether subjects could effectively index a subset of items on a display, such that a search task could be undertaken with respect to only the selected items.

Experimental setup

Burkell and Pylyshyn's experiments took advantage of a well-documented distinction between two types of visual search:

• Feature searches involve the identification of a target among a field of distractors that differ from the target in one dimension — for example, a blue object among a field of green distractors, or a square among a field of circles. Feature searches are quickly executed, due to the fact that the target item “pops out” of the display and so is easy to locate.
• In a conjunction search, the distractor items share more than one dimension with the target — for example a subject might be asked to locate a blue square among a field of blue circles and green squares. This type of search is far less efficient than a feature search, because target no longer “pops out” of the display. Subjects instead need to examine each object individually in order to identify the target, resulting in much longer response times.

The experimental setup is similar to a typical conjunction search task: 15 items are presented on a screen, each of which has one of two colours, and one of two orientations. Three of these items are designated as the subset by late onset (appearing after the others). The subset contains the target item and two distractors.

The key independent variable in this experiment is the nature of the subset selected. In some cases, the subset comprises a feature search set — i.e. the target differs from the two distractors in one dimension only. In other cases, the subset is equivalent to a conjunction search, with the target differing from the distractors in both dimensions. Because the total display contains items that differ from the target in both dimensions, if subjects are quicker to respond to the feature search subsets, this would suggest they had taken advantage of the "pop out" method of target identification. This in turn would mean that they had applied their visual search to the subsetted items only.

Results

Burkell and Pylyshyn found that subjects were indeed quicker to identify the target object in the subset feature search condition than they were in the subset conjunction search condition, suggesting that the subsetted objects were successfully prioritised. In other words, the subsets "could, in a number of important ways, be accessed by the visual system as though they were the only items present". Furthermore, the subsetted objects' particular positions within the display made no difference to subjects' ability to search across them — even when they were distally located. Watson and Humphreys (1997) reported similar findings. These results are consistent with the predictions of visual indexing theory: FINSTs provide a possible mechanism by which the subsets were prioritised.