Reverse correlation technique

The term has since been adopted in psychological experiments that usually do not analyze the temporal dimension, but also present noise to human participants. In contrast to the original meaning, the term is here thought to reflect that the standard psychological practice of presenting stimuli of defined categories to the participants is "reversed": Instead, the participant's mental representations of categories are estimated from interactions of the presented noise and the behavioral responses.

It is used to create composite pictures of individual and/or group mental representations of various items (e.g. faces, bodies, and the self) that depict characteristics of said items (e.g. trustworthiness and self-body image). This technique is helpful when evaluating the mental representations of those with and without mental illnesses.

Terms

This technique utilizes spike-triggered average to explain what areas of signal and noise in an image are valuable for the given research question. Signal is information used to produce objects of value that help explain and connect the world around us. Noise is commonly referred to as unwanted signal that obscures the information that the signal is trying to present. Most importantly for reverse correlation studies, noise is randomly varying information. To determine the areas of importance using reverse correlation, noise is applied to a base image and then evaluated by observers.

A base image is any image void of noise that relates to the research question. A base image that has noise superimposed on top is the stimuli that is presented to and evaluated by participants. Each time a new set of stimuli is presented to a participant, this is known as a trial. After a participant has responded to hundreds to thousands of trials, a researcher is ready to create a classification image.

A classification image (abbreviated as "CI" in some studies) is a single image that represents the average noise patterns in the images selected by participants. A classification image can also be computed for groups by averaging the individuals’ classification images. These classification images are what researchers use to interpret the data and draw conclusions. As a whole, the reverse correlation method is a process that results in a composite image (from an individual or group) that can be used to estimate and interpret mental representations.

One more chapter is free with a Lably account.

Basic study layout

The reverse correlation method is typically executed as an in-lab computer experiment. This method follows four broad steps. Each of the following steps are described in greater detail below.

After creating a research question and determining that the reverse correlation method is the most suitable technique to answer the question, a researcher must (1) design randomly varying stimuli. After the stimuli have been prepared, a researcher should (2) collect data from participants who will see and respond to approximately 300 -1,000 trials. Each trial will either consist of one or two images (side by side) derived from the same base image with noise superimposed on top. Participant responses will depend on the chosen study design; if a researcher presents only one image at a time, participants rate the image on a 4pt scale, but when two images are shown, the participant is asked to choose which best aligns with the given category (e.g. choose the image that looks the most aggressive).

Once all of the data is collected, the researcher will (3) compute classification images for each participant and using those images compute group classification images. Finally, with the classification images available, the researcher will (4) evaluate the images and draw conclusions about their results.

Step 1: making stimuli

When designing the stimuli for a reverse correlation study, the two primary factors that one should consider are (1) the base image and (2) the noise that will be used. While not all bases are images per se, the majority are and for this reason the base is typically referred to as a base image. The base image should represent whatever the research question is addressing. For example, if you are interested in peoples’ mental representations of Chinese people, it would not make sense to use a base image of a Spanish or Caucasian person. Again, if you are interested in the mental representations of male vocal patterns, it would make the most sense to use a base vocal pattern that has been produced by a male.

Having a base is important because it provides a kind of anchor for participants to work from. When there is no base image, the number of trials that are required increases dramatically, thus making it harder to collect data. While there are studies that have excluded a base image, (e.g. the S study), for more elaborate and nuanced research questions, it is important to have a base image that is a fair representation of what participants are being asked to categorize. Photographs of faces are generally the most popular base image.

Although the reverse correlation method is capable of investigating a wide variety of research questions, the most common application of the method is for evaluating faces on a single trait. Reverse correlation studies that address evaluations of the face are sometimes referred to as being a face space reverse correlation model (FSRCM). Thankfully, there are existing databases for face images of varying demographics and emotion that work well as base images.

The reverse correlation method can also be used to help researchers identify what areas of an image (e.g. the areas on the face) have diagnostic value. In order to identify these areas of value, researchers start by minimizing the space a participant can pull information from. By imposing a “mask” on an image (e.g. blur an image while leaving random areas un-blurred), this reduces the information individuals might see, and forces them to focus on certain areas. Then, if/when participants are able to correctly identify an image with a trait repeatedly, we can draw conclusions about what areas have diagnostic value.

While faces and visual stimuli are the most popular, this is not the only stimuli that can be used in a reverse correlation study. This method was originally designed for auditory stimuli which allows researchers to investigate how perceivers interpret auditory information and create trait based attributions to different sound patterns. For example, by segmenting a vocal recording of a single word (total sound time 426 ms) into six segments (71 ms each), and varying each segment's pitch using Gaussian distributions, researchers were able to uncover what vocal patterns people associated with certain traits. Specifically, this study investigated how listeners rated sound clips of the word “really” as sounding more interrogative (i.e. like the more common reverse correlation studies this study had participants listen to two sound clips per trial, choose which fit the category the best, and then created an average of the pitch contours). Beyond face and auditory perception, research utilizing the reverse correlation method has expanded to investigate how individuals see three-dimensional objects in images with noise (but no signal).

After selecting your base image, regardless of what the image is, it is helpful to apply a Gaussian blur to smooth noise in the image. While noise will be applied later, it is helpful to reduce existing noise in the photo before applying your chosen noise. There are three primary choices when it comes to noise: white noise, sine-wave noise, and Gabor noise. The latter two of these constrain the configurations that the noise can have, and because of this white noise is usually the most commonly used. Regardless of the type of noise that is chosen, it is crucial that the noise randomly varies.

Step 2: data collection

Once the stimuli for the study has been developed, the researcher must make a few decisions before actually collecting the data. The researcher must come to a conclusion on how many stimuli will be presented at a time and how many trials the participants will see.

In terms of stimuli presentation, a researcher can choose from either a 2-Image Forced Choice (2IFC) or a 4-Alternative Forced Choice (4AFC). The 2IFC presents two images at once (side by side) and requires participants to choose between the two on a specified category (e.g. which image looks the most like a male). Typically the noise from the left image is the mathematical inverse of the noise from the right image. This method was developed to better answer questions that could not be fully answered by the 4AFC method. As compared to the 2IFC, the 4AFC only shows participants one image per trial and requires them to rate the image on a 4-point scale ((1) Probably X, (2) Possibly X, (3) Possibly Y, (4) Probably Y). For example, here X might represent male and Y might represent female. Typically, during data analysis, only images that are chosen as a “probably” category are included.

As mentioned previously, the 2IFC was designed to address questions that could not be easily answered by the 4AFC. In the 4AFC, there is the possibility that participants may not choose a “probably” category, and if this happens, no classification image can be computed. For example, if the base image does not look like the mental representation participants are asked to report on, then participants may never make a confident choice and classify the image under a “probably” category. While this is a flaw in the 4AFC, one advantage to this method and scale structure is that researchers can see participants’ certainty judgements on their classification decisions (e.g. a probably X label would suggest greater confidence in their decision than a possibly X label).

As for choosing the number of trials, generally researchers conducting a reverse correlation study present participants with 300 - 1,000 trials.

Step 3: computing a classification image (CI)

Again, a classification image is the calculated average noise of all selected images (stimuli). Classification images can be generated for individuals or the group. Computing a classification image for individuals and groups are slightly different. To compute a classification image for an individual, the researcher will start by creating an average of the all selected images’ noise and then overlay that pattern onto the base image. Before the noise is superimposed, it is scaled to fit the base image (i.e. the smallest and largest pixel intensities are matched to the base image pixels). To generate a classification image for a group, the researcher will either handle each individual classification image separately (making sure to scale the pixels independently) or apply a dependent scaling. A dependent scaling is called such because the scaling that is applied to all classification images depends on the image with the greatest range of pixels. Using this single image and its pixel range, the researcher will match the pixels of the classification image to the pixels of the base image. The scaling factor used for this image is then applied to the remaining classification images. When choosing between these two approaches, keep in mind that in classification images with little signal, independent scaling amplifies signal and noise more than dependent scaling. If the researcher is interested in the strength of signal, it is suggested that they use dependent scaling.

When calculating a classification image, it is critical to consider how your external noise will impact your signal to noise ratio (SNR). The SNR is the ratio of desired input (e.g. signal) to undesired information (e.g. noise). One way to produce a high SNR (when observers are unbiased) is to use this formula C=(N_A_A+N_B_A)-(N_A_B+N_B_B). These researchers have found the optimal experimental parameters for different study designs that will result in high SNR.

Step 4: evaluating classification images and drawing conclusions

After computing classification images for individual participants and/or for the group, the researcher will use these images to draw conclusions about their research questions. However, while not always the case, occasionally after the first set of classification images have been generated, researchers will then take these images and present them to a new sample of participants and ask them to rate the images on a subsequent factor of interest. This process is referred to as a two-phase reverse correlation. For example, if a classification image was computed after participants were asked to choose the image that looked the most like a police officer, the generated classification images could then be presented to a new sample who would evaluate the images on how aggressive the faces look. This process makes it easier to draw conclusions on the data. While this step can ease in drawing conclusions, one must use caution to not collect too many participants in the second phase, because high numbers of participants will make the tiniest of differences appear significant, therefore resulting in a Type 1 Error.

While reverse correlation is typically used to create a visual representation of a single trait, this method does have the capability to create a visual representation of more than one trait in one image. By using the same base image and noise, one can create a classification image of trait 1 and a classification image of trait 2, and then create an aggregate photo of the two classification images (thus creating a new classification image incorporating two social traits).

Additionally, researchers have investigated how the decision-making process impacts and is reflected in the reverse correlation method and have found there is a significant relationship between them. Therefore, when interpreting results using the reverse correlation method, researchers must use caution to not ignore how the decision-making process may influence the data.

Reading signal in a classification image can be difficult. When attempting to interpret signal, researchers suggest that the best practice is to use a recently developed metric referred to as “infoVal”. “InfoVal” compares informational value in the computed classification image to a random distribution. Interpreting an “infoVal” measure is similar to interpreting a z-score.