Time courses of attended and ignored object representations

Participants

All participants (n=23; 11 males and 12 females) were healthy undergraduate and graduate student volunteers from the University of California, Davis. They had normal or corrected-to-normal vision, were free of neuropsychiatric conditions, and gave written informed consent for their participation; they received monetary compensation for their time. This sample size was chosen based on decoding effect sizes from previous work (Noah et al., 2020).

EEG datasets from three participants were rejected because of intractable noise in the EEG data or participant noncompliance with the task requirements, yielding a final EEG dataset including 20 participants (9 males and 11 females).

Apparatus and Stimuli

Participants were comfortably seated in an electrically shielded, sound-attenuating room (ETS-Lindgren, USA). Stimuli were presented on a VIEWPixx/EEG LED monitor (model VPX-2006A; VPixx Technologies Inc., Quebec Canada) at a viewing distance of 85 cm, and vertically centered at eye level. The display measured 23.6 inches diagonally, with a native resolution of 1920 by 1080 pixels, and a refresh rate of 120Hz. The recording room and objects in the room were painted black to reduce reflected light. The recording room was dimly illuminated using DC lights.

The stimuli to be discriminated were composites of two overlapping images (objects), such that on each trial, an image belonging to the target category (when present) was superimposed with an image belonging to a non-cued, distractor category (see Figure 1). The behavioral task for this experiment was to determine, on each trial, whether the briefly presented target image belonging to the cued object category (faces, scenes, or tools) was in-focus or slightly blurry. Crucially, both the target and the distractor objects in the composite image could be in-focus or blurry independently of each other, therefore, the task could not be performed solely by attending to and responding to the presence or absence of blur; the subject had to use the information in the cue to perform the task.

Each trial began with the pseudorandomly selected presentation of one of three possible cue types for 200 msec (1° × 1° triangle, square, or circle, using PsychToolbox; Brainard, 1997). Valid cues informed participants which target object category (face, scene, or tool) was likely to subsequently appear (80% probability), and the cue also instructed participants to attend selectively to the object image from the cued category. Cues were presented 1° above the central fixation point. Following pseudorandomly selected SOAs (ranging from 1000 – 2500 msec) from cue onset, target stimuli (5° × 5° square image) were presented at fixation for 50 msec. A pseudorandomly distributed inter-trial-interval (ITI; 1500 – 2500 msec) separated target offset from the onset of the cue in the next trial.

Twenty percent of trials were invalidly cued, allowing us to assess the effect of cue validity on behavioral performance. For the invalid trials, the stimulus image was a composite of an image from a randomly chosen non-cued object category, superimposed with a black and white checkerboard. The checkerboard could also be blurry or in-focus independently of the object image. Participants were instructed that whenever they encountered a trial where the blended stimulus did not include an image belonging to the cued object category, but instead contained only one object image and a checkerboard overlay, then they had to indicate whether the non-cued object image in the stimulus was blurry or in-focus. We predicted that participants would be slower to respond on invalidly cued trials, analogously to the behavioral effect of validity observed in standard cued spatial attention paradigms (e.g., Posner, Snyder, & Davidson, 1980).

The stimulus images spanned a square that was 5° × 5° of visual angle. To create blurred images, Gaussian blur with a standard deviation of 2 was applied to the images, using the Matlab function imgaussfilt(). All stimuli were presented against a gray background (RGB: 128, 128, 128). A white fixation dot was continuously present in the center of the display.

All three object categories included 40 different individual images. On each trial, random images were drawn to produce the composite stimulus image. All target images were gathered from the Internet. Face images were front-facing and neutral-expression, cropped and placed against a white background (Ma, Correll, & Wittenbrink, 2015). All face images were cropped to ovals centered on the face and placed against a white background. Full-frame scene images were drawn from the University of Texas at Austin’s natural scene collection (Geisler & Perry, 2011) and campus scene collection (Burge & Geisler, 2011). Tool images, cropped, and placed against a white background, were drawn from the Bank of Standardized Stimuli (Brodeur, Guérard, & Bouras, 2014). Unlike scene images, which contained visual details spanning the entire 5° × 5° square, face and tool images were set against white backgrounds and so did not contain visual information up to all the image boundaries. Therefore, to eliminate the possibility that participants could use cue information to focus spatial attention instead of object-based attention to perform the blurry/in-focus discrimination, on any trial where a face or tool image was included in the composite stimulus, the position of that face or tool image was randomly displaced from the center.

Procedure

Participants were instructed to maintain fixation on the center of the screen during each trial, and to anticipate the cued object category until the target image appeared. They were further instructed to indicate whether the target image was blurry or not-blurry with a button press as quickly as possible upon target presentation, using the index finger button for “blurry” and the middle finger button for “not-blurry.” Responses were only recorded during the interval between target onset and the next trial. Trials were classified as correct when the recorded response matched the target image subcategory, and incorrect when the response did not match, or when there was no recorded response.

Participants were instructed to respond as quickly as they could to the target stimulus, making it critical that the participants fully engaged preparatory attention toward the cued object category during the preparatory period. All participants were trained with at least 42 practice trials of the task and were required to achieve at least 60% response accuracy before beginning the task with EEG data collection; to achieve this, stimulus duration was adjusted on an individual participant basis during the initial training phase. Each participant completed 15 blocks of the experiment while EEG was recorded, with each block comprising 42 trials, totaling 630 trials.

EEG Recording

Raw EEG data were acquired with a 64-channel Brain Products actiCAP active electrode system (Brain Products GmbH), and amplified and digitized using a Neuroscan SynAmps2 input board and amplifier (Compumedics USA, Inc.). Signals were recorded with Scan 4.5 acquisition software (Compumedics USA, Inc.) at a sampling rate of 1000 Hz, and a DC to 200 Hz online band pass. Sixty-four Ag/AgCl active electrodes were placed in fitted elastic caps using the following montage, in accordance with the international 10–10 system (Jurcak, Tsuzuki, & Dan, 2007): FP1, FP2, AF7, AF3, AFz, AF4, AF8, F7, F5, F3, F1, Fz, F2, F4, F6, F8, FT9, FT7, FC5, FC3, FC1, FCz, FC2, FC4, FC6, FT8, FT10, T7, C5, C3, C1, Cz, C2, C4, C6, T8, TP9, TP7, CP5, CP3, CP1, CPz, CP2, CP4, CP6, TP8, TP10, P7, P5, P3, P1, Pz, P2, P4, P6, P8, PO7, PO3, POz, PO4, PO8, PO9, O1, Oz, O2, PO10; channels AFz and FCz were assigned as ground and online reference, respectively. Additionally, electrodes at sites TP9 and TP10 were placed directly on the left and right mastoids, respectively. The Cz electrode was placed at the vertex of each participant’s head by measuring anterior to posterior from nasion to inion, and right to left between preauricular points. High viscosity electrolyte gel was administered at each electrode site to facilitate conduction between electrode and scalp, and impedance values were kept below 25 kΩ. Continuous data were saved in individual files corresponding to each trial block of the stimulus paradigm.

EEG Preprocessing

All data preprocessing procedures were completed with the EEGLAB Matlab toolbox (Delorme & Makeig, 2004). For each participant, all EEG data files were merged into a single dataset before data processing. Each dataset was visually inspected for the presence of noisy channels, which were replaced by interpolation from neighboring electrodes. The data were Hamming window sinc FIR filtered (1 – 83 Hz) and down sampled to 250 Hz. The data were first low pass filtered below 83 Hz (filter roll-off −6 dB at 93 Hz) and then high pass filtered above 1 Hz (−6 dB at 0.5 Hz). Both filters were accomplished with the EEGLAB function pop_eegfiltnew(). Data were algebraically re-referenced to the average of all electrodes. Data were epoched from 500 msec before stimulus onset to 1000 msec after stimulus onset, so that stimulus-period data from all trials could be examined together. Independent component analysis (ICA) decomposition was used to remove artifacts associated with blinks and eye movements. To maintain fully balanced conditions across trials in subsequent decoding analyses, no manual rejection of trials was performed.

EEG Decoding Analysis

We implemented a decoding analysis to quantitatively assess whether representations of target and distractor visual information were systematically associated with changes in phase-locked ERP voltage topography. Invalid trials, in which the instructional cue did not predict the subsequently presented object image, were excluded from this analysis, because on invalid trials only one object image was presented in the stimulus: This object image did not belong to the cued object category, meaning that for invalid trial data, there is no available comparison between visual processing for attended and unattended object image information.

For the decoding analyses, we restricted the ERP data signals to those below 25 Hz. To do so, we performed a band pass filtering procedure over the preprocessed EEG data, using the eegfilt() function from the EEGLAB Matlab library.

Our analysis was adapted from a routine to decode attention and working memory representations from scalp EEG (G.-Y. Bae & Luck, 2018; Noah et al., 2020). Decoding was performed independently at each time point within the stimulus epoch of 500 msec before to 1000 msec after stimulus onset. We implemented the Matlab fitcsvm() function to use a support vector machine (SVM) classification algorithm. For every time point, a binary classifier was trained to classify whether single-timepoint ERP data belonged to a trial in which the target stimulus was blurry or not blurry, and a separate binary classifier was trained to classify whether the data belonged to a trial in which the distractor stimulus was blurry or not blurry. For each classification, decoding was considered correct when the classifier correctly determined the target or distractor blurriness from the two possible conditions, thus chance performance was set at 50%.

Because the blurriness condition of the target stimulus image on each trial is mapped to the instructed motor execution (index finger button press for blurry images and middle finger button press for not-blurry images), it is possible that cortical activity supporting motor response preparation and execution contributes to the decoding accuracy of target blurriness. In the Discussion below, we argue that it is unlikely that this confound is driving the observed difference between target and distractor decoding accuracy.

The decoding for each time point followed a six-fold cross-validation procedure. Within each fold, data were averaged across trials to improve the signal-to-noise ratio (G.-Y. Bae & Luck, 2018). Data from five-sixths of the trials, randomly selected, were used to train the classifier with the correct labeling. The remaining one-sixth of the trials were used to test the classifier, using the Matlab predict() function. This entire training and testing procedure was iterated 10 times, with new training and testing data assigned randomly in each iteration. Trial averaging followed random assignment of trials into folds, for each iteration. For each blurriness condition, each participant, and each time point, decoding accuracy was calculated by summing the number of correctly predicted labels across trials and iterations and dividing by the total number of labels.

We averaged together the decoding results for all 10 iterations to examine decoding accuracy across participants, at every time point in the stimulus epoch. At any given timepoint, above-chance decoding accuracy indicates that ERP topography, in its distribution, amplitude, or other characteristics, contains information about the blurriness condition of the target or the distractor. We utilized a Monte Carlo simulation-based significance assessment to correct for multiple comparisons across time and reveal statistically significant clusters of decoding accuracies (G.-Y. Bae & Luck, 2018; Noah et al., 2020).

By the Monte Carlo statistical method, decoding accuracy was assessed at each timepoint against a randomly chosen integer (1 or 2) representing an experimental condition. That is, we randomly shuffled the labels used to train the SVM. A t-test of classification accuracy across participants against chance was performed at each time point for the shuffled data. Clusters of consecutive time points with decoding accuracies determined to be statistically significant by t-test were identified, and a cluster t-mass was calculated for each cluster by summing the t-values given by each constituent t-test. Each cluster t-mass was saved. This procedure was iterated 1000 times, to generate a distribution of t-masses to represent the null hypothesis that a given cluster of t-masses from our decoding analysis was likely to have been found by random chance. The 95% cutoff t-mass value was determined from the permutation-based null distribution and used as the cutoff against which cluster t-masses calculated from our original decoding data could be compared. Clusters of consecutive time points in the original decoding results with t-masses exceeding the permutation-based threshold were deemed statistically significant (G.-Y. Bae & Luck, 2018).

SVM Weight Map Analysis

We constructed topographic maps of classifier weights to scrutinize the factors contributing to the SVM classification. These weight maps illustrate the value of the classifier weight at each electrode site, for both the target and distractor classifiers.

To construct the weight maps, we multiplied the covariance matrix of the training data and the model betas for each block of decoding to produce weights for that decoding block. Then we averaged these products across blocks, and finally across subjects, to produce the final maps. We constructed weight maps for target and distractor decoding SVM analyses separately.

Behavioral Control Experiment

In the main experiment, object-based attention was operationalized with a cueing paradigm, in which on each trial the appearance of one of three possible object categories (faces, scenes, or tools) was indicated ahead of time with an 80% predictive cue. For the 80% of trials that were validly cued, an image from the cued object category appeared superimposed with an image from an uncued object category. For the 20% of trials that were invalidly cued, an image from an uncued object category appeared superimposed with a checkerboard pattern instead of another object image (Figure 1). The invalid trial stimuli were designed such that only one object image was present, to preclude any ambiguity about what object image was the intended target of the discrimination task in the absence of an image from the cued object category. The checkerboard pattern was displayed overlaid with the invalidly cued object image to provide distracting visual information comparable to that from an overlaid uncued object image in the valid trials.

This paradigm was designed so that the effect of cued, anticipatory object-based attention would manifest as a difference in reaction time between valid and invalid trials. We predicted that invalid trials would display longer reaction times than valid trials, because the beneficial effects of anticipatory attention would only apply when the anticipated object category was present in the target stimulus. However, the systematic difference in visual stimulus characteristics between the valid and invalid trials, with valid trial stimuli composed of two object images and invalid trial stimuli composed of a checkerboard pattern and one object image, could lead to the question of whether the observed behavioral decrement on the invalid trials was truly the result of anticipatory object-based attention to a different object category, or if instead the invalid trial design was simply more difficult to visually parse. It is reasonable to wonder whether the checkerboard pattern overlaid on the uncued object image targets made discrimination of those targets more difficult than discrimination of an object image in superimposition with another object image. To investigate this, and to provide a stronger test of behavioral effects of object attention, we conducted a control experiment.

We collected behavioral data from ten new participants (4 females, 5 males, 1 non-binary). All participants were healthy undergraduate and graduate volunteers from the University of California, Davis, had normal or corrected-to-normal vision, gave informed consent, and received monetary compensation for their time.

The design, procedure and stimuli of this experiment were identical with those of the main experiment, with one modification, as follows. 60% of trials were the same validly cued trials as in the main experiment (two overlapping object images), and as before, another 20% of trials were the same invalidly cued trials as in the main experiment (invalidly cued object overlapping a checkerboard). In addition, however, the remaining 20% of trials belonged to a new valid checkerboard condition; the stimulus was composed of an object image from the cued object category superimposed with the same checkerboard pattern present in invalid trial stimuli.

Behavioral Results

ERP Decoding During the Stimulus Period

The participants’ task was to press one button if the target object image (belonging to the cued object category) was blurry, and a second button if that target image was not blurry. On validly cued trials, the image stimuli were composite images made of a randomly selected image from the cued object category overlaid with a randomly selected distractor image from one of the two uncued object categories. Both the target and the distractor object image could be blurry or not blurry, varying independently and randomly.

For the ERP decoding analysis presented here, the condition labels being decoded are blurry and not-blurry. The decoding routine is performed separately for the target and distractor objects in the composite stimuli, so that the effects of anticipatory object-based attention on stimulus representation can be compared for the attended object image and the to-be-ignored distractor object image.

Because the information being decoded in this analysis is whether the stimulus image (target or distractor, depending on the analysis) is blurry or not-blurry, theoretical chance was 50% (as shown by the red line in Figures 4, 5, and 7) and in the cluster-based tests for statistical significance.

The ERP decoding results for the target object image are presented in Figure 4A. Clusters of statistically significant time points persist from target stimulus onset to nearly the end of the decoding epoch. In the early sensory response period of roughly 0 – 200 msec, average decoding accuracy across participants peaks at 75%.

The ERP decoding results for the distractor object image are presented in Figure 4B. Clusters of statistically significant time points begin at stimulus onset and decline after 250 msec, with small clusters of statistically significant decoding appearing later in the decoding epoch. In the early sensory response period of roughly 0 – 200 msec, average decoding accuracy across participants once again peaks at about 70%. Overall, compared to the target image decoding result, the decoding accuracy is weaker throughout the stimulus period. To evaluate whether this observation reflects significant differences between target and distractor decoding, we performed a direct statistical test.

Comparison of Target and Distractor Decoding Accuracy

Using the same cluster t-mass permutation technique as described previously and utilized for all statistical significance tests of decoding accuracy against chance, statistically significantly greater decoding accuracy values were observed for target decoding than for distractor decoding. At each time point, the target and distractor decoding accuracy timeseries were compared by right-tailed paired-sample t-test to test the null hypothesis that target decoding was not more accurate than distractor decoding to a greater degree than might be expected by chance. The results of this statistical test are shown in Figure 5. The test revealed statistically significant clusters of time points close to 0 msec (stimulus onset), with significant differences in the decoding accuracies from about 300 msec to the end of the epoch.

Correlation Between Target and Distractor Decoding Timepoints

We analyzed the correlation between target and distractor decoding accuracy averaged across subjects. We separated the target and distractor decoding accuracy timeseries into two time windows of interest: 0 – 300 msec post-stimulus, and 300 – 1000 msec post-stimulus. This separation was based on the idea that a feedforward sweep of visual input would result in equal representation of target and distractor stimulus information in the early post-stimulus period when bottom-up sensory inputs dominate, as we have observed in related prior work (Noah et al., 2020). We calculated correlation coefficients and accompanying p values with a Pearson correlation of the subject-averaged target decoding accuracy and the subject-averaged distractor decoding accuracy. We found that in the first post-stimulus period, target and distractor decoding were highly positively correlated, while in the second period, target and distractor decoding were negatively correlated (Figure 6). Both correlations were statistically significant (p < 0.05).

Notably, these correlations are among across-subjects mean decoding values. Therefore, because subject variance is ignored, the scope of inference in these statistics is the population of timepoints in each stimulus time period. To draw inference about how early and late stimulus period decoding differences between targets and distractors differ across a population of individuals, we performed a separate analysis.

First, we calculated subject-specific target-distractor correlation coefficients in both the early and late stimulus period. Then, we performed a paired sample t-test between the early and late periods’ correlation coefficients. This test revealed a significant difference between the early and late period (p < 0.05).

SVM Weight Maps

We constructed topomaps of SVM classifier weights to examine how the trained SVM model loads onto different scalp regions. In Figure 7, we present weight maps for target and distractor stimulus classification at 5 timepoints in the stimulus epoch, from 100 msec after stimulus onset to 900 msec after stimulus onset.

This overview indicates that the way different electrodes are weighted in the classification are not drastically different between the target and distractor models, and suggests that target and distractor classifiers are therefore not drawing from drastically different signal sources, such as visual vs. motor cortex.