README.md

metadata

license: apache-2.0
tags:
  - pytorch
  - computer-vision
  - neuroscience
  - fmri
  - encoding-model
  - ridge-regression
  - variance-partitioning
  - natural-scenes-dataset
  - simclr
  - resnet18
  - egocentric-vision
  - visual-neuroscience
  - vedb
  - arxiv:2607.19316

VEDB and Reference SimCLR ResNet-18 — NSD Voxelwise Encoding Models

This repository contains subject-specific NSD voxelwise encoding-model fits and the variance-partitioning fits reported in the associated study from:

Diaz, D. M., & Henderson, M. M. (2026). Eccentricity-Constrained CNN Training Reveals Adaptive Information Coding Around the Visual Field. Proceedings of the 9th Conference on Cognitive Computational Neuroscience.

DOI: 10.32470/0416gfsq
arXiv: 2607.19316
Contributed Talk: CCN 2026 presentation on YouTube

Voxelwise encoding models were fit to fMRI responses from the Natural Scenes Dataset (NSD) using features extracted from seven SimCLR ResNet-18 models evaluated in the associated study.

The models comprise four ResNet-18 encoders pretrained on the Visual Experience Dataset (VEDB) under different visual-field conditions:

• Baseline
• Fovea-Gaze
• Periph
• Periph-NF

and three non-egocentric reference models pretrained on conventional image datasets:

• STL-10
• ImageNet-100
• ImageNet-1K

The ImageNet-100 and ImageNet-1K reference models were trained for the associated study using SimCLR implemented with the Lightly self-supervised learning framework. The STL-10 model was obtained from the pretrained ResNet-18 SimCLR release provided by Spijkervet/SimCLR.

The reference models provide comparison points for evaluating how representations learned from naturalistic, gaze-aligned egocentric visual experience correspond with human visual cortex relative to representations learned from conventional image datasets.

Code, preprocessing, analysis, and other related material are hosted on Github: DM-Diaz/eccentricity-constrained-simclr

Visual-Field Conditions

The four pretrained models were trained on complementary versions of the same source imagery:

• Baseline: full-field input without an eccentricity-specific restriction
• Fovea-Gaze: gaze-centered central-only input
• Periph: peripheral-only input produced by masking the gaze-centered central region
• Periph-NF: peripheral-only input with a NeuroFovea transform applied before central masking

Release Status

Component	Status
VEDB subject-specific encoding-model fits	Available
Reference-model encoding fits (STL-10, ImageNet-100, ImageNet-1K)	Available
Variance-partitioning fits reported in the paper	Available
Model card	Available
Feature-extraction / model-fitting code	Available github
Analysis code	Available github
VEDB imagery	Full dataset not redistributed; available via Databrary
NSD stimuli and fMRI data	Not redistributed; repository contains derived encoding-model and variance-partitioning fits only; see NSD

Repository Structure

baseline/
fovea-gaze/
periph/
periph-nf/
reference-models/
├── stl10/
├── imagenet-100/
└── imagenet-1k/
variance-partitioning/
├── fovea-gaze-vs-periph/
└── periph-vs-periph-nf/

Each encoding-model folder contains voxelwise fits for NSD subjects S1–S8. The four VEDB folders contain fits derived from the Baseline, Fovea-Gaze, Periph, and Periph-NF models. The reference-models/ directory contains the corresponding fits derived from the STL-10, ImageNet-100, and ImageNet-1K reference models. The variance-partitioning/ directory contains the two model comparisons reported in Figure 4C–D of the paper, with one fit for each of the same eight NSD participants.

Examples:

fovea-gaze/
├── NSD_S1_resnet18-Fovea-Gaze_concat.npy
├── NSD_S2_resnet18-Fovea-Gaze_concat.npy
├── ...
└── NSD_S8_resnet18-Fovea-Gaze_concat.npy

reference-models/
├── stl10/
│   └── NSD_S1_resnet18-simclr-stl10_concat.npy
├── imagenet-100/
│   └── NSD_S1_resnet18-simclr-imagenet100_concat.npy
└── imagenet-1k/
    └── NSD_S1_resnet18-simclr-imagenet1k_concat.npy

Architecture

Overview of the VEDB preprocessing, SimCLR pretraining, downstream linear probes, and voxelwise encoding workflow.

NSD Data and Evaluation

Voxelwise encoding models were fit using fMRI responses from the Natural Scenes Dataset (NSD), which contains 7T whole-brain fMRI measurements collected while participants viewed large sets of natural-scene images. The present analysis used data from 8 NSD participants.

For each participant, the 1,000 images shared across all NSD participants were reserved as the final held-out evaluation set. The remaining approximately 9,000 participant-specific images were used for model fitting and regularization selection. Within this fitting set, a nested holdout partition was used to select the ridge penalty independently for each voxel.

The same intact 224 × 224 NSD images were presented to each of the seven pretrained ResNet-18 models. The Baseline, Fovea-Gaze, Periph, and Periph-NF transformations were applied only during VEDB SimCLR pretraining and were not reapplied to NSD stimuli during feature extraction. The same NSD feature-extraction procedure was used for the STL-10, ImageNet-100, and ImageNet-1K reference models.

Encoding performance was evaluated on the held-out 1,000-image set using voxelwise coefficient of determination (R²) and Pearson correlation (corr). These held-out metrics are stored directly in each released encoding-model fit.

Encoding-Model Procedure

The encoding models were fit separately for each NSD subject (S1–S8) and each of the seven pretrained visual models.

NSD stimuli were presented to the pretrained ResNet-18 models as intact 224 × 224 images. The Baseline, Fovea-Gaze, Periph, and Periph-NF transformations were used during SimCLR pretraining and were not reapplied to NSD images during encoding-model evaluation.

Before feature extraction, NSD images were rescaled to [0, 1] and normalized using ImageNet channel statistics:

mean = [0.485, 0.456, 0.406]
std  = [0.229, 0.224, 0.225]

Features were extracted from six points in the ResNet-18 encoder:

conv1
layer1.1
layer2.1
layer3.1
layer4.1
avgpool

Convolutional feature maps were reduced with adaptive average pooling and then flattened. PCA was applied separately to each layer, retaining the top 200 principal components. The six 200-component representations were then concatenated before voxelwise model fitting.

PCA was fit separately for each subject, model condition, and feature layer. In the analysis used for the study, PCA was applied to the full subject-specific feature matrix before the encoding-model train and holdout partitions were applied.

Feature Normalization and Ridge Fitting

For each subject, the feature matrix was divided into a training partition, a nested holdout partition for ridge-penalty selection, and a final held-out evaluation partition containing the 1,000 NSD images shared across participants.

Feature-wise means and standard deviations were estimated from the combined training and nested-holdout partitions. The same parameters were then used to z-score all three partitions. The final held-out evaluation set was excluded when estimating these normalization parameters.

A column of ones was appended to the feature matrix to provide an intercept term.

Voxel responses were modeled with L2-regularized linear regression (ridge regression). Twenty candidate ridge penalties were evaluated. The penalty that minimized prediction error on the nested holdout partition was selected independently for each voxel.

Performance on the final held-out set was quantified with voxelwise coefficient of determination (R²) and Pearson correlation (corr).

Saved Fit Contents

Each .npy file contains a Python dictionary with the fitted encoding model and associated metadata:

• subject — NSD subject number
• model — visual model identifier
• features_file_list — feature files used for model fitting
• lambdas — candidate ridge penalties
• weights — fitted voxelwise ridge-regression weights
• best_lambda_inds — selected ridge-penalty index for each voxel
• r2 — held-out voxelwise coefficient of determination
• corr — held-out voxelwise correlation
• voxel_mask — subject voxel mask
• voxel_index — voxel indices
• voxel_nc — voxel noise-ceiling estimates
• brain_nii_shape — original volumetric brain-image shape

Because the intercept was added as a final column in the feature matrix, the final row of weights contains the fitted intercept term.

Internal Model Labels

The released .npy files retain the model identifiers used by the original NSD feature-extraction and encoding-model analysis pipeline. These identifiers are analysis-time labels and should not be interpreted as model names stored within the underlying SimCLR checkpoints.

Public model name	Encoding-analysis identifier
Baseline	resnet18-Baseline
Fovea-Gaze	resnet18-FoveaGaze
Periph	resnet18-PeriphNonTTM
Periph-NF	resnet18-PeriphTTM
STL-10	resnet18-pretrained-simclr
ImageNet-100	resnet18-simclr-imgnet100
ImageNet-1K	resnet18-simclr-imgnet1k

The ImageNet-100 and ImageNet-1K PyTorch Lightning checkpoints themselves store model parameters under backbone.* and projection_head.* namespaces and do not contain the dataset-specific analysis identifiers shown above. The STL-10 checkpoint obtained from Spijkervet/SimCLR uses its original encoder.* and projector.* parameter naming, which was remapped during feature extraction.

The internal identifier resnet18-pretrained-simclr corresponds to the STL-10 SimCLR ResNet-18 checkpoint obtained from the external Spijkervet/SimCLR release.

These analysis identifiers are preserved in the model field of the released encoding-model fits and, where applicable, in fields such as model1, model2, and variance-partitioning dictionary keys. They reflect the naming conventions used during model development and do not indicate additional model conditions.

The features_file_list, features_file_list1, and features_file_list2 fields also preserve the original analysis paths for provenance. These paths refer to the computing environment used for the original study and are not expected to resolve outside that environment.

Variance-Partitioning Fits

The repository also contains the subject-specific variance-partitioning fits reported in the associated paper:

• Fovea-Gaze vs. Periph — Figure 4C
• Periph vs. Periph-NF — Figure 4D

For each comparison and NSD participant, three voxelwise encoding models were fit using (1) features from model A alone, (2) features from model B alone, and (3) the concatenated feature spaces from both models. Unique variance was computed from held-out voxelwise R² using the equations reported in the paper:

$$R_{\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{q}\mathrm{u}\mathrm{e},A}^2=R_{\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{b}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}}^2-R_{B\text{-only}}^{2}R^2\_\{\backslash mathrm\{unique\},A\}\; =\; R^2\_\{\backslash mathrm\{combined\}\}\; -\; R^2\_\{B\backslash text\{-only\}\}$$

$$R_{\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{q}\mathrm{u}\mathrm{e},B}^2=R_{\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{b}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}}^2-R_{A\text{-only}}^{2}R^2\_\{\backslash mathrm\{unique\},B\}\; =\; R^2\_\{\backslash mathrm\{combined\}\}\; -\; R^2\_\{A\backslash text\{-only\}\}$$

Thus, the unique variance attributed to one model is the variance explained by the combined representation minus the variance explained by the other model alone.

Variance-partitioning files are stored under:

variance-partitioning/
├── fovea-gaze-vs-periph/
│   ├── NSD_S1_varpart_Fovea-Gaze_vs_Periph.npy
│   ├── ...
│   └── NSD_S8_varpart_Fovea-Gaze_vs_Periph.npy
└── periph-vs-periph-nf/
    ├── NSD_S1_varpart_Periph_vs_Periph-NF.npy
    ├── ...
    └── NSD_S8_varpart_Periph_vs_Periph-NF.npy

Each variance-partitioning file contains a Python dictionary with:

• subject
• model1
• model2
• features_file_list1
• features_file_list2
• lambdas
• voxel_mask
• voxel_index
• voxel_nc
• brain_nii_shape
• weights_varpart
• r2_varpart
• corr_varpart
• best_lambda_inds_varpart

The weights_varpart, r2_varpart, corr_varpart, and best_lambda_inds_varpart fields are dictionaries with entries for model A alone, model B alone, and the combined feature space. Their exact keys retain the original internal model identifiers listed above.

Because the six retained layers contribute 200 PCA components each, a single-model variance-partition fit contains 1,200 model features plus one intercept, giving weights_varpart matrices with 1,201 rows for the model-only fits. The combined fit contains 2,400 model features plus one intercept, giving 2,401 rows.

Example:

import numpy as np

fit = np.load(
    "variance-partitioning/fovea-gaze-vs-periph/"
    "NSD_S1_varpart_Fovea-Gaze_vs_Periph.npy",
    allow_pickle=True
).item()

r2 = fit["r2_varpart"]

model_a = fit["model1"]
model_b = fit["model2"]

r2_a_only = r2[f"{model_a}-only"]
r2_b_only = r2[f"{model_b}-only"]
r2_combined = r2["combined"]

r2_unique_a = r2_combined - r2_b_only
r2_unique_b = r2_combined - r2_a_only

Loading a Fit

import numpy as np

fit = np.load(
    "baseline/NSD_S1_resnet18-Baseline_concat.npy",
    allow_pickle=True
).item()

weights = fit["weights"]
r2 = fit["r2"]
corr = fit["corr"]

lambdas = fit["lambdas"]
best_lambda_inds = fit["best_lambda_inds"]

voxel_index = fit["voxel_index"]
voxel_nc = fit["voxel_nc"]

Scope of the Released Artifacts

The repository contains 56 subject-specific encoding-model fits across seven pretrained visual models: 32 fits from the four VEDB-pretrained models (4 models × 8 NSD participants) and 24 fits from the three non-egocentric reference models (3 models × 8 NSD participants).

The repository additionally contains 16 variance-partitioning fits corresponding to the two variance-partition analyses reported in the paper (2 model comparisons × 8 participants), for a total of 72 fitted .npy artifacts.

The released files contain fitted voxelwise weights, held-out prediction metrics, ridge-penalty information, and voxel metadata used in the study. The variance-partitioning artifacts additionally contain the model-A-only, model-B-only, and combined fits used to compute unique variance.

Reproducing voxel predictions for new images also requires the corresponding pretrained ResNet-18 checkpoint and the feature-extraction, spatial-pooling, PCA, concatenation, and normalization procedures used during fitting. These steps will be documented in the accompanying public code release.

Related Models

This repository is associated with the Eccentricity-Constrained SimCLR Models (VEDB) collection.

• VEDB SimCLR ResNet-18 — Baseline
• VEDB SimCLR ResNet-18 — Fovea-Gaze
• VEDB SimCLR ResNet-18 — Periph
• VEDB SimCLR ResNet-18 — Periph-NF
• VEDB NSD ResNet-18 — Encoding Models
• SimCLR ResNet-18 — ImageNet-1K
• SimCLR ResNet-18 — ImageNet-100
• SimCLR ResNet-18 — STL-10 (external pretrained reference model; checkpoint provided by Spijkervet/SimCLR and not redistributed by this project)

Computational Resources

The computational experiments and analyses for this study were conducted primarily using Carnegie Mellon University Neuroscience Institute's MiND computing cluster.

Citation

If you use these encoding-model fits in academic work, please cite the associated study:

@inproceedings{diaz2026eccentricity,
  author    = {Diaz, Dylan M. and Henderson, Margaret M.},
  title     = {Eccentricity-Constrained CNN Training Reveals Adaptive Information Coding Around the Visual Field},
  booktitle = {Proceedings of the 9th Conference on Cognitive Computational Neuroscience},
  address   = {New York, NY, USA},
  year      = {2026},
  doi       = {10.32470/0416gfsq}
}

Proceedings: Conference on Cognitive Computational Neuroscience 2026
Preprint: arXiv:2607.19316

NSD Citation

Researchers using the Natural Scenes Dataset should cite:

Allen, E. J., et al. (2022). A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25, 116–126. https://doi.org/10.1038/s41593-021-00962-x

See the Natural Scenes Dataset paper for the complete author list and dataset description.

VEDB Citation

Researchers using the underlying Visual Experience Dataset should also cite:

Greene, M. R., et al. (2024). The Visual Experience Dataset: Over 200 recorded hours of integrated eye movement, odometry, and egocentric video. Journal of Vision, 24(11), 6.

See the VEDB dataset paper for the complete author list and citation information.

License

The encoding-model fits in this repository are released under the Apache License 2.0.

The VEDB dataset and other third-party resources used in the associated study remain subject to their respective licenses, access requirements, and terms of use. This repository does not redistribute the full VEDB dataset; a small number of example frames are included to illustrate the published visual-field manipulations.