Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs - Archive ouverte HAL Access content directly
Conference Papers Year : 2022

Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs

(1) , (1) , (1, 2) , (1, 3) , (1, 3)
1
2
3

Abstract

Reconstructing perceived natural images from fMRI signals is one of the most engaging topics of neural decoding research. Prior studies had success in reconstructing either the low-level image features or the semantic/high-level aspects, but rarely both. In this study, we utilized an Instance-Conditioned GAN (IC-GAN) model to reconstruct images from fMRI patterns with both accurate semantic attributes and preserved low-level details. The IC-GAN model takes as input a 119-dim noise vector and a 2048-dim instance feature vector extracted from a target image via a self-supervised learning model (SwAV ResNet-50); these instance features act as a conditioning for IC-GAN image generation, while the noise vector introduces variability between samples. We trained ridge regression models to predict instance features, noise vectors, and dense vectors (the output of the first dense layer of the IC-GAN generator) of stimuli from corresponding fMRI patterns. Then, we used the IC-GAN generator to reconstruct novel test images based on these fMRI-predicted variables. The generated images presented state-of-the-art results in terms of capturing the semantic attributes of the original test images while remaining relatively faithful to low-level image details. Finally, we use the learned regression model and the IC-GAN generator to systematically explore and visualize the semantic features that maximally drive each of several regions-of-interest in the human brain.
Fichier principal
Vignette du fichier
Ozcelik-etal-arXiv-2202.12692.pdf (14.81 Mo) Télécharger le fichier
Origin : Publisher files allowed on an open archive

Dates and versions

hal-03859829 , version 1 (28-11-2022)

Licence

Attribution - CC BY 4.0

Identifiers

Cite

Furkan Ozcelik, Bhavin Choksi, Milad Mozafari, Leila Reddy, Rufin Van-Rullen. Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs. International Joint Conference on Neural Networks (IJCNN 2022), Jul 2022, Padua, Italy. pp.1-9, ⟨10.48550/arXiv.2202.12692⟩. ⟨hal-03859829⟩
0 View
0 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More