Feeling Positive? Predicting Emotional Image Similarity from Brain Signals

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review


The present notion of visual similarity is based on features derived from image contents. This ignores the users' emotional or affective experiences toward the content, and how users feel when they search for images. Here we consider valence, a positive or negative quantification of affective appraisal, as a novel dimension of image similarity. We report the largest neuroimaging experiment that quantifies and predicts the valence of visual content by using functional near-infrared spectroscopy from brain-computer interfacing. We show that affective similarity can be (1)∼decoded directly from brain signals in response to visual stimuli, (2)∼utilized for predicting affective image similarity with an average accuracy of 0.58 and an accuracy of 0.65 for high-arousal stimuli, and (3)∼effectively used to complement affective similarity estimates of content-based models; for example when fused fNIRS and image rankings the retrieval F-measure@20 is 0.70. Our work opens new research avenues for affective multimedia analysis, retrieval, and user modeling.

Original languageEnglish
Title of host publicationMM 2023 - Proceedings of the 31st ACM International Conference on Multimedia
Number of pages9
Publication date26 Oct 2023
ISBN (Electronic)979-8-4007-0108-5
Publication statusPublished - 26 Oct 2023
MoE publication typeA4 Article in conference proceedings
EventACM International Conference on Multimedia - Ottawa, Canada
Duration: 29 Oct 20233 Nov 2023
Conference number: 31

Fields of Science

  • affective computing
  • bci
  • ranking relevance
  • 113 Computer and information sciences

Cite this