Speech-based emotion recognition with self-supervised models using attentive channel-wise correlations and label smoothing

Sofoklis Kakouros, Themos Stafylakis, Ladislav Mošner, Lukáš Burget

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review


When recognizing emotions from speech, we encounter two common problems: how to optimally capture emotion-relevant information from the speech signal and how to best quantify or categorize the noisy subjective emotion labels. Self-supervised pre-trained representations can robustly capture information from speech enabling state-of-the-art results in many downstream tasks including emotion recognition. However, better ways of aggregating the information across time need to be considered as the relevant emotion information is likely to appear piecewise and not uniformly across the signal. For the labels, we need to take into account that there is a substantial degree of noise that comes from the subjective human annotations. In this paper, we propose a novel approach to attentive pooling based on correlations between the representations' coefficients combined with label smoothing, a method aiming to reduce the confidence of the classifier on the training labels. We evaluate our proposed approach on the benchmark dataset IEMOCAP, and demonstrate high performance surpassing that in the literature. The code to reproduce the results is available at github.com/skakouros/s3prl_attentive_correlation.
Original languageEnglish
Title of host publicationICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Number of pages5
Place of PublicationNew York
Publication date2023
ISBN (Electronic)978-1-7281-6327-7
Publication statusPublished - 2023
MoE publication typeA4 Article in conference proceedings
EventIEEE International Conference on Acoustics, Speech, and Signal Processing : Signal Processing in the AI era - Rodos Palace Luxury Convention Resort, Rhodes, Greece
Duration: 4 Jun 202310 Jun 2023
Conference number: 48

Fields of Science

  • 213 Electronic, automation and communications engineering, electronics
  • 113 Computer and information sciences
  • 6121 Languages

Cite this