Self-supervised representation learning using multimodal Transformer for emotion recognition
Goetz T, Arora P, Erick FX, Holzer N, Sawant S (2023)
Publication Type: Conference contribution
Publication year: 2023
Publisher: Association for Computing Machinery
Conference Proceedings Title: ACM International Conference Proceeding Series
Event location: Lubeck, DEU
ISBN: 9798400708169
Abstract
In this paper, we present a Modality-Agnostic Transformer based Self-Supervised Learning (MATS2L) for emotion recognition using physiological signals. The proposed approach consists of two stages: a) Pretext stage, where the transformer model is pre-trained with unlabeled physiological signal data using masked signal prediction as pre-training task and form contextualized signal representations. b) Downstream stage, where self-supervised learning (SSL) representations extracted from a pre-trained model are utilized for emotion recognition tasks. Modality-agnostic approach allows the transformer model to focus on exploring mutual features among different physiological signals and learning more meaningful embeddings to estimate emotions effectively. We conduct several experiments on a public dataset WESAD and perform comparisons with fully supervised and other competitive SSL approaches. Experimental results showed that the proposed approach is capable of learning meaningful features and superior to other competitive SSL approaches. Moreover, a transformer model trained on SSL features outperforms fully supervised transformer model. We also present detailed ablation studies to prove the robustness of our approach.
Involved external institutions
Germany (DE)How to cite
APA:
Goetz, T., Arora, P., Erick, F.X., Holzer, N., & Sawant, S. (2023). Self-supervised representation learning using multimodal Transformer for emotion recognition. In Denys J.C. Matthies, Marcin Grzegorzek, Arjan Kuijper, Heike Leutheuser (Eds.), ACM International Conference Proceeding Series. Lubeck, DEU: Association for Computing Machinery.
MLA:
Goetz, Theresa, et al. "Self-supervised representation learning using multimodal Transformer for emotion recognition." Proceedings of the 8th International Workshop on Sensor-based Activity Recognition and Artificial Intelligence, iWOAR 2023, Lubeck, DEU Ed. Denys J.C. Matthies, Marcin Grzegorzek, Arjan Kuijper, Heike Leutheuser, Association for Computing Machinery, 2023.
BibTeX: Download