Cross-Attention Fusion of Acoustic and Linguistic Features for Parkinson’s Disease Detection
Mercado-Agudelo JF, Escobar-Grisales D, Ríos-Urrego CD, Gallo-Aristizábal JD, Orozco-Arroyave JR (2027)
Publication Type: Conference contribution
Publication year: 2027
Journal
Publisher: Springer Science and Business Media Deutschland GmbH
Book Volume: 16940 LNCS
Pages Range: 363-375
Conference Proceedings Title: Lecture Notes in Computer Science
Event location: Brno, CZE
ISBN: 9783032372482
DOI: 10.1007/978-3-032-37249-9_30
Abstract
Parkinson’s disease (PD) causes speech and language disorders, including reduced articulatory precision, monopitch, high semantic variability, and impaired processing of action concepts. This paper presents a multimodal framework that combines window-level Whisper acoustic embeddings with word-level, clinically motivated linguistic features and is trained on PC-GITA and evaluated on an independent test set recorded under uncontrolled conditions. The speech-only model achieves a test AUC of 0.90, while the language-only model achieves 0.63. Cross-attention fusion, in which acoustic window embeddings are enriched with linguistic context, achieves a test AUC of 0.83 with 12% points higher sensitivity than the speech-only baseline, yielding a more balanced sensitivity–specificity profile suited to clinical screening. Attention-weight analysis identifies class-discriminative words present in healthy speech but absent in PD speech, revealing significantly higher motor-semantic and affective activation in healthy controls and linking model behavior to known characteristics of PD-related discourse impairment.
Involved external institutions
Colombia (CO)How to cite
APA:
Mercado-Agudelo, J.F., Escobar-Grisales, D., Ríos-Urrego, C.D., Gallo-Aristizábal, J.D., & Orozco-Arroyave, J.R. (2027). Cross-Attention Fusion of Acoustic and Linguistic Features for Parkinson’s Disease Detection. In Elmar Nöth, Aleš Horák, Petr Sojka (Eds.), Lecture Notes in Computer Science (pp. 363-375). Brno, CZE: Springer Science and Business Media Deutschland GmbH.
MLA:
Mercado-Agudelo, Jhon Fredy, et al. "Cross-Attention Fusion of Acoustic and Linguistic Features for Parkinson’s Disease Detection." Proceedings of the 29th International Conference on Text, Speech, and Dialogue, TSD 2026, Brno, CZE Ed. Elmar Nöth, Aleš Horák, Petr Sojka, Springer Science and Business Media Deutschland GmbH, 2027. 363-375.
BibTeX: Download