Joint Learning of Localized Representations from Medical Images and Reports
Mueller P, Kaissis G, Zou C, Rueckert D (2022)
Publication Type: Conference contribution
Publication year: 2022
Journal
Publisher: Springer Science and Business Media Deutschland GmbH
Book Volume: 13686 LNCS
Pages Range: 685-701
Conference Proceedings Title: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Event location: Tel Aviv, ISR
ISBN: 9783031198083
DOI: 10.1007/978-3-031-19809-0_39
Abstract
Contrastive learning has proven effective for pre-training image models on unlabeled data with promising results for tasks such as medical image classification. Using paired text (like radiological reports) during pre-training improves the results even further. Still, most existing methods target image classification downstream tasks and may not be optimal for localized tasks like semantic segmentation or object detection. We therefore propose Localized representation learning from Vision and Text (LoVT), a text-supervised pre-training method that explicitly targets localized medical imaging tasks. Our method combines instance-level image-report contrastive learning with local contrastive learning on image region and report sentence representations. We evaluate LoVT and commonly used pre-training methods on an evaluation framework of 18 localized tasks on chest X-rays from five public datasets. LoVT performs best on 10 of the 18 studied tasks making it the preferred method of choice for localized tasks.
Involved external institutions
Germany (DE)How to cite
APA:
Mueller, P., Kaissis, G., Zou, C., & Rueckert, D. (2022). Joint Learning of Localized Representations from Medical Images and Reports. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner (Eds.), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (pp. 685-701). Tel Aviv, ISR: Springer Science and Business Media Deutschland GmbH.
MLA:
Mueller, Philip, et al. "Joint Learning of Localized Representations from Medical Images and Reports." Proceedings of the 17th European Conference on Computer Vision, ECCV 2022, Tel Aviv, ISR Ed. Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner, Springer Science and Business Media Deutschland GmbH, 2022. 685-701.
BibTeX: Download