Home Publications Research Grants Inventions & Patents Awards Additional Research Activities Faculties & Institutions Research Areas

The Pitfalls of Sample Selection: A Case Study on Lung Nodule Classification

Baltatzis V, Bintsi KM, Folgoc LL, Martinez Manzanera OE, Ellis S, Nair A, Desai S, Glocker B, Schnabel JA (2021)

---

Publication Type: Conference contribution

Publication year: 2021

Journal

Publisher: Springer Science and Business Media Deutschland GmbH

Book Volume: 12928 LNCS

Pages Range: 201-211

Conference Proceedings Title: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Event location: Virtual, Online

ISBN: 9783030876012

DOI: 10.1007/978-3-030-87602-9_19

Abstract

Using publicly available data to determine the performance of methodological contributions is important as it facilitates reproducibility and allows scrutiny of the published results. In lung nodule classification, for example, many works report results on the publicly available LIDC dataset. In theory, this should allow a direct comparison of the performance of proposed methods and assess the impact of individual contributions. When analyzing seven recent works, however, we find that each employs a different data selection process, leading to largely varying total number of samples and ratios between benign and malignant cases. As each subset will have different characteristics with varying difficulty for classification, a direct comparison between the proposed methods is thus not always possible, nor fair. We study the particular effect of truthing when aggregating labels from multiple experts. We show that specific choices can have severe impact on the data distribution where it may be possible to achieve superior performance on one sample distribution but not on another. While we show that we can further improve on the state-of-the-art on one sample selection, we also find that on a more challenging sample selection, on the same database, the more advanced models underperform with respect to very simple baseline methods, highlighting that the selected data distribution may play an even more important role than the model architecture. This raises concerns about the validity of claimed methodological contributions. We believe the community should be aware of these pitfalls and make recommendations on how these can be avoided in future work.

Involved external institutions

How to cite

APA:

Baltatzis, V., Bintsi, K.M., Folgoc, L.L., Martinez Manzanera, O.E., Ellis, S., Nair, A.,... Schnabel, J.A. (2021). The Pitfalls of Sample Selection: A Case Study on Lung Nodule Classification. In Islem Rekik, Ehsan Adeli, Sang Hyun Park, Julia Schnabel (Eds.), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (pp. 201-211). Virtual, Online: Springer Science and Business Media Deutschland GmbH.

MLA:

Baltatzis, Vasileios, et al. "The Pitfalls of Sample Selection: A Case Study on Lung Nodule Classification." Proceedings of the 4th International Workshop on Predictive Intelligence in Medicine, PRIME 2021, held in conjunction with 24th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2021, Virtual, Online Ed. Islem Rekik, Ehsan Adeli, Sang Hyun Park, Julia Schnabel, Springer Science and Business Media Deutschland GmbH, 2021. 201-211.

BibTeX: Download