Effects of Label Noise on Deep Learning-Based Skin Cancer Classification

Hekler A, Kather JN, Krieghoff-Henning E, Utikal JS, Meier F, Gellrich FF, Upmeier zu Belzen J, French L, Schlager JG, Ghoreschi K, Wilhelm T, Kutzner H, Berking C, Heppt M, Haferkamp S, Sondermann W, Schadendorf D, Schilling B, Izar B, Maron R, Schmitt M, Fröhling S, Lipka DB, Brinker TJ (2020)

Publication Type: Journal article

Publication year: 2020

Journal

Frontiers in Medicine Frontiers Media

Book Volume: 7

Article Number: 177

DOI: 10.3389/fmed.2020.00177

Abstract

Recent studies have shown that deep learning is capable of classifying dermatoscopic images at least as well as dermatologists. However, many studies in skin cancer classification utilize non-biopsy-verified training images. This imperfect ground truth introduces a systematic error, but the effects on classifier performance are currently unknown. Here, we systematically examine the effects of label noise by training and evaluating convolutional neural networks (CNN) with 804 images of melanoma and nevi labeled either by dermatologists or by biopsy. The CNNs are evaluated on a test set of 384 images by means of 4-fold cross validation comparing the outputs with either the corresponding dermatological or the biopsy-verified diagnosis. With identical ground truths of training and test labels, high accuracies with 75.03% (95% CI: 74.39–75.66%) for dermatological and 73.80% (95% CI: 73.10–74.51%) for biopsy-verified labels can be achieved. However, if the CNN is trained and tested with different ground truths, accuracy drops significantly to 64.53% (95% CI: 63.12–65.94%, p < 0.01) on a non-biopsy-verified and to 64.24% (95% CI: 62.66–65.83%, p < 0.01) on a biopsy-verified test set. In conclusion, deep learning methods for skin cancer classification are highly sensitive to label noise and future work should use biopsy-verified training images to mitigate this problem.

Authors with CRIS profile

Carola Berking Lehrstuhl für Haut- und Geschlechtskrankheiten Markus Heppt Department of Dermatology

Involved external institutions

Deutsches Krebsforschungszentrum (DKFZ) DE Germany (DE) Dana–Farber Cancer Institute US United States (USA) (US) Ruprecht-Karls-Universität Heidelberg DE Germany (DE) Ludwig-Maximilians-Universität (LMU) DE Germany (DE) Charité - Universitätsmedizin Berlin DE Germany (DE) Universitätsklinikum Carl Gustav Carus Dresden DE Germany (DE) Universitätsklinikum Essen DE Germany (DE) Universitätsklinikum Regensburg DE Germany (DE) Universitätsklinikum Würzburg DE Germany (DE) Berliner Institut für Gesundheitsforschung (BIH) DE Germany (DE)

How to cite

APA:

Hekler, A., Kather, J.N., Krieghoff-Henning, E., Utikal, J.S., Meier, F., Gellrich, F.F.,... Brinker, T.J. (2020). Effects of Label Noise on Deep Learning-Based Skin Cancer Classification. Frontiers in Medicine, 7. https://doi.org/10.3389/fmed.2020.00177

MLA:

Hekler, Achim, et al. "Effects of Label Noise on Deep Learning-Based Skin Cancer Classification." Frontiers in Medicine 7 (2020).

BibTeX: Download