Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth

Valindria VV, Lavdas I, Bai W, Kamnitsas K, Aboagye EO, Rockall AG, Rueckert D, Glocker B (2017)

Publication Type: Journal article

Publication year: 2017

Journal

IEEE Transactions on Medical Imaging Institute of Electrical and Electronics Engineers (IEEE)

Book Volume: 36

Pages Range: 1597-1606

Article Number: 7902121

Journal Issue: 8

DOI: 10.1109/TMI.2017.2665165

Abstract

When integrating computational tools, such as automatic segmentation, into clinical practice, it is of utmost importance to be able to assess the level of accuracy on new data and, in particular, to detect when an automatic method fails. However, this is difficult to achieve due to the absence of ground truth. Segmentation accuracy on clinical data might be different from what is found through cross validation, because validation data are often used during incremental method development, which can lead to overfitting and unrealistic performance expectations. Before deployment, performance is quantified using different metrics, for which the predicted segmentation is compared with a reference segmentation, often obtained manually by an expert. But little is known about the real performance after deployment when a reference is unavailable. In this paper, we introduce the concept of reverse classification accuracy (RCA) as a framework for predicting the performance of a segmentation method on new data. In RCA, we take the predicted segmentation from a new image to train a reverse classifier, which is evaluated on a set of reference images with available ground truth. The hypothesis is that if the predicted segmentation is of good quality, then the reverse classifier will perform well on at least some of the reference images. We validate our approach on multi-organ segmentation with different classifiers and segmentation methods. Our results indicate that it is indeed possible to predict the quality of individual segmentations, in the absence of ground truth. Thus, RCA is ideal for integration into automatic processing pipelines in clinical routine and as a part of large-scale image analysis studies. © 1982-2012 IEEE.

Involved external institutions

Imperial College London / The Imperial College of Science, Technology and Medicine GB United Kingdom (GB) Royal Marsden Hospital / The Royal Marsden NHS Foundation Trust GB United Kingdom (GB)

How to cite

APA:

Valindria, V.V., Lavdas, I., Bai, W., Kamnitsas, K., Aboagye, E.O., Rockall, A.G.,... Glocker, B. (2017). Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth. IEEE Transactions on Medical Imaging, 36(8), 1597-1606. https://doi.org/10.1109/TMI.2017.2665165

MLA:

Valindria, Vanya V., et al. "Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth." IEEE Transactions on Medical Imaging 36.8 (2017): 1597-1606.

BibTeX: Download