Understanding metric-related pitfalls in image analysis validation

Reinke A, Tizabi MD, Baumgartner M, Eisenmann M, Heckmann-Nötzel D, Kavur AE, Rädsch T, Sudre CH, Acion L, Antonelli M, Arbel T, Bakas S, Benis A, Buettner F, Cardoso MJ, Cheplygina V, Chen J, Christodoulou E, Cimini BA, Farahani K, Ferrer L, Galdran A, van Ginneken B, Glocker B, Godau P, Hashimoto DA, Hoffman MM, Huisman M, Isensee F, Jannin P, Kahn CE, Kainmueller D, Kainz B, Karargyris A, Kleesiek J, Kofler F, Kooi T, Kopp-Schneider A, Kozubek M, Kreshuk A, Kurc T, Landman BA, Litjens G, Madani A, Maier-Hein K, Martel AL, Meijering E, Menze B, Moons KG, Müller H, Nichyporuk B, Nickel F, Petersen J, Rafelski SM, Rajpoot N, Reyes M, Riegler MA, Rieke N, Saez-Rodriguez J, Sánchez CI, Shetty S, Summers RM, Taha AA, Tiulpin A, Tsaftaris SA, Van Calster B, Varoquaux G, Yaniv ZR, Jäger PF, Maier-Hein L (2024)

Publication Type: Journal article

Publication year: 2024

Journal

Book Volume: 21

Pages Range: 182-194

Journal Issue: 2

DOI: 10.1038/s41592-023-02150-0

Abstract

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Authors with CRIS profile

Involved external institutions

Deutsches Krebsforschungszentrum (DKFZ) Germany (DE) University College London (UCL) United Kingdom (GB) Universidad de Buenos Aires (UBA) / University of Buenos Aires Argentina (AR) Indiana University – Purdue University Indianapolis United States (USA) (US) Holon Institute of Technology Israel (IL) King’s College London United Kingdom (GB) IT University of Copenhagen Germany (DE) Leibniz-Institut für Analytische Wissenschaften / Leibniz Institute for Analytical Sciences (ISAS) Germany (DE) Eli and Edythe L. Broad Institute of MIT and Harvard United States (USA) (US) National Cancer Institute (NCI) United States (USA) (US) Universitat Pompeu Fabra (UPF) Spain (ES) Fraunhofer-Institut für Bildgestützte Medizin (MEVIS) Germany (DE) Imperial College London / The Imperial College of Science, Technology and Medicine United Kingdom (GB) Perelman School of Medicine University of Pennsylvania United States (USA) (US) Princess Margaret Cancer Centre / Princess Margaret Hospital Canada (CA) Radboud University Nijmegen Medical Centre / Radboudumc of voluit Radboud Universitair Medisch Centrum (UMC) Netherlands (NL) Université de Rennes 1 / University of Rennes 1 France (FR) Penn Medicine United States (USA) (US) Max-Delbrück-Centrum für Molekulare Medizin / Max Delbrück Center for Molecular Medicine (MDC) Berlin-Buch Germany (DE) Helmholtz-Gemeinschaft / Hermann von Helmholtz-Gemeinschaft Deutscher Forschungszentren e.V. Germany (DE) Lunit Inc. Korea, Republic of (KR) Masaryk University Czech Republic (CZ) European Molecular Biology Laboratory (EMBL) Germany (DE) State University of New York at Albany (UNY Albany / UAlbany) United States (USA) (US) Vanderbilt University United States (USA) (US) St. Luke's University Health Network (SLUHN) United States (USA) (US) University of Toronto Canada (CA) University of New South Wales (UNSW) Australia (AU) University of Zurich / Universität Zürich (UZH) Switzerland (CH) University Medical Centre Utrecht (UMC Utrecht) Netherlands (NL) Haute école spécialisée de Suisse occidentale (HES-SO) / Fachhochschule Westschweiz Switzerland (CH) Quebec AI Institute / Quebec Artificial Intelligence Institute / Montreal Institute for Learning Algorithms (MILA) Canada (CA) Universitätsklinikum Hamburg-Eppendorf (UKE) Germany (DE) Allen Institute for Cell Science United States (USA) (US) University of Warwick United Kingdom (GB) Universität Bern Switzerland (CH) Simula Metropolitan Center for Digital Engineering (SimulaMet) Norway (NO) Nvidia Corporation United States (USA) (US) Ruprecht-Karls-Universität Heidelberg Germany (DE) University of Amsterdam Netherlands (NL) McGill University Canada (CA) Institut de Chirurgie Guidée par l’Image de Strasbourg (IHU) / Institute of Image-Guided Surgery France (FR) Universitätsklinikum Essen Germany (DE) Google Ireland Limited Ireland (IE) National Institutes of Health Clinical Center United States (USA) (US) Technische Universität Wien / Vienna University of Technology Austria (AT) Oulun Yliopisto / University of Oulo Finland (FI) University of Edinburgh United Kingdom (GB) Katholieke Universiteit Leuven (KUL) / Catholic University of Leuven Belgium (BE) Inria Saclay - Île-de-France Research Centre France (FR) National Institute of Allergy and Infectious Diseases United States (USA) (US)

How to cite

APA:

Reinke, A., Tizabi, M.D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A.E.,... Maier-Hein, L. (2024). Understanding metric-related pitfalls in image analysis validation. Nature Methods, 21(2), 182-194. https://doi.org/10.1038/s41592-023-02150-0

MLA:

Reinke, Annika, et al. "Understanding metric-related pitfalls in image analysis validation." Nature Methods 21.2 (2024): 182-194.

BibTeX: Download