Measuring Progress in Robotics: Benchmarking and the ‘Measure-Target Confusion’
Müller VC (2020)
Publication Type: Book chapter / Article in edited volumes
Publication year: 2020
Journal
Publisher: Springer Verlag
Edited Volumes: Metrics of Sensory Motor Coordination and Integration in Robots and Animals
Series: Cognitive Systems Monographs
Book Volume: 36
Pages Range: 169-179
ISBN: 978-3-030-14126-4
DOI: 10.1007/978-3-030-14126-4_9
Abstract
While it is often said that in order to qualify as a true science robotics should aspire to reproducible and measurable results that allow benchmarking, I argue that a focus on benchmarking will be a hindrance for progress. Several academic disciplines that have been led into pursuing only reproducible and measurable ‘scientific’ results—robotics should be careful not to fall into that trap. Results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems independently of a specific context—so working towards progress on the technical measure risks missing that target. It would constitute aiming for the measure rather than the target: what I call ‘measure-target confusion’. The role of benchmarking in robotics shows that the more general problem to measure progress towards more intelligent machines will not be solved by technical benchmarks; we need a balanced approach with technical benchmarks, real-life testing and qualitative judgment.
Authors with CRIS profile
Involved external institutions
United Kingdom (GB)How to cite
APA:
Müller, V.C. (2020). Measuring Progress in Robotics: Benchmarking and the ‘Measure-Target Confusion’. In Fabio Bonsignorio, Elena Messina, Angel P. del Pobil, John Hallam (Eds.), Metrics of Sensory Motor Coordination and Integration in Robots and Animals. (pp. 169-179). Springer Verlag.
MLA:
Müller, Vincent C.. "Measuring Progress in Robotics: Benchmarking and the ‘Measure-Target Confusion’." Metrics of Sensory Motor Coordination and Integration in Robots and Animals. Ed. Fabio Bonsignorio, Elena Messina, Angel P. del Pobil, John Hallam, Springer Verlag, 2020. 169-179.
BibTeX: Download