Guide Me: Interacting with Deep Networks
Rupprecht C, Laina I, Navab N, Hager GD, Tombari F (2018)
Publication Type: Conference contribution
Publication year: 2018
Journal
Publisher: IEEE Computer Society
Pages Range: 8551-8561
Conference Proceedings Title: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Event location: Salt Lake City, UT, USA
ISBN: 9781538664209
Abstract
Interaction and collaboration between humans and intelligent machines has become increasingly important as machine learning methods move into real-world applications that involve end users. While much prior work lies at the intersection of natural language and vision, such as image captioning or image generation from text descriptions, less focus has been placed on the use of language to guide or improve the performance of a learned visual processing algorithm. In this paper, we explore methods to flexibly guide a trained convolutional neural network through user input to improve its performance during inference. We do so by inserting a layer that acts as a spatio-semantic guide into the network. This guide is trained to modify the network's activations, either directly via an energy minimization scheme or indirectly through a recurrent model that translates human language queries to interaction weights. Learning the verbal interaction is fully automatic and does not require manual text annotations. We evaluate the method on two datasets, showing that guiding a pre-trained network can improve performance, and provide extensive insights into the interaction between the guide and the CNN.
Involved external institutions
Germany (DE)How to cite
APA:
Rupprecht, C., Laina, I., Navab, N., Hager, G.D., & Tombari, F. (2018). Guide Me: Interacting with Deep Networks. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 8551-8561). Salt Lake City, UT, USA: IEEE Computer Society.
MLA:
Rupprecht, Christian, et al. "Guide Me: Interacting with Deep Networks." Proceedings of the 31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA IEEE Computer Society, 2018. 8551-8561.
BibTeX: Download