Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs

Dhamo H, Manhardt F, Navab N, Tombari F (2021)

Publication Type: Conference contribution

Publication year: 2021

Publisher: Institute of Electrical and Electronics Engineers Inc.

Pages Range: 16332-16341

Conference Proceedings Title: Proceedings of the IEEE International Conference on Computer Vision

Event location: Virtual, Online, CAN

ISBN: 9781665428125

DOI: 10.1109/ICCV48922.2021.01604

Abstract

Controllable scene synthesis consists of generating 3D information that satisfy underlying specifications. Thereby, these specifications should be abstract, i.e. allowing easy user interaction, whilst providing enough interface for detailed control. Scene graphs are representations of a scene, composed of objects (nodes) and inter-object relationships (edges), proven to be particularly suited for this task, as they allow for semantic control on the generated content. Previous works tackling this task often rely on synthetic data, and retrieve object meshes, which naturally limits the generation capabilities. To circumvent this issue, we instead propose the first work that directly generates shapes from a scene graph in an end-to-end manner. In addition, we show that the same model supports scene modification, using the respective scene graph as interface. Leveraging Graph Convolutional Networks (GCN) we train a variational Auto-Encoder on top of the object and edge categories, as well as 3D shapes and scene layouts, allowing latter sampling of new scenes and shapes.

Involved external institutions

Technische Universität München (TUM) DE Germany (DE) Google Inc. US United States (USA) (US)

How to cite

APA:

Dhamo, H., Manhardt, F., Navab, N., & Tombari, F. (2021). Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs. In Proceedings of the IEEE International Conference on Computer Vision (pp. 16332-16341). Virtual, Online, CAN: Institute of Electrical and Electronics Engineers Inc..

MLA:

Dhamo, Helisa, et al. "Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs." Proceedings of the 18th IEEE/CVF International Conference on Computer Vision, ICCV 2021, Virtual, Online, CAN Institute of Electrical and Electronics Engineers Inc., 2021. 16332-16341.

BibTeX: Download