Monocular Reconstruction of Neural Face Reflectance Fields

Mallikarjun BR, Tewari A, Oh TH, Weyrich T, Bickel B, Seidel HP, Pfister H, Matusik W, Elgharib M, Theobalt C (2021)

Publication Type: Conference contribution

Publication year: 2021

Journal

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition IEEE Computer Society; 1999

Publisher: IEEE Computer Society

Pages Range: 4789-4798

Conference Proceedings Title: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Event location: Virtual, Online, USA

ISBN: 9781665445092

DOI: 10.1109/CVPR46437.2021.00476

Abstract

The reflectance field of a face describes the reflectance properties responsible for complex lighting effects including diffuse, specular, inter-reflection and self shadowing. Most existing methods for estimating the face reflectance from a monocular image assume faces to be diffuse with very few approaches adding a specular component. This still leaves out important perceptual aspects of reflectance such as higher-order global illumination effects and self-shadowing. We present a new neural representation for face reflectance where we can estimate all components of the reflectance responsible for the final appearance from a monocular image. Instead of modeling each component of the reflectance separately using parametric models, our neural representation allows us to generate a basis set of faces in a geometric deformation-invariant space, parameterized by the input light direction, viewpoint and face geometry. We learn to reconstruct this reflectance field of a face just from a monocular image, which can be used to render the face from any viewpoint in any light condition. Our method is trained on a light-stage dataset, which captures 300 people illuminated with 150 light conditions from 8 viewpoints. We show that our method outperforms existing monocular reflectance reconstruction methods due to better capturing of physical effects, such as sub-surface scattering, specularities, self-shadows and other higher-order effects.

Authors with CRIS profile

Tim Weyrich

Involved external institutions

Max Planck Institute for Informatics / Max-Planck-Institut für Informatik DE Germany (DE) Pohang University of Science and Technology (POSTECH) KR Korea, Republic of (KR) University College London (UCL) GB United Kingdom (GB) IST Austria AT Austria (AT) Harvard University US United States (USA) (US) Massachusetts Institute of Technology (MIT) US United States (USA) (US)

How to cite

APA:

Mallikarjun, B.R., Tewari, A., Oh, T.-H., Weyrich, T., Bickel, B., Seidel, H.-P.,... Theobalt, C. (2021). Monocular Reconstruction of Neural Face Reflectance Fields. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 4789-4798). Virtual, Online, USA: IEEE Computer Society.

MLA:

Mallikarjun, B. R., et al. "Monocular Reconstruction of Neural Face Reflectance Fields." Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2021, Virtual, Online, USA IEEE Computer Society, 2021. 4789-4798.

BibTeX: Download