Surface Snapping Optimization Layer for Single Image Object Shape Reconstruction

University of Illinois at Urbana-Champaign

1Purdue University

International Conference on Machine Learning (ICML), 2023


Reconstructing the 3D shape of objects observed in a single image is a challenging task. Recent approaches rely on visual cues extracted from a given image learned from a deep net. In this work, we leverage recent advances in monocular scene understanding to incorporate an additional geometric cue of surface normals. For this, we proposed a novel optimization layer that encourages the face normals of the reconstructed shape to be aligned with estimated surface normals. We develop a computationally efficient conjugate-gradient-based method that avoids the computation of high-dimensional sparse matrices. We show this framework to achieve compelling shape reconstruction results on the challenging Pix3D and ShapeNet datasets.





  title = {Surface Snapping Optimization Layer for Single Image Object Shape Reconstruction},
  author = {Yuan-Ting Hu and Schwing, Alexander G and Yeh, Raymond A},
  booktitle = {International Conference on Machine Learning (ICML)},
  year = {2023},