Learning Lightprobes for Mixed Reality Illumination

David Mandl, Kwang Moo Yi, Peter Mohr, Peter M. Roth, Pascal Fua, Vincent Lepetit, Dieter Schmalstieg, Denis Kalkofen

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

32 Citations (Scopus)

Abstract

This paper presents the first photometric registration pipeline for Mixed Reality based on high quality illumination estimation using convolutional neural networks (CNNs). For easy adaptation and deployment of the system, we train the CNNs using purely synthetic images and apply them to real image data. To keep the pipeline accurate and efficient, we propose to fuse the light estimation results from multiple CNN instances and show an approach for caching estimates over time. For optimal performance, we furthermore explore multiple strategies for the CNN training. Experimental results show that the proposed method yields highly accurate estimates for photo-realistic augmentations.

Original languageEnglish
Title of host publicationProceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2017
EditorsWolfgang Broll, Holger Regenbrecht, J Edward Swan
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages82-89
Number of pages8
ISBN (Electronic)978-1-5386-2943-7
DOIs
Publication statusPublished - 23 Nov 2017
Externally publishedYes
Event16th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2017 - Nantes, France
Duration: 9 Oct 201713 Oct 2017

Publication series

NameProceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2017

Conference

Conference16th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2017
Country/TerritoryFrance
CityNantes
Period9/10/1713/10/17

Keywords

  • Artificial
  • augmented
  • Virtual Realities
  • Image Processing and Computer Vision
  • Photometric registration
  • 3D Reconstruction

Fingerprint

Dive into the research topics of 'Learning Lightprobes for Mixed Reality Illumination'. Together they form a unique fingerprint.

Cite this