Neural Bokeh: Learning Lens Blur for Computational Videography and Out-of-Focus Mixed Reality

David Mandl, Shohei Mori, Peter Mohr, Yifan Peng, Tobias Langlotz, Dieter Schmalstieg, Denis Kalkofen

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

We present Neural Bokeh, a deep learning approach for synthesizing convincing out-of-focus effects with applications in Mixed Reality (MR) image and video compositing. Unlike existing approaches that solely learn the amount of blur for out-of-focus areas, our approach captures the overall characteristic of the bokeh to enable the seamless integration of rendered scene content into real images, ensuring a consistent lens blur over the resulting MR composition. Our method learns spatially varying blur shapes, i.e., bokeh, from a dataset of real images acquired using the physical camera that is used to capture the photograph or video of the MR composition. Accordingly, those learned blur shapes mimic the characteristics of the physical lens. As the run-time and the resulting quality of Neural Bokeh increase with the resolution of input images, we employ low-resolution images for the MR view finding at runtime and high-resolution renderings for compositing with high-resolution photographs or videos in an offline process. We envision a variety of applications, including visual enhancement of image and video compositing containing creative utilization of out-of-focus effects.

Original languageEnglish
Title of host publicationProceedings - 2024 IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2024
PublisherInstitute of Electrical and Electronics Engineers
Pages870-880
Number of pages11
ISBN (Electronic)9798350374025
DOIs
Publication statusPublished - 15 Apr 2024
Event31st IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2024 - Orlando, United States
Duration: 16 Mar 202421 Mar 2024

Publication series

NameProceedings - 2024 IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2024

Conference

Conference31st IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2024
Country/TerritoryUnited States
CityOrlando
Period16/03/2421/03/24

Keywords

  • Bokeh
  • Computational Videography
  • Deep Learning
  • Depth of Field
  • Mixed Reality

Fingerprint

Dive into the research topics of 'Neural Bokeh: Learning Lens Blur for Computational Videography and Out-of-Focus Mixed Reality'. Together they form a unique fingerprint.

Cite this