Good Keyframes to Inpaint

Shohei Mori, Dieter Schmalstieg, Denis Kalkofen

Research output: Contribution to journalArticlepeer-review


Diminished Reality (DR) propagates pixels from a keyframe to subsequent frames for real-time inpainting. Keyframe selection has a significant impact on the inpainting quality, but untrained users struggle to identify good keyframes. Automatic selection is not straightforward either, since no previous work has formalized or verified what determines a good keyframe. We propose a novel metric to select good keyframes to inpaint. We examine the heuristics adopted in existing DR inpainting approaches and derive multiple simple criteria measurable from SLAM. To combine these criteria, we empirically analyze their effect on the quality using a novel representative test dataset. Our results demonstrate that the combined metric selects RGBD keyframes leading to high-quality inpainting results more often than a baseline approach in both color and depth domains. Also, we confirmed that our approach has a better ranking ability of distinguishing good and bad keyframes. Compared to random selections, our metric selects keyframes that would lead to higher-quality and more stably converging inpainting results. We present three DR examples, automatic keyframe selection, user navigation, and marker hiding.

Original languageEnglish
Number of pages11
JournalIEEE Transactions on Visualization and Computer Graphics
Early online date23 May 2022
Publication statusE-pub ahead of print - 23 May 2022
Externally publishedYes


  • Cameras
  • Diminished reality
  • good keyframes to inpaint
  • Image reconstruction
  • inpainting
  • keyframe
  • Measurement
  • Real-time systems
  • Simultaneous localization and mapping
  • SLAM
  • Streaming media
  • Three-dimensional displays


Dive into the research topics of 'Good Keyframes to Inpaint'. Together they form a unique fingerprint.

Cite this