Markerless Full Body Tracking: Depth Sensing Technology within Virtual Environments

Over the last decade there has been growing recognition of the potential value of virtual reality and game technology for creating a new generation of tools for advancing rehabilitation, training and exercise activities. However, until recently the only way people could interact with digital games and virtual reality simulations, was by using relatively constrained gamepad, joystick and keyboard interface devices. Thus, rather than promoting physical activity, these modes of interaction encourage a more sedentary approach to playing games, typically while seated on the couch or in front of a desk. More complex and expensive motion tracking systems enable immersive interactions but are only available at restricted locations and are not readily available in the home setting. Recent advances in video game technology have fueled a proliferation of low-cost devices that can sense the user’s motion. This paper will present and discuss three potential applications of the new depth-sensing camera technology from PrimeSense and Microsoft Kinect. The paper will outline the technology underlying the sensor, the development of our open source middleware allowing developers to make applications and provide examples of applications that enhance interaction within virtual environments and game-based training/rehabilitation tools. The PrimeSense or Kinect sensors, along with open source middleware, provide markerless full-body tracking on a conventional PC using a single plug and play USB sensor. This technology provides a fully articulated skeleton that digitizes the user’s body pose and directly quantizes their movements in real time without encumbering the user with tracking devices or markers. We have explored the integration of the depth sensing technology and middleware within three applications: 1) virtual environments, 2) gesture controlled PC games, 3) a game developed to target specific movements for rehabilitation. The benefits of implementing this technology in these three areas demonstrate the potential to provide needed applications for modern-day warfighters.