Automation of Recording in Smart Classrooms via Deep Learning and Bayesian Maximum a Posteriori Estimation of Instructor's Pose

Internet of Things is making objects smarter and more autonomous. At the other side, online education is gaining momentum and many universities are now offering online degrees. Content preparation for such programs usually involves recording the classes. In this article, we intend to introduce a deep learning-based camera management system as a substitute for the academic filming crew. The solution mainly consists of two cameras and a wearable gadget for the instructor. The fixed camera is used for the instructor's position and pose detection and the pan-tilt-zoom (PTZ) camera does the filming. In the proposed solution, image processing and deep learning techniques are merged together. Face recognition and skeleton detection algorithms are used to detect the position of instructor. But the main contribution lies in the application of deep learning for instructor's skeleton detection and postprocessing of the deep network output for correction of the pose detection results using a Bayesian Maximum A Posteriori (MAP) estimator. This estimator is defined on a Markov state machine. The pose detection result along with the position info is then used by the PTZ camera controller for filming purposes. The proposed solution is implemented by using OpenPose which is a convolutional neural network for detection of body parts. Feeding a neural network pose classifier with 12 features extracted from the output of the deep network yields an accuracy of 89%. However, as we show, the accuracy can be improved by the Markov model and MAP estimator to reach as high as 95.5%.