Mirror world: creating digital twins of the space and persons from video streamings

Creating digital twins of scenes has been widely studied in smart city applications. The development of 3D virtual games attracts many researchers. However, most products have high costs, neglect 3D recovery of the moving people, and lack structured analysis. In this paper, we present a mirror world based on PTZ cameras, which creates digital twins of the space and persons from video streamings, including real-time video registration for PTZ cameras and 3D recovery of the human pose and shape. Our goal is to quickly build a digital space to represent and visualize the physical world, leveraging the two tasks to improve the performance of structured scene understanding. We use the scene images from PTZ cameras to create the 3D scene model and propose an image edge alignment method to optimize the texture mismatching during real-time video registration. Then we propose a human analysis network for 3D recovery of the human pose and shape and add refinement to improve performance on two datasets. We place the 3D human bodies in the right position in the mirror world and utilize joint data from specific scenarios to drive the optimization of the system. Consequently, the mirror world provides 3D visualization and structured analysis of the scene with low cost, thereby enhancing users’ spatial understanding.