Panoramic Ray Tracing for Interactive Mixed Reality Rendering Based on 360° RGBD Video

This article presents an interactive panoramic ray tracing method for rendering real-time realistic lighting and shadow effects when virtual objects are inserted in 360° RGBD videos. First, we approximate the geometry of the real scene. We propose a sparse sampling ray generation method to speed up the tracing process by reducing the number of rays that need to be emitted in ray tracing. After that, an irradiance estimation channel is introduced to generate noisy Monte Carlo images. Finally, the final result is smoothed and synthesized by interpolation, temporal filtering, and differential rendering. We tested our method in a number of natural and synthesized scenes and compared our method with results from ground truth and image-based illumination methods. The results show that our method can generate visually realistic frames for dynamic virtual objects in 360° RGBD videos in real time, making the rendering results more natural and believable.