Interactive Panoramic Ray Tracing for Mixed 360° RGBD Videos

This paper introduces an interactive panoramic ray tracing method for rendering real-time photo-realistic illumination and shadow effects when inserting virtual objects into 360° RGBD videos. First, we approximate the geometry of a real scene with a panoramic depth buffer and a screen space depth buffer. Then, a sparse sampling ray generation method is proposed to accelerate the tracing process by reducing the number of rays that need to be emitted in ray tracing. After this, an irradiance estimation pass is introduced to generate a noisy Monte-Carlo image. At last, the final result is smoothed by interpolation, spatial-temporal filtering, and differential rendering. We tested our method in some natural and synthetic scenes and compared the results of our approach with the ground truth and that of the Image-Based Lighting method. The results show that our method can generate visually photo-realistic frames for virtual objects in 360° RGBD videos in real-time, making the rendering results more natural and credible.