RISE-Editing: Rotation-invariant neural point fields with interactive segmentation for fine-grained and efficient editing

Neural Radiance Fields (NeRF) have shown great potential for synthesizing novel views. Currently, despite the existence of some initial controllable and editable NeRF methods, they remain limited in terms of efficient and fine-grained editing capabilities, hinders the creative editing abilities and potential applications for NeRF. In this paper, we present the rotation-invariant neural point fields with interactive segmentation for fine-grained and efficient editing. Editing the implicit field presents a significant challenge, as varying the orientation of the corresponding explicit scaffold—whether point, mesh, volume, or other representations—may lead to a notable decline in rendering quality. By leveraging the complementary strengths of implicit NeRF-based representations and explicit point-based representations, we introduce a novel rotation-invariant neural point field representation. This representation enables the learning of local contents using Cartesian coordinates, leading to significant improvements in scene rendering quality after fine-grained editing. To achieve this rotation-invariant representation, we carefully design a Rotation-Invariant Neural Inverse Distance Weighting Interpolation (RNIDWI) module to aggregate the neural points. To enable more efficient and flexible cross-scene compositing, we disentangle the traditional NeRF representation into two components: a scene-agnostic rendering module and the scene-specific neural point fields. Furthermore, we present a multi-view ensemble learning strategy to lift the 2D inconsistent zero-shot segmentation results to 3D neural points field in real-time without post retraining. With simple click-based prompts on 2D images, user can efficiently segment the 3D neural point field and manipulate the corresponding neural points, enabling fine-grained editing of the implicit fields. Extensive experimental results demonstrate that our method offers enhanced editing capabilities and simplified editing process for users, delivers photorealistic rendering quality for novel views, and surpasses related methods in terms of the space–time efficiency and the types of editing functions they can achieve. The code is available at https://github.com/yuzewang1998/RISE-Editing.