Progressive Self-supervised Spatio-temporal Feature Learning Based on Video Sequence Saliency

We observe that videos have different levels of frame/clip sequence saliency, and the effective utilization of frame/clip sequence saliency is beneficial to spatio-temporal feature learning. Therefore, we propose a new concept called video sequence saliency (VSS) to measure the degree of difficulty of models in identifying the correct frames/clip orders of videos. Accordingly, we developed a novel method named progressive self-supervised spatio-temporal feature learning based on VSS (PSSFL-VSS). For the pretext task of clip order prediction, the videos are input into networks in descending order by VSS values rather than randomly, as in traditional methods. In addition, we update the VSS value of each video based on clip order prediction results. The effectiveness of our pre-trained models is verified by carrying out the downstream tasks of clip/video retrieval and action recognition, and experimental results show that our method achieves apparent improvements over the state-of-the-art methods.