Deep Manifold Structure Transfer for Action Recognition

While intrinsic data structure in subspace provides useful information for visual recognition, it has not yet been well studied in deep feature learning for action recognition. In this paper, we introduce a new spatio-temporal manifold network (STMN) that leverages data manifold structures to regularize deep action feature learning, aiming at simultaneously minimizing the intra-class variations of learned deep features and alleviating the over-fitting problem. To this end, the manifold prior is imposed from the top layer of a convolutional neural network (CNN) and propagated across convolutional layers during forward-backward propagation. The observed correspondence of manifold structures in the data space and feature space validates that the manifold priori can be transferred across the CNN layers. The STMN theoretically recasts the problem of transferring the data structure prior into the deep learning architectures as a projection over the manifold via an embedding method, which can be easily solved by an alternating direction method of multipliers and backward propagation (ADMM-BP) algorithm. The STMN is generic in the sense that it can be plugged into various backbone architectures to learn more discriminative representation for action recognition. The extensive experimental results show that our method achieves comparable or even better performance compared with the state-of-the-art approaches on four benchmark datasets.