Terrain-Aware Stride-Level Human Hip Joint Angle Forecasting via Vision and Kinematics Fusion

Forecasting human joint angle has attracted considerable attention in the field of exoskeletons and prostheses, as one promising solution for human intention understanding. Multi-modal information fusion approaches have been employed to achieve terrain adaptability, among which vision encompassing terrain images possesses cross-subject invariance and convenience, showing great potential for practical applications. However, the real-time performance and accuracy of existing methods still need to be improved. In this letter, we share a real-world dataset focusing on the hip joints of 10 healthy subjects walking through level ground, stairs and ramps with stride-level label. We design a network called Sandwich Fusion Transformer for Image and Kinematics (SFTIK), which predicts the thigh angle of the ensuing stride given the terrain images at the beginning of the preceding and the ensuing stride and the IMU time series during the preceding stride. We introduce width-level patchify, tailored for egocentric terrain images, to reduce the computational demands. We demonstrate the proposed sandwich input and fusion mechanism could significantly improve the forecasting performance. Overall, the SFTIK outperforms baseline methods, achieving a computational efficiency of 3.31 G Flops, and root mean square error (RMSE) of 3.445 ± 0.804° and Pearson’s correlation coefficient (PCC) of 0.971 ± 0.025. The results demonstrate that SFTIK could forecast the thigh’s angle accurately with low computational cost, which could serve as a terrain adaptive trajectory planning method for hip exoskeletons.