MS-MH-TCN: Multi-Stage and Multi-Head Temporal Convolutional Network for Action Segmentation

The segmentation of manual action in long video has important application value in the fields of demonstration programming and human-computer interaction. The state-of-the-art manual action segmentation method uses multi-stage temporal convolution. Although it can capture the long temporal dependence between actions, there are still excessive segmentation errors in the predicted results. In this paper, we propose a multistage and multi-head temporal convolutional network (MS-MH-TCN) for improving the performance on the action segmentation task. A multi-head calculation is performed at each stage, and the results of the multi-head are pooled to average and fed into the next stage. This approach improves the model’s prediction and generalization of input information, because different heads can focus on different aspects of the input and can adapt to learn how to combine them to generate final action predictions. We also propose a new segment smoothing loss function to punish over-segmentation errors. An extensive evaluation showed the effectiveness of the proposed model in capturing long-term dependencies and identifying action segments. Our model achieved the most advanced results on the 50salad dataset.