MagiaPose: Pose Recognition and Quality Assessment with ViViT

To address the scarcity of objective feedback in training complex somatic skills, such as martial arts, we propose MagiaPose, a video-based analysis framework designed for automated pose recognition and action quality assessment. The framework adopts a multi-task learning paradigm, utilizing a video vision encoder backbone to capture spatiotemporal dependencies from cropped human regions and keyframe features. We evaluate the framework on a martial arts dataset comprising 15 distinct action categories. In our experiments, we compare the performance of fine-tuning a pre-trained ViViT against training lightweight variants (including Attention-based, Mamba-based, and hybrid architectures) from scratch using high-quality, pre-extracted features. Empirical results demonstrate that the Mamba-based variant achieves accuracy nearly comparable to the standard ViViT while significantly reducing model size and inference latency. Notably, the Mamba block exhibits superior accuracy compared to the Attention-based baseline.