A Hand Air-Writing System Using MIMO Radar and Deep Learning

Recently, radar-based Hand Gesture Recognition (HGR) has gained increased attention in several applications involving contactless Human-Computer Interaction (HCI). Air-writing, as a definite HGR, requires real-time target positioning and trajectory tracking, followed by alphanumerical recognition via deep learning. Benefiting from the large virtual array provided by Multiple Input Multiple Output (MIMO) radar, this chapter presents an interferometry-based processing approach to acquire the subtle range and azimuth displacements of finger motions, thus fulfilling the tracking of the alphanumerical trajectory. A ResNet50 Convolutional Neural Network (CNN) trained with the trajectory is used to recognise the correct writings. Additionally, spatial interferometry is exploited to identify multiple strokes when writing specific characters, punctuations or words requiring non-contiguous hand motion articulation. This is achieved by utilizing the subtle elevation change induced by hand lifts among different strokes. Experimental results show that the proposed air-writing system performs rather well in sensing and tracking hand movements, achieving high recognition accuracy for different types of air writings.