mPose: Environment- and subject-agnostic 3D skeleton posture reconstruction leveraging a single mmWave device

This paper is an invited paper of IEEE/ACM Conference on Connected Health Applications, Systems, and Engineering Technologies (IEEE/ACM CHASE 2021), and was presented in the conference.

Abstract: Human skeleton posture reconstruction is an essential component for human–computer interactions (HCI) in various application domains. Traditional approaches usually rely on either cameras or on-body sensors, which induce privacy concerns or inconvenient practical setups. To address these practical concerns, this paper proposes a low-cost contactless skeleton posture reconstruction system, mPose, which can reconstruct a user’s 3D skeleton postures using a single mmWave device. mPose does not require the user to wear any sensors and can enable a broad range of emerging mobile applications (e.g., VR gaming and pervasive user input) via mmWave-5G ready Internet of Things (IoT) devices. Particularly, the system extracts multi-dimensional spatial information from mmWave signals which characterizes the skeleton postures in a 3D space. To mitigate the impacts of environmental changes, mPose dynamically detects the user location and extracts spatial features from the mmWave signals reflected only from the user. Furthermore, we develop a deep regression method with a domain discriminator to learn a mapping between the spatial features and the joint coordinates of human body while removing subject-specific characteristics, realizing robust posture reconstruction across users. Extensive experiments, involving 17 representative body postures, 7 subjects, and 3 indoor environments, show that mPose outperforms contemporary state-of-the-art RF-based solutions with a lower average joint error of only ~30 mm, while achieving transferability across environments and subjects at the same time.

View the full paper