volume | PIER Journals

Abstract

Millimeter-wave (mmWave) radar supports non-contact action sensing without optical imagery, but its sparse point clouds vary with multipath, aspect angle, and detection quality. We address within-protocol recognition with a radar-informed Multi-Stream Gated Transformer (MSG-Transformer). Separate streams encode target-centered geometry, Doppler velocity, radar-reported SNR, and tracker-estimated horizontal-centroid context prior to temporal fusion. Raw trials are split before frame stacking, and augmentation perturbs only centered coordinates, leaving measured Doppler and SNR unchanged. On 820 archived trials from eight anonymized participant identifiers and nine action categories, MSG-Transformer achieved 97.67% development-set macro recall with 1.03 million parameters and a 3.92 MiB FP32 footprint. This single-split result is not subject-independent or cross-environment evidence.

1. Zhang, Jia, Rui Xi, Yuan He, Yimiao Sun, Xiuzhen Guo, Weiguo Wang, Xin Na, Yunhao Liu, Zhenguo Shi, and Tao Gu, "A survey of mmWave-based human sensing: Technology, platforms and applications," IEEE Communications Surveys & Tutorials, Vol. 25, No. 4, 2052-2087, 2023.
doi:10.1109/comst.2023.3298300 Google Scholar

2. Yu, Chengxi, Zhezhuang Xu, Kun Yan, Ying-Ren Chien, Shih-Hau Fang, and Hsiao-Chun Wu, "Noninvasive human activity recognition using millimeter-wave radar," IEEE Systems Journal, Vol. 16, No. 2, 3036-3047, 2022.
doi:10.1109/jsyst.2022.3140546 Google Scholar

3. Li, Jiapeng, Xiaodan Shao, Feng Chen, Shaohua Wan, Chang Liu, Zhiqiang Wei, and Derrick Wing Kwan Ng, "Networked integrated sensing and communications for 6G wireless systems," IEEE Internet of Things Journal, Vol. 11, No. 17, 29062-29075, 2024.
doi:10.1109/jiot.2024.3406598 Google Scholar

4. Aldirmaz-Colak, Sultan, Mustafa Namdar, Arif Basgumus, Serdar Özyurt, Selman Kulac, Nurullah Calik, Mehmet Akif Yazici, Ahmet Serbes, and Lutfiye Durak-Ata, "A comprehensive review on ISAC for 6G: Enabling technologies, security, and AI/ML perspectives," IEEE Access, Vol. 13, 97152-97193, 2025.
doi:10.1109/access.2025.3573371 Google Scholar

5. Ben-Shabat, Yizhak, Oren Shrout, and Stephen Gould, "3DInAction: Understanding human actions in 3D point clouds," 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 19978-19987, Seattle, WA, USA, 2024.
doi:10.1109/CVPR52733.2024.01888

6. Yin, Wei, Ling-Feng Shi, and Yifan Shi, "Indoor human action recognition based on millimeter-wave radar micro-Doppler signature," Measurement, Vol. 235, 114939, 2024.
doi:10.1016/j.measurement.2024.114939 Google Scholar

7. Yu, Zheqi, Adnan Zahid, William Taylor, Adnan Zahid, Khalid Rajab, Hadi Heidari, Muhammad A. Imran, and Qammer H. Abbasi, "A radar-based human activity recognition using a novel 3-D point cloud classifier," IEEE Sensors Journal, Vol. 22, No. 19, 18218-18227, 2022.
doi:10.1109/jsen.2022.3198395 Google Scholar

8. Feng, Ruoyu, Eddy De Greef, Maxim Rykunov, Hichem Sahli, Sofie Pollin, and André Bourdoux, "Multipath ghost recognition for indoor MIMO radar," IEEE Transactions on Geoscience and Remote Sensing, Vol. 60, 1-10, 2022.
doi:10.1109/tgrs.2021.3109381 Google Scholar

9. Wei, Yimin, Hao Liu, Tingting Xie, Qiuhong Ke, and Yulan Guo, "Spatial-temporal transformer for 3D point cloud sequences," 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 657-666, Waikoloa, HI, USA, 2022.
doi:10.1109/WACV51458.2022.00073

10. Zhong, Jiandan, Hongyu Jiang, Yulin Ji, Yingxiang Li, and Yajuan Xue, "CSP-Former: A transformer-based network for point cloud analysis with compressed sensing and spatial self-attention," Electronics, Vol. 14, No. 2, 347, 2025.
doi:10.3390/electronics14020347 Google Scholar

11. Fan, Hehe, Yi Yang, and Mohan Kankanhalli, "Point 4D transformer networks for spatio-temporal modeling in point cloud videos," 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14199-14208, Nashville, TN, USA, 2021.
doi:10.1109/CVPR46437.2021.01398

12. Zeller, Matthias, Vardeep S. Sandhu, Benedikt Mersch, Jens Behley, Michael Heidingsfeld, and Cyrill Stachniss, "Radar velocity transformer: Single-scan moving object segmentation in noisy radar point clouds," 2023 IEEE International Conference on Robotics and Automation (ICRA), 7054-7061, London, United Kingdom, 2023.
doi:10.1109/ICRA48891.2023.10161152

13. Ouaknine, Arthur, Alasdair Newson, Patrick Pérez, Florence Tupin, and Julien Rebut, "Multi-view radar semantic segmentation," 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 15651-15660, Montreal, QC, Canada, 2021.
doi:10.1109/ICCV48922.2021.01538

14. Paek, Dong-Hee, Seung-Hyun Kong, and Kevin Tirta Wijaya, "K-radar: 4D radar object detection for autonomous driving in various weather conditions," Advances in Neural Information Processing Systems, Vol. 35, 2022.
doi:10.52202/068431-0276

15. Sheeny, Marcel, Emanuele De Pellegrin, Saptarshi Mukherjee, Alireza Ahrabian, Sen Wang, and Andrew Wallace, "RADIATE: A radar dataset for automotive perception in bad weather," 2021 IEEE International Conference on Robotics and Automation (ICRA), 1-7, Xi'an, China, 2021.
doi:10.1109/ICRA48506.2021.9562089

16. Chen, Zhan, Sicheng Li, Bing Yang, Qinghan Li, and Hong Liu, "Multi-scale spatial temporal graph convolutional network for skeleton-based action recognition," Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, No. 2, 1113-1122, 2021.
doi:10.1609/aaai.v35i2.16197

17. Shazeer, Noam, "Glu variants improve transformer," arXiv preprint arXiv:2002.05202, 2020.
doi:10.48550/arXiv.2002.05202 Google Scholar

18. Liu, Ze, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo, "Swin Transformer V2: Scaling up capacity and resolution," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11999-12009, New Orleans, LA, USA, 2022.
doi:10.1109/CVPR52688.2022.01170

19. Yang, Jianfei, He Huang, Yunjiao Zhou, Xinyan Chen, Yuecong Xu, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu, and Lihua Xie, "MM-Fi: Multi-modal non-intrusive 4D human dataset for versatile wireless sensing," Advances in Neural Information Processing Systems, Vol. 36, 2023.
doi:10.52202/075280-0822

20. Lu, Dening, Qian Xie, Mingqiang Wei, Kyle Gao, Linlin Xu, and Jonathan Li, "Transformers in 3D point clouds: A survey," arXiv preprint arXiv:2205.07417, 2022.
doi:10.48550/arXiv.2205.07417 Google Scholar

21. Zhang, Huang, Changshuo Wang, Shengwei Tian, Baoli Lu, Liping Zhang, Xin Ning, and Xiao Bai, "Deep learning-based 3D point cloud classification: A systematic survey and outlook," Displays, Vol. 79, 102456, Sep. 2023.
doi:10.1016/j.displa.2023.102456 Google Scholar

22. Qi, Charles R., Hao Su, Kaichun Mo, and Leonidas J. Guibas, "PointNet: Deep learning on point sets for 3D classification and segmentation," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 77-85, Honolulu, HI, USA, 2017.
doi:10.1109/CVPR.2017.16

23. Qi, Charles Ruizhongtai, Li Yi, Hao Su, and L. Guibas, "PointNet++: Deep hierarchical feature learning on point sets in a metric space," Advances in Neural Information Processing Systems, Vol. 30, 2017.

24. Wang, Yue, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon, "Dynamic graph CNN for learning on point clouds," ACM Transactions on Graphics (TOG), Vol. 38, No. 5, 1-12, 2019.
doi:10.1145/3326362 Google Scholar

25. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, "Attention is all you need," Advances in Neural Information Processing Systems, Vol. 30, 2017.

26. Zhao, Hengshuang, Li Jiang, Jiaya Jia, Philip H. S. Torr, and Vladlen Koltun, "Point transformer," Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 16259-16268, 2021.

27. Guo, Meng-Hao, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R. Martin, and Shi-Min Hu, "PCT: Point cloud transformer," Computational Visual Media, Vol. 7, No. 2, 187-199, 2021.
doi:10.1007/s41095-021-0229-5 Google Scholar

28. Wu, Xiaoyang, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao, "Point Transformer V2: Grouped vector attention and partition-based pooling," Advances in Neural Information Processing Systems, Vol. 35, 2022.
doi:10.52202/068431-2415

29. Gao, Xiangyu, Guanbin Xing, Sumit Roy, and Hui Liu, "RAMP-CNN: A novel neural network for enhanced automotive radar object recognition," IEEE Sensors Journal, Vol. 21, No. 4, 5119-5132, Feb. 2021.
doi:10.1109/jsen.2020.3036047 Google Scholar

30. Texas Instruments "mmWave SDK and radar toolbox output data formats: Detected-point side-information and tracker output field definitions," https://software-dl.ti.com/ra-processors/esd/MMWAVE-L-SDK/05_04_00_01/exports/api_guide_xwrL64xx/MMWAVE_DEMO.html, 2024.

31. Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun, "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling," arXiv preprint arXiv:1803.01271, 2018.
doi:10.48550/arXiv.1803.01271 Google Scholar

32. Guo, Jianyuan, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu, "CMT: Convolutional neural networks meet vision transformers," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12165-12175, New Orleans, LA, USA, 2022.
doi:10.1109/CVPR52688.2022.01186

33. Szegedy, Christian, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna, "Rethinking the inception architecture for computer vision," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2818-2826, Las Vegas, NV, USA, 2016.
doi:10.1109/CVPR.2016.308

Dr. Su Liu

Dr. Jianguo Liu

Dr. Fei Gao

Dr. Jietao Cheng

Professor Jun Tang