volume | PIER Journals

Abstract

Radar-based hand gestures recognition have played an important role in developing human-computer interaction (HCI). However, when radar-based hand gesture recognition techniques are applied to multi-target scenarios, the challenges mainly involve problems of mutual interference and inaccurate recognition of hand motion when both hands move within the same plane. Here, we propose a dual-hand trajectory perception prototype based on a 60 GHz frequency-modulated continuous-wave (FMCW) multiple-input multiple-output (MIMO) radar sensor platform with an L-shaped virtual antenna array. To address the challenges, the approach involves estimating azimuth and elevation angles separately from the two data components derived from the L-shaped array through multiple signal classification (MUSIC) algorithm, incorporating spatial division techniques combined with digital beam formation (DBF). The dual-hand applications mainly include angle targets at two distinct distances or dual-angle targets at the same distance. Therefore, the target distances are first determined using range fast fourier transform (range FFT). If a single target distance is identified, we proceed to solve for the angles of two targets. Alternatively, if two distinct target distances are distinguished, we individually solve for the single-angle target corresponding to each distance. Furthermore, to mitigate noise inherent in the raw data of visualization, a frame point removal and smoothing algorithm is devised to refine the trajectories. Experimental verifications prove that the proposed multi-target motion perception algorithm by using a MIMO FMCW radar sensor platform can realize accurate recognition of air-writing gestures and enable tracking the trajectories of both single-handed and dual-handed targets in three-dimensional space. It also gives a new option for controlling the HCI.

1. Li, Guanglin, Aimee E. Schultz, and Todd A. Kuiken, "Quantifying pattern recognition --- Based myoelectric control of multifunctional transradial prostheses," IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 18, No. 2, 185-192, 2010.
doi:10.1109/tnsre.2009.2039619

2. Li, Yunan, Qiguang Miao, Kuan Tian, Yingying Fan, Xin Xu, Rui Li, and Jianfeng Song, "Large-scale gesture recognition with a fusion of RGB-D data based on the C3D model," 2016 23rd International Conference on Pattern Recognition (ICPR), 25-30, Cancun, Mexico, 2016.
doi:10.1109/icpr.2016.7899602

3. Almasre, Miada A. and Hana Al-Nuaim, "Recognizing arabic sign language gestures using depth sensors and a KSVM classifier," 2016 8th Computer Science and Electronic Engineering (CEEC), 146-151, Colchester, UK, 2016.
doi:10.1109/ceec.2016.7835904

4. Farina, Dario, Ning Jiang, Hubertus Rehbaum, Aleš Holobar, Bernhard Graimann, Hans Dietl, and Oskar C. Aszmann, "The extraction of neural information from the surface EMG for the control of upper-limb prostheses: Emerging avenues and challenges," IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 22, No. 4, 797-809, 2014.
doi:10.1109/tnsre.2014.2305111

5. Amsüss, Sebastian, Peter M. Goebel, Ning Jiang, Bernhard Graimann, Liliana Paredes, and Dario Farina, "Self-correcting pattern recognition system of surface EMG signals for upper limb prosthesis control," IEEE Transactions on Biomedical Engineering, Vol. 61, No. 4, 1167-1176, 2014.
doi:10.1109/tbme.2013.2296274

6. Zhang, Xuebo, Xiang Chen, Farsam Farzadpour, and Yongchun Fang, "A visual distance approach for multicamera deployment with coverage optimization," IEEE/ASME Transactions on Mechatronics, Vol. 23, No. 3, 1007-1018, 2018.
doi:10.1109/tmech.2018.2834393

7. Song, Zhan, Ronald Chung, and Xiao-Ting Zhang, "An accurate and robust strip-edge-based structured light means for shiny surface micromeasurement in 3-D," IEEE Transactions on Industrial Electronics, Vol. 60, No. 3, 1023-1032, 2013.
doi:10.1109/tie.2012.2188875

8. May, Stefan, Bjorn Werner, Hartmut Surmann, and Kai Pervolz, "3D time-of-flight cameras for mobile robotics," 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, 790-795, Beijing, China, 2006.
doi:10.1109/iros.2006.281670

9. Fhager, Lars Ohlsson, Sebastian Heunisch, Hannes Dahlberg, Anton Evertsson, and Lars-Erik Wernersson, "Pulsed millimeter wave radar for hand gesture sensing and classification," IEEE Sensors Letters, Vol. 3, No. 12, 1-4, 2019.
doi:10.1109/lsens.2019.2953022

10. Heunisch, Sebastian, Lars Ohlsson Fhager, and Lars-Erik Wernersson, "Millimeter-wave pulse radar scattering measurements on the human hand," IEEE Antennas and Wireless Propagation Letters, Vol. 18, No. 7, 1377-1380, 2019.
doi:10.1109/lawp.2019.2917081

11. Wang, Fu-Kang, Mu-Cyun Tang, Yen-Chen Chiu, and Tzyy-Sheng Horng, "Gesture sensing using retransmitted wireless communication signals based on Doppler radar technology," IEEE Transactions on Microwave Theory and Techniques, Vol. 63, No. 12, 4592-4602, 2015.
doi:10.1109/tmtt.2015.2495298

12. Fan, Tenglong, Chao Ma, Zhitao Gu, Qinyi Lv, Jialong Chen, Dexin Ye, Jiangtao Huangfu, Yongzhi Sun, Changzhi Li, and Lixin Ran, "Wireless hand gesture recognition based on continuous-wave Doppler radar sensors," IEEE Transactions on Microwave Theory and Techniques, Vol. 64, No. 11, 4012-4020, 2016.
doi:10.1109/tmtt.2016.2610427

13. Sun, Yuliang, Tai Fei, Xibo Li, Alexander Warnecke, Ernst Warsitz, and Nils Pohl, "Real-time radar-based gesture detection and recognition built in an edge-computing platform," IEEE Sensors Journal, Vol. 20, No. 18, 10706-10716, 2020.
doi:10.1109/jsen.2020.2994292

14. Zhang, Yi, Chengkai Zhu, Shuqin Dong, Zhitao Gu, Marcel Balle, Bin Zhang, Changzhi Li, and Lixin Ran, "3-D motion imaging in a multipath coordinate space based on a TDM-MIMO radar sensor," IEEE Transactions on Microwave Theory and Techniques, Vol. 68, No. 11, 4642-4651, 2020.
doi:10.1109/tmtt.2020.3025563

15. Amin, Moeness G., Zhengxin Zeng, and Tao Shan, "Hand gesture recognition based on radar micro-Doppler signature envelopes," 2019 IEEE Radar Conference (RadarConf), 1-6, Boston, MA, USA, 2019.
doi:10.1109/radar.2019.8835661

16. Pramudita, Aloysius Adya, et al., "Contactless hand gesture sensor based on array of CW radar for human to machine interface," IEEE Sensors Journal, Vol. 21, No. 13, 15196-15208, 2021.
doi:10.1109/jsen.2021.3073263

17. Lien, Jaime, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev, "Soli: Ubiquitous gesture sensing with millimeter wave radar," ACM Transactions on Graphics (TOG), Vol. 35, No. 4, 1-19, 2016.
doi:10.1145/2897824.2925953

18. Wang, Pengcheng, Junyang Lin, Fuyue Wang, Jianping Xiu, Yue Lin, Na Yan, and Hongtao Xu, "A gesture air-writing tracking method that uses 24 GHz SIMO radar SoC," IEEE Access, Vol. 8, 152728-152741, 2020.
doi:10.1109/access.2020.3017869

19. Gu, Changzhan, Jian Wang, and Jaime Lien, "Motion sensing using radar: Gesture interaction and beyond," IEEE Microwave Magazine, Vol. 20, No. 8, 44-57, 2019.
doi:10.1109/mmm.2019.2915490

20. Li, Yuchen, Changzhan Gu, and Junfa Mao, "4-D gesture sensing using reconfigurable virtual array based on a 60-GHz FMCW MIMO radar sensor," IEEE Transactions on Microwave Theory and Techniques, Vol. 70, No. 7, 3652-3665, 2022.
doi:10.1109/tmtt.2022.3174075

21. Wang, Zhu, Zhiwen Yu, Xinye Lou, Bin Guo, and Liming Chen, "Gesture-radar: A dual Doppler radar based system for robust recognition and quantitative profiling of human gestures," IEEE Transactions on Human-Machine Systems, Vol. 51, No. 1, 32-43, 2021.
doi:10.1109/thms.2020.3036637

22. Leem, Seong Kyu, Faheem Khan, and Sung Ho Cho, "Detecting mid-air gestures for digit writing with radio sensors and a CNN," IEEE Transactions on Instrumentation and Measurement, Vol. 69, No. 4, 1066-1081, 2020.
doi:10.1109/tim.2019.2909249

23. Khan, Faheem, Seong Kyu Leem, and Sung Ho Cho, "In-air continuous writing using UWB impulse radar sensors," IEEE Access, Vol. 8, 99302-99311, 2020.
doi:10.1109/access.2020.2994281

24. Wang, Zetao, Gang Li, and Le Yang, "Dynamic hand gesture recognition based on micro-Doppler radar signatures using hidden Gauss-Markov models," IEEE Geoscience and Remote Sensing Letters, Vol. 18, No. 2, 291-295, 2021.
doi:10.1109/lgrs.2020.2974821

25. Malysa, Greg, Dan Wang, Lorin Netsch, and Murtaza Ali, "Hidden Markov model-based gesture recognition with FMCW radar," 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP), 1017-1021, Washington, DC, USA, 2016.
doi:10.1109/globalsip.2016.7905995

26. Sakamoto, Takuya, Xiaomeng Gao, Ehsan Yavari, Ashikur Rahman, Olga Boric-Lubecke, and Victor M. Lubecke, "Hand gesture recognition using a radar echo I-Q plot and a convolutional neural network," IEEE Sensors Letters, Vol. 2, No. 3, 1-4, 2018.
doi:10.1109/LSENS.2018.2866371

27. Sun, Yuliang, Tai Fei, Shangyin Gao, and Nils Pohl, "Automatic radar-based gesture detection and classification via a region-based deep convolutional neural network," ICASSP 2019 --- 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4300-4304, Brighton, UK, 2019.
doi:10.1109/icassp.2019.8682277

28. Skaria, Sruthy, Akram Al-Hourani, Margaret Lech, and Robin J. Evans, "Hand-gesture recognition using two-antenna Doppler radar with deep convolutional neural networks," IEEE Sensors Journal, Vol. 19, No. 8, 3041-3048, 2019.
doi:10.1109/jsen.2019.2892073

29. Zhang, Zhenyuan, Zengshan Tian, and Mu Zhou, "Latern: Dynamic continuous hand gesture recognition using FMCW radar sensor," IEEE Sensors Journal, Vol. 18, No. 8, 3278-3289, 2018.
doi:10.1109/jsen.2018.2808688

30. Arsalan, Muhammad and Avik Santra, "Character recognition in air-writing based on network of radars for human-machine interface," IEEE Sensors Journal, Vol. 19, No. 19, 8855-8864, 2019.
doi:10.1109/jsen.2019.2922395

31. Du, Hao, Yuan He, and Tian Jin, "Transfer learning for human activities classification using micro-Doppler spectrograms," 2018 IEEE International Conference on Computational Electromagnetics (ICCEM), 1-3, Chengdu, China, 2018.
doi:10.1109/compem.2018.8496654

32. Yin, Wei, Ling-Feng Shi, and Yifan Shi, "Indoor human action recognition based on millimeter-wave radar micro-Doppler signature," Measurement, Vol. 235, 114939, 2024.
doi:10.1016/j.measurement.2024.114939

33. Jin, Biao, Yu Peng, Xiaofei Kuang, Zhenkai Zhang, Zhuxian Lian, and Biao Wang, "Robust dynamic hand gesture recognition based on millimeter wave radar using Atten-TsNN," IEEE Sensors Journal, Vol. 22, No. 11, 10861-10869, 2022.
doi:10.1109/jsen.2022.3170311

Dr. Haipeng Wang

Dr. Zhongfang Ren

Dr. Wei Pan

Dr. Zheng Xiao

Prof. Yunbo Li