volume | PIER Journals

Abstract

Unimodal remote-sensing data is susceptible to interference such as illumination changes and texture loss. Visible and infrared images have respective limitations in object detection, which multimodal fusion can address. To address these challenges, this study proposes a Triple-stage Mamba Cross-modal Fusion Network (TMCFNet) for cross-modal remote-sensing object detection. The model performs cross-modal image fusion at the input stage to preserve each modality's inherent information while keeping computational cost low. It consists of three key components: the Adaptive Multi-scale Convolution Module (AMC), Cross-Modal Shared Mamba Interaction Module (SMIM), and Dual-Global Pooling Enhanced Multimodal Feature Fusion Module (G2MFF). They are responsible for unimodal feature enhancement, cross-modal information alignment, and in-depth feature fusion, which effectively improve the model's detection accuracy and generalization ability in complex scenarios. To comprehensively evaluate the model performance, we conducted experiments on multiple public real-world multimodal datasets and dedicated remote sensing datasets. Compared with state-of-the-art unimodal and multimodal detection methods, the proposed model achieved mean average precision mAP@0.5:0.95 values of 58.5% and 69.0% on the DroneVehicle and LLVIP datasets, respectively. Results demonstrate that deep multimodal feature fusion can remarkably boost object detection performance in complex environments.

1. Yang, Jian, Dawei Song, Qiurong Yan, Kang Liu, and Rui Zhang, "Perceptual gating mamba network for salient object detection in remote sensing observation," IEEE Internet of Things Journal, Vol. 13, No. 15, 35046-35059, August 2026.
doi:10.1109/jiot.2026.3695867 Google Scholar

2. Gui, Shengxi, Shuang Song, Rongjun Qin, and Yang Tang, "Remote sensing object detection in the deep learning era --- A review," Remote Sensing, Vol. 16, No. 2, 327, January 2024.
doi:10.3390/rs16020327 Google Scholar

3. Sun, Zhongzhen, Xiangguang Leng, Xianghui Zhang, Zheng Zhou, Boli Xiong, Kefeng Ji, and Gangyao Kuang, "Arbitrary-direction SAR ship detection method for multiscale imbalance," IEEE Transactions on Geoscience and Remote Sensing, Vol. 63, 1-21, 2025.
doi:10.1109/tgrs.2025.3559701 Google Scholar

4. Sun, Shijun, Shuai Ma, Xuyang Feng, Chen Sun, Baolong Ding, Yaoyao Ran, and Yihong Zhang, "LDSDet: Long-range context and dynamic cross-modal alignment for multimodal object detection under challenging illumination," Remote Sensing, Vol. 18, No. 11, 1827, June 2026.
doi:10.3390/rs18111827 Google Scholar

5. Gao, Yaoyue, Xinru Cheng, Yimeng Li, Dawei Xu, Desheng Sun, and Yaoyi Hu, "ISDG-Net: Efficient RGB-infrared object detection for remote sensing imagery," Remote Sensing, Vol. 18, No. 10, 1570, May 2026.
doi:10.3390/rs18101570 Google Scholar

6. Nan, Guozheng, Yue Zhao, Liyong Fu, and Qiaolin Ye, "Object detection by channel and spatial exchange for multimodal remote sensing imagery," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 17, 8581-8593, 2024.
doi:10.1109/jstars.2024.3388013 Google Scholar

7. Zhao, Fei, Chengcui Zhang, and Baocheng Geng, "Deep multimodal data fusion," ACM Computing Surveys, Vol. 56, No. 9, 1-36, April 2024.
doi:10.1145/3649447 Google Scholar

8. Chen, Qiao-Ping, Hongyu Liu, Zejun Sun, Li-Li Wang, and Guang-Hui Zheng, "Inverse random source problem for the Helmholtz equation from statistical phaseless data," Journal of Scientific Computing, Vol. 107, No. 1, 18, March 2026.
doi:10.1007/s10915-026-03233-8 Google Scholar

9. Deng, Li-Ping, Hongyu Liu, Zhi-Qiang Miao, and Guang-Hui Zheng, "Phaseless uniqueness for determining internal source in photo-thermal effect," SIAM Journal on Applied Mathematics, Vol. 85, No. 5, 2318-2336, October 2025.
doi:10.1137/24m1691363 Google Scholar

10. Yin, Weishi, Wenhong Yang, and Hongyu Liu, "A neural network scheme for recovering scattering obstacles with limited phaseless far-field data," Journal of Computational Physics, Vol. 417, 109594, September 2020.
doi:10.1016/j.jcp.2020.109594 Google Scholar

11. Gao, Yu, Hongyu Liu, Xianchao Wang, and Kai Zhang, "On an artificial neural network for inverse scattering problems," Journal of Computational Physics, Vol. 448, 110771, January 2022.
doi:10.1016/j.jcp.2021.110771 Google Scholar

12. Guo, Yingman, Kaijun Yang, Ruyi Feng, and Li Cao, "SA-CNN model reveals opposite seasonal trends and drivers of water quality in dongting lake using multi-source remote sensing," Remote Sensing, Vol. 18, No. 10, 1565, May 2026.
doi:10.3390/rs18101565 Google Scholar

13. Yi, Zhiwei, Lingjia Gu, Ruifei Zhu, Junwei Tian, and He Mi, "GDBNet: A three-branch semantic segmentation network integrating CNN and transformer for land cover classification in ski resorts," Remote Sensing, Vol. 18, No. 10, 1666, May 2026.
doi:10.3390/rs18101666 Google Scholar

14. Dao, T. and A. Gu, "Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality," arXiv:2405.21060, 2024.
doi:10.48550/arXiv.2405.21060 Google Scholar

15. Li, Hui and Xiao-Jun Wu, "CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach," Information Fusion, Vol. 103, 102147, March 2024.
doi:10.1016/j.inffus.2023.102147 Google Scholar

16. Wang, Xiangxiang, Lixing Fang, Junli Zhao, Zhenkuan Pan, Hui Li, and Yi Li, "MMAE: A universal image fusion method via mask attention mechanism," Pattern Recognition, Vol. 158, 111041, February 2025.
doi:10.1016/j.patcog.2024.111041 Google Scholar

17. Yang, Bin, Yuxuan Hu, Xiaowen Liu, and Jing Li, "CEFusion: An infrared and visible image fusion network based on cross-modal multi-granularity information interaction and edge guidance," IEEE Transactions on Intelligent Transportation Systems, Vol. 25, No. 11, 17794-17809, November 2024.
doi:10.1109/tits.2024.3426539 Google Scholar

18. Bao, Wei, Meiyu Huang, Jingjing Hu, and Xueshuang Xiang, "Dual-dynamic cross-modal interaction network for multimodal remote sensing object detection," IEEE Transactions on Geoscience and Remote Sensing, Vol. 63, 1-13, 2025.
doi:10.1109/tgrs.2025.3530085 Google Scholar

19. Wang, Fasheng, Yiming Su, Ruimin Wang, Jing Sun, Fuming Sun, and Haojie Li, "Cross-modal and cross-level attention interaction network for salient object detection," IEEE Transactions on Artificial Intelligence, Vol. 5, No. 6, 2907-2920, 2024.
doi:10.1109/TAI.2023.3333827 Google Scholar

20. Se, Stephen, David Lowe, and Jim Little, "Mobile robot localization and mapping with uncertainty using scale-invariant visual landmarks," The International Journal of Robotics Research, Vol. 21, No. 8, 735-758, August 2002.
doi:10.1177/027836402761412467 Google Scholar

21. Woo, Sanghyun, Jongchan Park, Joon-Young Lee, and In So Kweon, "CBAM: Convolutional block attention module," Computer Vision --- ECCV 2018, 3-19, Springer, Cham, 2018.
doi:10.1007/978-3-030-01234-2_1

22. Chen, Jun, HongSheng Mai, Linbo Luo, Xiaoqiang Chen, and Kangle Wu, "Effective feature fusion network in BIFPN for small object detection," 2021 IEEE International Conference on Image Processing (ICIP), 699-703, Anchorage, AK, USA, September 2021.
doi:10.1109/icip42928.2021.9506347

23. Dong, Wenhao, Haodong Zhu, Shaohui Lin, Xiaoyan Luo, Yunhang Shen, Guodong Guo, and Baochang Zhang, "Fusion-mamba for cross-modality object detection," IEEE Transactions on Multimedia, Vol. 27, 7392-7406, 2025.
doi:10.1109/tmm.2025.3599020 Google Scholar

24. Li, Xue, Guo Zhang, Hao Cui, Shasha Hou, Yujia Chen, Zhijiang Li, Haifeng Li, and Huabin Wang, "Progressive fusion learning: A multimodal joint segmentation framework for building extraction from optical and SAR images," ISPRS Journal of photogrammetry and remote sensing, Vol. 195, 178-191, January 2023.
doi:10.1016/j.isprsjprs.2022.11.015 Google Scholar

25. Ye, Yuanxin, Jiacheng Zhang, Liang Zhou, Jinjin Li, Xiaoyue Ren, and Jianwei Fan, "Optical and SAR image fusion based on complementary feature decomposition and visual saliency features," IEEE Transactions on Geoscience and Remote Sensing, Vol. 62, 1-15, 2024.
doi:10.1109/tgrs.2024.3366519 Google Scholar

26. Jiang, Qianyin, Jianshang Liao, Qiuyu Lin, and Junkang Zhang, "Harnessing foundation models for optical-SAR object detection via gated-guided fusion," ISPRS International Journal of Geo-Information, Vol. 15, No. 4, 160, April 2026.
doi:10.3390/ijgi15040160 Google Scholar

27. Zhang, Xuchong, Han Zhai, Jiaxing Liu, Zhiping Wang, and Hongbin Sun, "Real-time infrared and visible image fusion network using adaptive pixel weighting strategy," Information Fusion, Vol. 99, 101863, November 2023.
doi:10.1016/j.inffus.2023.101863 Google Scholar

28. Dai, Li, Xuefeng Yan, Jiamei Xiong, Shijie Zhang, and Xiangping Zhai, "PMF-YOLO: A multimodal framework for robust object detection via pixel-level and adaptive multi-scale fusion," Neural Computing for Advanced Applications, 220-234, Springer, Singapore, 2025.
doi:10.1007/978-981-95-3736-5_16

29. Shen, Jifeng, Yifei Chen, Yue Liu, Xin Zuo, Heng Fan, and Wankou Yang, "ICAFusion: Iterative cross-attention guided feature fusion for multispectral object detection," Pattern Recognition, Vol. 145, 109913, January 2024.
doi:10.1016/j.patcog.2023.109913 Google Scholar

30. Li, Wenqiang, Aimin Li, Xiaotong Kong, Yuechen Zhang, and Zhiyao Li, "MF-YOLO: Multimodal fusion for remote sensing object detection based on YOLOv5s," 2024 27th International Conference on Computer Supported Cooperative Work in Design (CSCWD), 897-903, Tianjin, China, May 2024.
doi:10.1109/cscwd61410.2024.10580205

31. Chen, Yi-Ting, Jinghao Shi, Zelin Ye, Christoph Mertz, Deva Ramanan, and Shu Kong, "Multimodal object detection via probabilistic ensembling," Computer Vision --- ECCV 2022, 139-158, Springer, Cham, 2022.
doi:10.1007/978-3-031-20077-9_9

32. Wang, Siyu, Xiaogang Yang, Ruitao Lu, Dingwen Zhang, Weiying Xie, Shuang Su, and Zhenyu Zhang, "MROD-YOLO: Multimodal joint representation for small object detection in remote sensing imagery via multiscale iterative aggregation," IEEE Transactions on Geoscience and Remote Sensing, Vol. 63, 1-14, 2025.
doi:10.1109/tgrs.2025.3590447 Google Scholar

33. Yang, Jiyuan, Kun Gao, Baiyang Hu, Zefeng Zhang, Jingyi Wang, Yuqing He, and Yunpeng Feng, "DAAINet: Domain adversarial anti-interference network for bi-temporal change detection," Remote Sensing, Vol. 18, No. 10, 1656, May 2026.
doi:10.3390/rs18101656 Google Scholar

34. Wang, Kun, Zi Wang, Zhang Li, Ang Su, Xichao Teng, Erting Pan, Minhao Liu, and Qifeng Yu, "Oriented object detection in optical remote sensing images using deep learning: A survey," Artificial Intelligence Review, Vol. 58, No. 11, 350, August 2025.
doi:10.1007/s10462-025-11256-0 Google Scholar

35. Ren, Shaoqing, Kaiming He, Ross Girshick, and Jian Sun, "Faster R-CNN: Towards real-time object detection with region proposal networks," Advances in Neural Information Processing Systems, Vol. 28, 2015.

36. Pang, Jiangmiao, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin, "Libra R-CNN: Towards balanced learning for object detection," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 821-830, Long Beach, CA, USA, June 2019.
doi:10.1109/cvpr.2019.00091

37. Liu, Wei, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg, "SSD: Single shot multibox detector," Computer Vision --- ECCV 2016, 21-37, Springer, Cham, 2016.
doi:10.1007/978-3-319-46448-0_2

38. Zeng, Ying, Yushi Chen, Xue Yang, Qingyun Li, and Junchi Yan, "ARS-DETR: Aspect ratio-sensitive detection transformer for aerial oriented object detection," IEEE Transactions on Geoscience and Remote Sensing, Vol. 62, 1-15, 2024.
doi:10.1109/tgrs.2024.3364713 Google Scholar

39. Zhang, Jiaqing, Jie Lei, Weiying Xie, Zhenman Fang, Yunsong Li, and Qian Du, "SuperYOLO: Super resolution assisted object detection in multimodal remote sensing imagery," IEEE Transactions on Geoscience and Remote Sensing, Vol. 61, 1-15, 2023.
doi:10.1109/tgrs.2023.3258666 Google Scholar

40. Yang, Xiao, Yongjia Wang, Yong Wang, Wangyuan Li, Beiyuan Liu, and Ganchao Liu, "CSSA-YOLO: A clutter-suppressed and scale-aware framework for robust object detection in UAV imagery," Remote Sensing, Vol. 18, No. 10, 1533, May 2026.
doi:10.3390/rs18101533 Google Scholar

41. Han, Jianfeng, Feijie Sun, Zihan Xu, Lili Song, and Jiandong Fang, "An enhanced algorithm integrating YOLOv11 and bytetrack for small-object detection and tracking in low-altitude remote sensing imagery," Remote Sensing, Vol. 18, No. 10, 1547, May 2026.
doi:10.3390/rs18101547 Google Scholar

42. Ge, Yueran, Wenqiang Ji, Shasha Yin, and Wei Zhang, "Small object detection in remote sensing based on feature fusion and channel attention," 2024 8th International Symposium on Computer Science and Intelligent Control (ISCSIC), 195-200, Zhengzhou, China, September 2024.
doi:10.1109/iscsic64297.2024.00048

43. Yi, Hao, Bo Liu, Bin Zhao, and Enhai Liu, "Small object detection algorithm based on improved YOLOv8 for remote sensing," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 17, 1734-1747, 2024.
doi:10.1109/jstars.2023.3339235 Google Scholar

44. Sharma, Manish, Mayur Dhanaraj, Srivallabha Karnam, Dimitris G. Chachlakis, Raymond Ptucha, Panos P. Markopoulos, and Eli Saber, "YOLOrs: Object detection in multimodal remote sensing imagery," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 14, 1497-1508, 2021.
doi:10.1109/jstars.2020.3041316 Google Scholar

45. Khanam, R. and Muhammad Hussain, "Yolov11: An overview of the key architectural enhancements," arXiv:2410.17725, 2024.
doi:10.48550/arXiv.2410.17725 Google Scholar

46. Sun, Yiming, Bing Cao, Pengfei Zhu, and Qinghua Hu, "Drone-based RGB-infrared cross-modality vehicle detection via uncertainty-aware learning," IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, No. 10, 6700-6713, October 2022.
doi:10.1109/tcsvt.2022.3168279 Google Scholar

47. Jia, Xinyu, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou, "LLVIP: A visible-infrared paired dataset for low-light vision," 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 3489-3497, Montreal, BC, Canada, October 2021.
doi:10.1109/iccvw54120.2021.00389

Dr. Cheng Zhang

Dr. Xiao-Zhen Ren

Dr. Xiaqiong Fan

Dr. Zhipeng Guo

Dr. Gang Liu