volume | PIER Journals

Abstract

Due to the optical diffraction limit, the resolution of a wide-field (WF) microscope cannot easily go below a few hundred nanometers. Super-resolution microscopy has the disadvantages of high cost, complex optical equipment, and high experimental environment requirements. Deep-learning-based super-resolution (DLSR) has the advantages of simple operation and low cost, and has attracted much attention recently. Here we propose a novel DLSR model named Modified Hybrid Task Cascade U-Net (MHTCUN) for image super-resolution in optical microscopy using the public biological image dataset BioSR. The MHTCUN has three stages, and we introduce a novel module named Feature Refinement Module (FRM) to extract deeper features in each stage. In each FRM, a U-Net is introduced to refine the features, and the Fourier Channel Attention Block (FCAB) is introduced in the U-Net to learn the high-level representation of the high-frequency information of different feature maps. Compared with six state-of-the-art DLSR models used for single-image super-resolution (SISR), our MHTCUN achieves the highest signal-to-noise ratio (PSNR) of 26.87 and structural similarity (SSIM) of 0.746, demonstrating that our MHTCUN has achieved the state-of-the-art in DLSR. Compared with the DLSR model DFCAN used for image super-resolution in optical microscopy specially, MHTCUN has a significant improvement in PSNR and a slight improvement in SSIM on BioSR. Finally, we fine-tune the trained MHTCUN on the other biological images. MHTCUN also shows good performance on denoising, contrast enhancement, and resolution enhancement.

1. Zhao, Z.-Q., P. Zheng, S.-T. Xu, and X. Wu, "Object detection with deep learning: A review," IEEE Transactions on Neural Networks and Learning Systems, Vol. 30, No. 11, 3212-3232, 2019.
doi:10.1109/TNNLS.2018.2876865 Google Scholar

2. Trajanovski, S., C. Shan, P. J. C. Weijtmans, S. G. B. de Koning, and T. J. M. Ruers, "Tongue tumor detection in hyperspectral images using deep learning semantic segmentation," IEEE Transactions on Biomedical Engineering, Vol. 68, No. 4, 1330-1340, 2020.
doi:10.1109/TBME.2020.3026683 Google Scholar

3. Zhao, S., D. M. Zhang, and H. W. Huang, "Deep learning-based image instance segmentation for moisture marks of shield tunnel lining," Tunnelling and Underground Space Technology, Vol. 95, 103156, 2020.
doi:10.1016/j.tust.2019.103156 Google Scholar

4. Yang, W., X. Zhang, Y. Tian, W. Wang, J.-H. Xue, and Q. Liao, "Deep learning for single image super-resolution: A brief review," IEEE Transactions on Multimedia, Vol. 21, No. 12, 3106-3121, 2019.
doi:10.1109/TMM.2019.2919431 Google Scholar

5. Ledig, C., L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z.Wang, et al. "Photo-realistic single image super-resolution using a generative adversarial network," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4681-4690, 2017. Google Scholar

6. Kim, K. I. and Y. Kwon, "Single-image super-resolution using sparse regression and natural image prior," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No. 6, 1127-1133, 2010.
doi:10.1109/TPAMI.2010.25 Google Scholar

7. Kirkland, E. J., "Bilinear interpolation," Advanced Computing in Electron Microscopy, 261-263, Springer, 2010.
doi:10.1007/978-1-4419-6533-2_12 Google Scholar

8. Liu, T., K. De Haan, Y. Rivenson, Z. Wei, X. Zeng, Y. Zhang, and A. Ozcan, "Deep learning-based super-resolution in coherent imaging systems," Scientic Reports, Vol. 9, No. 1, 1-13, 2019. Google Scholar

9. Dong, C., C. C. Loy, K. He, and X. Tang, "Image super-resolution using deep convolutional networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38, No. 2, 295-307, 2015.
doi:10.1109/TPAMI.2015.2439281 Google Scholar

10. He, K., X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770-778, 2016. Google Scholar

11. Lim, B., S. Son, H. Kim, S. Nah, and K. M. Lee, "Enhanced deep residual networks for single image super-resolution," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 136-144, 2017. Google Scholar

12. Zhang, Y., K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, "Image superresolution using very deep residual channel attention networks," Proceedings of the European Conference on Computer Vision (ECCV), 286-301, 2018. Google Scholar

13. Huang, G., Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4700-4708, 2017. Google Scholar

14. Zhang, Y., Y. Tian, Y. Kong, B. Zhong, and Y. Fu, "Residual dense network for image super-resolution," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2472-2481, 2018. Google Scholar

15. Li, Z., J. Yang, Z. Liu, X. Yang, G. Jeon, and W. Wu, "Feedback network for image super-resolution," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3867-3876, 2019. Google Scholar

16. Ronneberger, O., P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," International Conference on Medical Image Computing and Computer-assisted Intervention, 234-241, Springer, 2015. Google Scholar

17. Chen, K., J. Pang, J. Wang, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Shi, W. Ouyang, et al. "Hybrid task cascade for instance segmentation," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4974-4983, 2019. Google Scholar

18. Cai, Z. and N. Vasconcelos, "Cascade r-cnn: Delving into high quality object detection," Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 6154-6162, 2018. Google Scholar

19. Hell, S. W. and J. Wichmann, "Breaking the diffraction resolution limit by stimulated emission: Stimulated-emission-depletion uorescence microscopy," Optics Letters, Vol. 19, No. 11, 780-782, 1994. Google Scholar

20. Hess, S. T., T. P. K. Girirajan, and M. D. Mason, "Ultra-high resolution imaging by fluorescence photoactivation localization microscopy," Biophysical Journal, Vol. 91, No. 11, 4258-4272, 2006. Google Scholar

21. Rust, M. J., M. Bates, and X. Zhuang, "Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (storm)," Nature Methods, Vol. 3, No. 10, 793-796, 2006. Google Scholar

22. Gustafsson, M. G. L., "Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy," Journal of Microscopy, Vol. 198, No. 2, 82-87, 2000. Google Scholar

23. Weigert, M., U. Schmidt, T. Boothe, A. Muller, A. Dibrov, A. Jain, B. Wilhelm, D. Schmidt, C. Broaddus, S. Culley, et al. "Content-aware image restoration: Pushing the limits of fluorescence microscopy," Nature Methods, Vol. 15, No. 12, 1090-1097, 2018. Google Scholar

24. Wang, H., Y. Rivenson, Y. Jin, Z. Wei, R. Gao, H. Gunaydin, L. A. Bentolila, C. Kural, and A. Ozcan, "Deep learning enables cross-modality superresolution in fluorescence microscopy," Nature Methods, Vol. 16, No. 1, 103-110, 2019. Google Scholar

25. Qiao, C., D. Li, Y. Guo, C. Liu, T. Jiang, Q. Dai, and D. Li, "Evaluation and development of deep neural networks for image super-resolution in optical microscopy," Nature Methods, Vol. 18, No. 2, 194-202, 2021. Google Scholar

26. Shi, W., J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1874-1883, 2016. Google Scholar

27. Howard, A. G., M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, "Mobilenets: Efficient convolutional neural networks for mobile vision applications,", arXiv preprint arXiv:1704.04861, 2017. Google Scholar

28. Ramachandran, P., B. Zoph, and Q. V. Le, "Searching for activation functions,", arXiv preprint arXiv:1710.05941, 2017. Google Scholar

29. Lin, T.-Y., P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2117-2125, 2017. Google Scholar

30. Allen, D. M., "Mean square error of prediction as a criterion for selecting variables," Technometrics, Vol. 13, No. 3, 469-475, 1971. Google Scholar

31. Wang, Z., A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity," IEEE Transactions on Image Processing, Vol. 13, No. 4, 600-612, 2004. Google Scholar

32. Szegedy, C., W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1-9, 2015. Google Scholar

33. Descloux, A., K. S. Gruβmayer, and A. Radenovic, "Parameter-free image resolution estimation based on decorrelation analysis," Nature Methods, Vol. 16, No. 9, 918-924, 2019. Google Scholar

34. Abramoff, M. D., P. J. Magalhaes, and S. J. Ram, "Image processing with imagej," Biophotonics International, Vol. 11, No. 7, 36-42, 2004. Google Scholar

Dr. Dawei Gong

Dr. Tengfei Ma

Dr. Julian Evans

Prof. Sailing He