The Cycle-consistent Adversarial Semantics-Texture Learning for Image Synthesis in Internet of Medical Things

Jianing  Zhang; Zhikui  Chen; Yingshu  Liu; Jing  Gao; Peng  Li

doi:10.6180/jase.202505_28(5).0011

The Cycle-consistent Adversarial Semantics-Texture Learning for Image Synthesis in Internet of Medical Things

Computer Science and Information Engineering

Jianing Zhang¹, Zhikui Chen¹, Yingshu Liu², Jing Gao¹This email address is being protected from spambots. You need JavaScript enabled to view it., and Peng Li³

¹School of Software Technology, Dalian University of Technology, Dalian, 116620, China

²Department of Endocrinology, Central Hospital of Dalian University of Technology, Dalian, 116033, China

³School of Computer Science and Technology, Dalian University of Technology, Dalian, 116620, China

Received: March 30, 2024
Accepted: June 5, 2024
Publication Date: July 11, 2024

Copyright The Author(s). This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are cited.

Download Citation: ||https://doi.org/10.6180/jase.202505_28(5).0011

With the wide deployment of Internet of Medical Things, a great number of incomplete multi-modal medical data that violate the data integrity are collected in many applications, producing negative impacts on diagnosis for patients. AI-driven medical image synthesis is of great significance in recovering the missing images with complementary information. However, existing methods depend on single-modal schemes devised for wild image synthesis, which neglect information loss caused by high heterogeneity between modalities of medical images in transferring semantics of source domains. Meanwhile, they cannot take into account semanticstexture consistencies in generating medical images, which causes distortions of lesions in synthesizing medical images. To address challenges above, the cycle-consistent adversarial semantics-texture learning (Cycle-STAR) is proposed for medical image synthesis via defining a cycle-consistent adversarial auto-encoder for medical image synthesis. In detail, a hierarchical disentangled cycle-consistent adversarial paradigm is designed to learn the semantics-texture consistencies of medical images. Then, a cycle alignment fusion loss is proposed for the network training, which encourages the alignment of semantics between domains to prevent information loss in transferring semantics of source domain. Finally, extensive experimental results in two typical medical scenarios illustrate that Cycle-STAR achieves the superior performance to thirteen methods.

Keywords: Medical image synthesis; Generative adversarial learning; Cycle-consistent adversarial auto-encoder; Semantics alignment

[1] A. Ghubaish, T. Salman, M. Zolanvari, D. Unal, A. K. Al-Ali, and R. Jain, (2021) “Recent advances in the Internet-of-Medical-Things (IoMT) systems security" IEEE Internet of Things Journal 8(11): 8707–8718. DOI: 10.1109/jiot.2020.3045653.
[2] O. Dalmaz, M. Yurt, and T. Çukur, (2022) “ResViT: Residual vision transformers for multimodal medical image synthesis" IEEE Transactions on Medical Imaging 41(10): 2598–2614. DOI: 10.1109/tmi.2022.3167808.
[3] A. Jog, A. Carass, S. Roy, D. L. Pham, and J. L. Prince, (2015) “MR image synthesis by contrast learning on neighborhood ensembles" Medical Image Analysis 24(1): 63–76. DOI: 10.1016/j.media.2015.05.002.
[4] Y. Huang, L. Shao, and A. F. Frangi, (2018) “Crossmodality image synthesis via weakly coupled and geometry co-regularized joint dictionary learning" IEEE Transactions on Medical Imaging 37(3): 815–827. DOI: 10.1109/tmi.2017.2781192.
[5] P. Li, J. Gao, J. Zhang, S. Jin, and Z. Chen, (2023) “Deep reinforcement clustering" IEEE Transactions on Multimedia 25: 8183–8193. DOI: 10.1109/tmm.2022.3233249.
[6] J. Gao, M. Liu, P. Li, J. Zhang, and Z. Chen, (2023) “Deep multiview adaptive clustering with semantic invariance" IEEE Transactions on Neural Networks and Learning Systems: 1–14. DOI: 10.1109/tnnls.2023.3265699.
[7] J. Gao, P. Li, A. A. Laghari, G. Srivastava, T. R. Gadekallu, S. Abbas, and J. Zhang, (2024) “Incomplete multiview clustering via semidiscrete optimal transport for multimedia data mining in IoT" ACM Transactions on Multimedia Computing, Communications and Applications 20(6): 158:1–158:20. DOI: 10.1145/3625548.
[8] P. Li, A. A. Laghari, M. Rashid, J. Gao, T. R. Gadekallu, A. R. Javed, and S. Yin, (2023) “A Deep multimodal adversarial cycle-consistent network for smart enterprise system" IEEE Transactions on Industrial Informatics 19(1): 693–702. DOI: 10.1109/tii.2022.3197201.
[9] P. Subramaniam, T. Kossen, K. Ritter, A. Hennemuth, K. Hildebrand, A. Hilbert, J. Sobesky, M. Livne, I. Galinovic, A. A. Khalil, J. B. Fiebach, D. Frey, and V. I. Madai, (2022) “Generating 3D TOF-MRA volumes and segmentation labels using generative adversarial networks" Medical Image Analysis 78: 102396. DOI: 10.1016/j.media.2022.102396.
[10] W. Li, J. Li, J. Polson, Z. Wang, W. Speier, and C. W. Arnold, (2022) “High resolution histopathology image generation and segmentation through adversarial training" Medical Image Analysis 75: 102251. DOI: 10.1016/j.media.2021.102251.
[11] M. Hamghalam and A. L. Simpson, (2024) “Medical image synthesis via conditional GANs: Application to segmenting brain tumours" Computers in Biology and Medicine 170: 107982. DOI: 10.1016/j.compbiomed.2024.107982.
[12] Y. Luo, L. Zhou, B. Zhan, F. Wang, J. Zhou, Y. Wang, and D. Shen, (2022) “Adaptive rectification based adversarial network with spectrum constraint for high-quality PET image synthesis" Medical Image Analysis 77: 102335. DOI: 10.1016/j.media.2021.102335.
[13] S. Hu, B. Lei, S. Wang, Y. Wang, Z. Feng, and Y. Shen, (2022) “Bidirectional mapping generative adversarial networks for brain MR to PET synthesis" IEEE Transactions on Medical Imaging 41(1): 145–157. DOI: 10.1109/tmi.2021.3107013.
[14] L. Han, Y. Lyu, C. Peng, and S. K. Zhou, (2022) “GANbased disentanglement learning for chest X-ray rib suppression" Medical Image Analysis 77: 102369. DOI: 10.1016/j.media.2022.102369.
[15] L. Shen, L. Yu, W. Zhao, J. M. Pauly, and L. Xing, (2022) “Novel-view X-ray projection synthesis through geometry-integrated deep learning" Medical Image Analysis 77: 102372. DOI: 10.1016/j.media.2022.102372.
[16] Y. Heng, M. Yinghua, F. G. Khan, A. Khan, and Z. Hui, (2024) “HLSNC-GAN: Medical image synthesis using hinge loss and switchable normalization in CycleGAN" IEEE Access 12: 55448–55464. DOI: 10.1109/access.2024.3390245.
[17] X. Lu, X. Liang, W. Liu, X. Miao, and X. Guan, (2024) “ReeGAN: MRI image edge-preserving synthesis based on GANs trained with misaligned data" Medical and Biological Engineering and Computing 62: 1851–1868. DOI: 10.1007/s11517-024-03035-w.
[18] K. Wei, W. Kong, L. Liu, J. Wang, B. Li, B. Zhao, Z. Li, J. Zhu, and G. Yu, (2024) “CT synthesis from MR images using frequency attention conditional generative adversarial network" Computers in Biology and Medicine 170: 107983. DOI: 10.1016/j.compbiomed.2024.107983.
[19] X. Li, S. Zhang, J. Hu, L. Cao, X. Hong, X. Mao, F. Huang, Y. Wu, and R. Ji. “Image-to-image translation via hierarchical style disentanglement”. In: IEEE Conference on Computer Vision and Pattern Recognition. Computer Vision Foundation / IEEE, 2021, 8639– 8648. DOI: 10.1109/cvpr46437.2021.00853.
[20] J. Cao, L. Hou, M. Yang, R. He, and Z. Sun. “ReMix: Towards image-to-image translation with limited data”. In: IEEE Conference on Computer Vision and Pattern Recognition. Computer Vision Foundation / IEEE, 2021, 15018–15027. DOI: 10.1109/cvpr46437.2021.01477.
[21] J. Liang, H. Zeng, and L. Zhang. “High-resolution photorealistic image translation in real-time: A Laplacian pyramid translation network”. In: IEEE Conference on Computer Vision and Pattern Recognition. Computer Vision Foundation / IEEE, 2021, 9392–9400. DOI: 10.1109/cvpr46437.2021.00927.
[22] F. Zhan, Y. Yu, K. Cui, G. Zhang, S. Lu, J. Pan, C. Zhang, F. Ma, X. Xie, and C. Miao. “Unbalanced feature transport for exemplar-based image translation”. In: IEEE Conference on Computer Vision and Pattern Recognition. Computer Vision Foundation / IEEE, 2021, 15028–15038. DOI: 10.1109/cvpr46437.2021.01478.
[23] X. Zhou, B. Zhang, T. Zhang, P. Zhang, J. Bao, D. Chen, Z. Zhang, and F. Wen. “CoCosNet v2: Fullresolution correspondence learning for image translation”. In: IEEE Conference on Computer Vision and Pattern Recognition. Computer Vision Foundation / IEEE, 2021, 11465–11475. DOI: 10.1109/cvpr46437.2021.01130.
[24] G. Liu, Q. Zhou, X. Xie, and Q. Yu, (2023) “Dual conditional GAN based on external attention for semantic image synthesis" Connection Science 35(1): DOI: 10.1080/09540091.2023.2259120.
[25] B. Yang, X. Xiang, W. Kong, J. Zhang, and Y. Peng, (2024) “DMF-GAN: Deep multimodal fusion generative adversarial networks for text-to-image synthesis" IEEE Transactions on Multimedia 26: 6956–6967. DOI: 10.1109/tmm.2024.3358086.
[26] A. Radford, L. Metz, and S. Chintala. “Unsupervised representation learning with deep convolutional generative adversarial networks”. In: International Conference on Learning Representations. OpenReview.net, 2016.
[27] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. “Improved training of Wasserstein GANs”. In: Annual Conference on Neural Information Processing Systems. 2017, 5767–5777.
[28] X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley. “Least squares generative adversarial networks”. In: IEEE International Conference on Computer Vision. IEEE Computer Society, 2017, 2813–2821. DOI: 10.1109/iccv.2017.304.
[29] J. Yang, A. Kannan, D. Batra, and D. Parikh. “LRGAN: Layered recursive generative adversarial networks for image generation”. In: International Conference on Learning Representations. OpenReview.net, 2017.
[30] T. Karras, T. Aila, S. Laine, and J. Lehtinen. “Progressive growing of GANs for improved quality, stability, and variation”. In: International Conference on Learning Representations. OpenReview.net, 2018.
[31] H. Zhang, I. J. Goodfellow, D. N. Metaxas, and A. Odena. “Self-attention generative adversarial networks”. In: International Conference on Machine Learning. PMLR, 2019, 7354–7363.
[32] P. Isola, J. Zhu, T. Zhou, and A. A. Efros. “Imageto-image translation with conditional adversarial networks”. In: IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, 2017, 5967–5976. DOI: 10.1109/cvpr.2017.632.
[33] J. Zhu, T. Park, P. Isola, and A. A. Efros. “Unpaired image-to-image translation using cycle-consistent adversarial networks”. In: IEEE International Conference on Computer Vision. IEEE Computer Society, 2017, 2242–2251. DOI: 10.1109/iccv.2017.244.
[34] J. Kim, M. Kim, H. Kang, and K. Lee. “U-GAT-IT: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-toimage translation”. In: International Conference on Learning Representations. OpenReview.net, 2020.
[35] T. Zhou, H. Fu, G. Chen, J. Shen, and L. Shao, (2020) “Hi-Net: Hybrid-fusion network for multi-modal MR image synthesis" IEEE Transactions on Medical Imaging 39(9): 2772–2781. DOI: 10.1109/tmi.2020.2975344.
[36] J. Gao, W. Zhao, P. Li, W. Huang, and Z. Chen, (2022) “LEGAN: A light and effective generative adversarial network for medical image synthesis" Computers in Biology and Medicine 148: 105878. DOI: 10.1016/j.compbiomed.2022.105878.