GNNTA-BiLSTM: Transformer-attention mechanism bidirectional short-term memory network-based graph neural network for English text attribute extraction

Liuxin  Gao

doi:10.6180/jase.202511_28(11).0012

GNNTA-BiLSTM: Transformer-attention mechanism bidirectional short-term memory network-based graph neural network for English text attribute extraction

Research Categories

Liuxin GaoThis email address is being protected from spambots. You need JavaScript enabled to view it.

School of Foreign Languages, Zhengzhou University of Science and Technology Zhengzhou 450064, China

Received: December 10, 2024
Accepted: February 2, 2025
Publication Date: March 16, 2025

Copyright The Author(s). This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are cited.

Download Citation: ||https://doi.org/10.6180/jase.202511_28(11).0012

Attribute extraction is one of the sub-tasks of fine-grained sentiment analysis, its goal is to extract the attributes evaluated by users fromtheEnglishcommenttext. Theattribute classification accuracy of fine-grained sentiment analysis based on traditional methods is low. Therefore, this paper proposes a novel English text attribute extraction model based on Transformer-attention mechanism bidirectional short-term memory network (BERT-Att-BiLSTM) and graph neural network. Firstly, the model constructs the object information extraction layer based onsyntactic dependency in graph neural network, and extracts attribute-view pairs. Secondly, in the word embedding layer, Transformer (BERT) module is used to implement the pre-training of word vector combined with contextual dynamic features. Then, in the feature extraction layer, attention mechanism bidirectional short termmemorynetworkmoduleisintegratedtoreducethedimensionoffeaturespace. Finally, intheclassification layer, the attribute category of the genre-view pair is output through the activation function. Experiments are conducted on the laptop and restaurant domain datasets provided by the International Semantic Evaluation Congress in 2014 and 2016. Compared with the baseline model, the F1 value of the two English datasets improves by 2.33% and 1.44%, respectively, and the overall performance is higher than that of the current cutting-edge techniques.

Keywords: English text attribute extraction, Transformer-attention mechanism, bidirectional short-term memory network, graph neural network, word vector

[1] S. Yang, Y. Zhou, Z. Zheng, Y. Wang, L. Zhu, and Y. Wu. “Towards unified text-based person retrieval: A large-scale multi-attribute and language search benchmark”. In: Proceedings of the 31st ACM Interna tional Conference on Multimedia. 2023, 4492–4501. DOI: 10.1145/3581783.361170.
[2] M. Geva, J. Bastings, K. Filippova, and A. Glober son, (2023) “Dissecting recall of factual associations in auto-regressive language models" arXiv preprint arXiv:2304.14767: DOI: 10.48550/arXiv.2304.14767.
[3] S. Sivarajkumar, M. Kelley, A. Samolyk-Mazzanti, S. Visweswaran, and Y. Wang, (2024) “An empirical eval uation of prompting strategies for large language models in zero-shot clinical natural language processing: algo rithm development and validation study" JMIR Medical Informatics 12: e55318. DOI: 10.2196/55318.
[4] J. May, E. Shutova, A. Herbelot, X. Zhu, M. Apidi anaki, and S. Mohammad. “Proceedings of the 13th International Workshop on Semantic Evaluation”. In: Proceedings of the 13th International Workshop on Se mantic Evaluation. 2019.
[5] B. M.Shoja and N. Tabrizi, (2019) “Customer reviews analysis with deep neural networks for e-commerce recom mender systems" IEEE access 7: 119121–119130. DOI: 10.1109/ACCESS.2019.2937518.
[6] S. Yin, H. Li, L. Teng, A. A. Laghari, and V. V. Estrela, (2024) “Attribute-based multiparty searchable encryption model for privacy protection of text data" Multimedia Tools and Applications 83(15): 45881–45902. DOI: 10.1007/s11042-023-16818-4.
[7] J. Wang, J. Zhang, and X. Wang, (2017) “Bilateral LSTM:Atwo-dimensional long short-term memory model with multiply memory units for short-term cycle time forecasting in re-entrant manufacturing systems" IEEE Transactions on Industrial Informatics 14(2): 748 758. DOI: 10.1109/TII.2017.2754641.
[8] A. Arnab, S. Jayasumana, S. Zheng, and P. H. Torr. “Higher order conditional random fields in deep neu ral networks”. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Octo ber 11-14, 2016, Proceedings, Part II 14. Springer. 2016, 524–540. DOI: 10.1007/978-3-319-46475-6_33.
[9] R. Towe, J. Tawn, R. Lamb, and C. Sherlock, (2019) “Model-based inference of conditional extreme value distri butions with hydrological applications" Environmetrics 30(8): e2575. DOI: 10.1002/env.2575.
[10] H. Peng, L. Xu, L. Bing, F. Huang, W. Lu, and L. Si. “Knowingwhat,howandwhy:Anearcompletesolu tion for aspect-based sentiment analysis”. In: Proceed ings of the AAAI conference on artificial intelligence. 34. 05. 2020, 8600–8607. DOI: 10.1609/aaai.v34i05.6383.
[11] L. Wang, Y. Shoulin, H. Alyami, A. A. Laghari, M. Rashid, J. Almotiri, H. J. Alyamani, and F. Alturise. Anovel deep learning-based single shot multibox detector model for object detection in optical remote sensing images. 2024. DOI: 10.1002/gdj3.162.
[12] N. Engel, V. Belagiannis, and K. Dietmayer, (2021) “Point transformer" IEEE access 9: 134826–134840. DOI: 10.1109/ACCESS.2021.3116304.
[13] T.Chen,R.Xu,Y. He,andX.Wang,(2017) “Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN" Expert Systems with Appli cations 72: 221–230. DOI: 10.1016/j.eswa.2016.10.065.
[14] W. Zou, W. Zhang, W. Wu, and Z. Tian, (2024) “A multi-task shared cascade learning for aspect sentiment triplet extraction using bert-mrc" Cognitive Computa tion: 1–18. DOI: 10.1007/s12559-024-10247-7.
[15] Y. Chen, Y. Shao, J. Wang, G. Li, and C.-C. J. Kuo. “Point cloud attribute compression via successive subspace graph transform”. In: 2020 IEEE Interna tional Conference on Visual Communications and Image Processing (VCIP). IEEE. 2020, 66–69. DOI: 10.1109/VCIP49819.2020.9301784.
[16] Y. Chen, Y. Shao, J. Wang, G. Li, and C.-C. J. Kuo. “Point cloud attribute compression via successive subspace graph transform”. In: 2020 IEEE Interna tional Conference on Visual Communications and Image Processing (VCIP). IEEE. 2020, 66–69. DOI: 10.1109/VCIP49819.2020.9301784.
[17] M.Chen,W.Wu,Y.Zhang,andZ.Zhou.“Combining adversarial training and relational graph attention network for aspect-based sentiment analysis with BERT”. In: 2021 14th International Congress on Image and Signal Processing, BioMedical Engineering and In formatics (CISP-BMEI). IEEE. 2021, 1–6. DOI: 10.1109/CISP-BMEI53629.2021.9624384.
[18] R. Li, H. Chen, F. Feng, Z. Ma, X. Wang, and E. Hovy. “Dual graph convolutional networks for aspect-based sentiment analysis”. In: Proceedings of the 59th Annual Meeting of the Association for Computa tional Linguistics and the 11th International Joint Con ference on Natural Language Processing (Volume 1: Long Papers). 2021, 6319–6329. DOI: 10.18653/v1/2021.acl-long.494.
[19] C. Li, L. Bai, W. Liu, L. Yao, and S. T. Waller, (2021) “Amulti-task memory network with knowledge adaptation for multimodal demand forecasting" Transportation Re search Part C: Emerging Technologies 131: 103352. DOI: 10.1016/j.trc.2021.103352.
[20] C. Ahn, E. Kim, and S. Oh. “Deep elastic networks with model selection for multi-task learning”. In: Pro ceedings of the IEEE/CVF international conference on computer vision. 2019, 6529–6538. DOI: 10.1109/ICCV.2019.00663.
[21] Z.-T. Liu, M.-T. Han, B.-H. Wu, and A. Rehman, (2023) “Speech emotion recognition based on convolu tional neural network with attention-based bidirectional long short-term memory network and multi-task learn ing" Applied Acoustics 202: 109178. DOI: 10.1016/j.apacoust.2022.109178.
[22] X. Meng, X. Wang, S. Yin, and H. Li, (2023) “Few-shot image classification algorithm based on attention mecha nism and weight fusion" Journal of Engineering and Applied Science 70(1): 14. DOI: 10.1186/s44147-023 00186-9.
[23] J. H. Caufield, H. Hegde, V. Emonet, N. L. Har ris, M. P. Joachimiak, N. Matentzoglu, H. Kim, S. Moxon, J. T. Reese, M. A. Haendel, et al., (2024) “Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowl edge bases using zero-shot learning" Bioinformatics 40(3): btae104. DOI: 10.1093/bioinformatics/btae104.
[24] A. P. Darko, D. Liang, Z. Xu, K. Agbodah, and S. Obiora, (2023) “A novel multi-attribute decision-making for ranking mobile payment services using online con sumer reviews" Expert Systems with Applications 213: 119262. DOI: 10.1016/j.eswa.2022.119262.
[25] W. Wang, S. J. Pan, and D. Dahlmeier, (2017) “Multi-task memory networks for category-specific as pect and opinion terms co-extraction" arXiv preprint arXiv:1702.01776: DOI: 10.48550/arXiv.1702.01776.