Behavior Anomaly Detection Based on Multi-modal Feature Fusion and Its Application in English Teaching

Lei  Kan; Man  Wang

doi:10.6180/jase.202509_28(9).0002

Behavior Anomaly Detection Based on Multi-modal Feature Fusion and Its Application in English Teaching

Research Categories

Lei Kan¹This email address is being protected from spambots. You need JavaScript enabled to view it. and Man Wang^2,3

¹Zhengzhou Vocational College of Intelligent Technology, 451161 Zhengzhou, China

²Graduate school of Party School of the Central Committee of CP.C (National Academy of Governance), 100089 Beijing, China

³College of Marxism, Shenzhen MSU-BIT University, 518000 Shenzhen China

Received: October 8, 2024
Accepted: November 5, 2024
Publication Date: December 7, 2024

Copyright The Author(s). This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are cited.

Download Citation: ||https://doi.org/10.6180/jase.202509_28(9).0002

In order to improve the teaching quality, this paper proposes a multi-modal feature fusion-based abnormal behavior detection method, aiming at the problems of false detection, missing detection and imbalance of positive and negative samples in the abnormal behavior detection of students in class. The new method consists of encoder module, detection module and decoder module. The encoder module is used to extract the characteristic information of students behavior image and transfer it to the detection module. The behavior detection module obtains moreimageinformationthroughthefeaturefusiongrouptoreducethecolordistortion and artifacts of the behavior image, and transfers the obtained image information to the deep normalization correction convolution block to reduce the covariate shift and make the model easier to train. The multi-path feature convolution block can obtain image information with richer texture details. Finally, the decoder module converts the low-dimensional feature mapping back to the high-dimensional original input space through deconvolution and up-sampling operations to obtain the behavior detection image.

Keywords: Abnormal behavior detection, Multimodal feature fusion, Encoder-decoder, Multi-path feature convolution block

[1] H.Wang,S.Zhang,andS.e.a.Zhao,(2022)“Real-time detectionandtrackingoffishabnormalbehaviorbased onimprovedYOLOV5andSiamRPN++"Computers andElectronicsinAgriculture192:DOI:10.1016/j.compag.2021.106512.
[2] L. Xia and Z. Li, (2021) “A new method of abnormal behavior detection using LSTM network with temporal at tention mechanism" The Journal of Supercomputing 77(4): 3223–3241. DOI: 10.1007/s11227-020-03391-y.
[3] J. AandS. Yin, (2021) “A New Feature Fusion Network for Student Behavior Recognition in Education" Journal of Applied Science and Engineering 24(2): 133–140. DOI: 10.6180/jase.202104_24(2).0002.
[4] A. Lentzas and D. Vrakas, (2020) “Non-intrusive hu man activity recognition and abnormal behavior detec tion on elderly people: a review" Springer Netherlands 53(3): 1975–2021. DOI: 10.1007/S10462-019-09724-5.
[5] J. M. Dealmeida and C. F. T. e. a. Pontes, (2021) “Ab normal Behavior Detection Based on Traffic Pattern Cate gorization in Mobile Networks" IEEE transactions on network and service management 18(18-4): 4213 4224. DOI: 10.1109/TNSM.2021.3125019.
[6] X. Hu, J. Dai, and Y. H. et al., (2020) “A weakly su pervised framework for abnormal behavior detection and localization in crowded scenes" Neurocomputing 383: 270–281. DOI: 10.1016/J.NEUCOM.2019.11.087.
[7] M. Guo, R. Li, Y. Wang, S. Ma, Y. Zhang, S. Li, H. Zhang, Z. Liu, C. You, and H. Zheng, (2022) “Lacto bacillus plantarum ST-III modulates abnormal behavior and gut microbiota in a mouse model of autism spectrum disorder" Physiology & Behavior 257: 113965. DOI: 10.1016/j.physbeh.2022.113965.
[8] H.A.e. a. Liguo Wang Yin Shoulin, (2024) “A novel deep learning-based single shot multibox detector model for object detection in optical remote sensing images" Physiology & Behavior 11(3): 237–251. DOI: 10.1002/gdj3.162.
[9] K. Rezaee, M. R. Khosravi, and M. S. Anari, (2022) “Deep-Transfer-Learning-Based Abnormal Behavior Recog nition Using Internet of Drones for Crowded Scenes" IEEE Internet Things Mag. 5(2): 41–44. DOI: 10.1109/IOTM.001.2100138.
[10] L.Luo,S.Xie,H.Yin,C.Peng,andY.Ong,(2024)“De tecting and Quantifying Crowd-Level Abnormal Behav iors in Crowd Events" IEEE Trans. Inf. Forensics Se cur. 19: 6810–6823. DOI: 10.1109/TIFS.2024.3423388.
[11] Y. Pan, (2021) “Network security and user abnormal be havior detection by using deep neural network" Internet Technol. Lett. 4(3): DOI: 10.1002/ITL2.260.
[12] L. Luo, Y. Li, H. Yin, S. Xie, R. Hu, and W. Cai. “Crowd-Level Abnormal Behavior Detection via Multi-Scale MotionConsistencyLearning”.In: Thirty Seventh AAAI Conference on Artificial Intelligence, AAAI2023, Thirty-Fifth Conference on Innovative Appli cations of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intel ligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023. Ed. by B. Williams, Y. Chen, and J. Neville. AAAI Press, 2023, 8984–8992. DOI: 10.1609/AAAI.V37I7.26079.
[13] H.Cao, (2024) “The Detection of Abnormal Behavior by Artificial Intelligence Algorithms Under Network Secu rity" IEEE Access 12: 118605–118617. DOI: 10.1109/ACCESS.2024.3436541.
[14] B. Li, X. Guo, J. Yu, H. Sun, X. Zhao, Y. Sun, X. Dai, Q. Kuang, J. Ling, and J. Liu, (2024) “Targeting cancer stress-associated hyperinsulinemia and abnormal behavior mitigates lung carcinoma in postmenopausal mouse: In tervention function of peimine" Journal of Functional Foods 121: 106400. DOI: 10.1016/j.jff.2024.106400.
[15] A. A. A Eghtesad A Saghaei, (2024) “Prediction and monitoring of ton-kilometers and waybill for detecting ab normal behavior" Journal of Industrial Management Perspective 121(4): 249–273. DOI: 10.1016/j.jff.2024.106400.
[16] Y. Zhang, C. Chen, and K. Chen, (2023) “Combined exposure to microplastics and amitriptyline induced ab normal behavioral responses and oxidative stress in the eyes of zebrafish (Danio rerio)" Comparative Biochem istry and Physiology, Part C 273: DOI: 10.1016/j.cbpc.2023.109717.
[17] S. Li, F. Liu, and L. Jiao. “Self-Training Multi Sequence Learning with Transformer for Weakly Su pervised Video Anomaly Detection”. In: Thirty-Sixth AAAIConference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Sympo sium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22- March 1, 2022. AAAI Press, 2022, 1395–1403. DOI: 10.1609/AAAI.V36I2.20028.
[18] J.Shi,X.Chang,P.Guo,S.Watanabe,Y.Fujita,J.Xu,B. Xu, and L. Xie. “Sequence to Multi-Sequence Learn ing via Conditional Chain Mapping for Mixture Sig nals”. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin. 2020.
[19] P. Wu, X. Liu, and J. Liu, (2023) “Weakly Supervised Audio-Visual Violence Detection" IEEE Trans. Multim. 25: 1674–1685. DOI: 10.1109/TMM.2022.3147369.
[20] S. Mai, H. Hu, and S. Xing, (2022) “A Unimodal Repre sentation Learning and Recurrent Decomposition Fusion Structure for Utterance-Level Multimodal Embedding Learning" IEEE Trans. Multim. 24: 2488–2501. DOI: 10.1109/TMM.2021.3082398.
[21] E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov. “Analyzing Multi-Head Self-Attention: Spe cialized Heads Do the Heavy Lifting, the Rest Can Be Pruned”. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. Ed. by A. Korhonen, D. R. Traum, and L. Màrquez. Association for Computational Linguistics, 2019, 5797–5808. DOI: 10.18653/V1/P19-1580.
[22] L.Teng,Y. Qiao, M. Shafiq, G. Srivastava, A. R. Javed, T. R. Gadekallu, and S. Yin, (2023) “FLPK-BiSeNet: Federated Learning Based on Priori Knowledge and Bilat eral Segmentation Network for Image Edge Extraction" IEEE Trans. Netw. Serv. Manag. 20(2): 1529–1542. DOI: 10.1109/TNSM.2023.3273991.
[23] V. Valarmathi and S. Sudha, (2024) “Enhancing pub lic safety: a hybrid Conv_Trans-OptBiSVM approach for real-time abnormal behavior detection in crowded environ ments" Signal, Image and Video Processing 18(11): 7513–7525.
[24] Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma. “Single-Image Crowd Counting via Multi-Column Convolutional Neural Network”. In: 2016 IEEE Con ference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, 2016, 589–597. DOI: 10.1109/CVPR.2016.70.
[25] S. Yin, H. Li, A. A. Laghari, T. R. Gadekallu, G. A. R. Sampedro, and A. S. Almadhor, (2024) “An Anomaly Detection Model Based on Deep Auto-Encoder and Cap sule Graph Convolution via Sparrow Search Algorithm in 6G Internet of Everything" IEEE Internet Things J. 11(18): 29402–29411. DOI: 10.1109/JIOT.2024.3353337.
[26] H. Shi, X. Xu, Y. Fan, C. Zhang, and Y. Peng. “An Auto Encoder Network Based Method for Abnormal behavior detection”. In: ICSIM 2021: 2021 The 4th International Conference on Software Engineering and Information Management, Yokohama Japan, January 16 18, 2021. Ed. by Y. Li and H. Nishi. ACM, 2021, 243 251. DOI: 10.1145/3451471.3451509.
[27] J. Wang, L. Su, S. Han, D. Song, and F. Miao, (2024) “Towards Safe Autonomy in Hybrid Traffic: Detecting Un predictable Abnormal Behaviors of Human Drivers via Information Sharing" ACM Trans. Cyber Phys. Syst. 8(2): 16. DOI: 10.1145/3616398.
[28] M.Jiwei, L. Zhipeng, and L. Xuejiao, (2020) “Feature Contour Construction Methods Based on Neural Network for Network Abnormal Behavior Detection" Journal of Physics Conference Series 1693: 012001.
[29] Y. Zhao, R. Li, X. Zhang, Y. Cao, and X. Chen. “Re search on Recognition Algorithm of Abnormal Be havior of Workers in Two-Stream Convolutional Net work”. In: Journal of Physics: Conference Series. 1621. 1. IOP Publishing. 2020, 012020. DOI: 10.1088/1742-6596/1621/1/012020.
[30] S. LIU, C. LI, Y. LIU, and Y. WANG, (2024) “SH YOLO: Small Target High Performance YOLO for abnormal behavior detection in escalator scene" IE ICE TRANSACTIONS on Information and Sys tems 107(11): 1468–1471. DOI: 10.1587/transinf.2024EDL8011.
[31] A. Khayrat, P. Malak, M. Victor, S. Ahmed, H. Metawie, V. Saber, and M. Elshalakani. “An intel ligent surveillance system for detecting abnormal behaviors on campus using YOLO and CNN-LSTM networks”. In: 2022 2nd International Mobile, Intelli gent, and Ubiquitous Computing Conference (MIUCC). IEEE. 2022, 104–109. DOI: 10.1109/MIUCC55081.2022.9781786.
[32] T. S. RR Rachmadi A Sudarsono, (2021) “Performance Analysis of Driver Abnormal Behavior Classification Sys tem Using Enhanced Multi-Layer Perceptron" IEEE 2021 International Electronics Symposium (IES): DOI: 10.1109/IES53407.2021.9593974.
[33] X.Q.Dong,X.C.Wang,B.J.Li, H.Y. Wang,andG.C. Chen, (2024) “MP-Abr: a framework for intelligent recog nition of abnormal behaviour inmulti-person scenarios" Multimedia tools and applications (18): 83.
[34] K. H. ZWChouYKLu,(2021) “SVM-based learning model for abnormal behaviors in smart home" The Inter national Journal of Engineering and Science (IJES) 10: 57–63.