Multimodal Affective Sensing Driven Adaptive Intervention Model for Second Language Learning Stress

Yixin Hua

doi:10.54254/2753-7064/2025.25788

Communications in Humanities ResearchOpen access

Multimodal Affective Sensing Driven Adaptive Intervention Model for Second Language Learning Stress

Research Article

Open Access

Multimodal Affective Sensing Driven Adaptive Intervention Model for Second Language Learning Stress

Yixin Hua ^1*

¹ University of Melbourne, Melbourne, Australia

^*Corresponding author: yixinhua2002@outlook.com

Published on 6 August 2025

CHR Vol.73

ISSN (Print): 2753-7072

ISSN (Online): 2753-7064

ISBN (Print): 978-1-80590-267-6

ISBN (Online): 978-1-80590-268-3

Download Cover

Abstract

In high-pressure learning contexts, second language (L2) learners often struggle with the interplay between anxiety and cognitive overload, significantly impairing their language acquisition efficiency. However, most existing intelligent education systems fail to dynamically detect and regulate learners’ psychological states throughout the learning process. To address this gap, this study proposes an adaptive intervention model that integrates multimodal emotion recognition with reinforcement learning-based strategy optimization. By capturing learners’ facial expressions, vocal prosody, and physiological signals at key task stages, the system enables real-time emotional state recognition and generates personalized intervention strategies based on emotional feedback. Sixty university-level English learners were recruited for an 8-week randomized controlled trial. While the control group followed a conventional instructional approach, the experimental group was supported by the emotion-sensing intervention system. The study employed standardized language tests and system performance metrics to assess the effectiveness of the intervention, supplemented by learning logs and interviews to collect subjective user feedback. Results show that the experimental group outperformed the control group in terms of emotion recognition accuracy, intervention response latency, and language performance improvement. Learners also reported high acceptance and positive evaluations of the system. This research validates the feasibility of multimodal affective sensing in mitigating learning stress and provides both technical and empirical foundations for emotion-adaptive intelligent education systems, thereby expanding the application boundary of affective computing in intelligent instruction.

Keywords:

Multimodal sensing, Emotion recognition, Learning stress, Personalized intervention, Second language acquisition

View PDF

References

[1]. Immadisetty, Praneeta, et al. "Multimodality in online education: a comparative study." Multimedia Tools and Applications (2025): 1-34.

[2]. Guo, Xiaoshuang. "Multimodality in language education: implications of a multimodal affective perspective in foreign language teaching." Frontiers in Psychology 14 (2023): 1283625.

[3]. Alqarni, Nada A. "Predictors of foreign language proficiency: Emotion regulation, foreign language enjoyment, or academic stress?." System 126 (2024): 103462.

[4]. Tanaka, Hiroki, et al. "4th Workshop on Social Affective Multimodal Interaction for Health (SAMIH)." Proceedings of the 25th International Conference on Multimodal Interaction. 2023.

[5]. Govea, Jaime, et al. "Implementation of deep reinforcement learning models for emotion detection and personalization of learning in hybrid educational environments." Frontiers in Artificial Intelligence 7 (2024): 1458230.

[6]. Yan, Lixiang, et al. "Scalability, sustainability, and ethicality of multimodal learning analytics." LAK22: 12th international learning analytics and knowledge conference. 2022.

[7]. Pathirana, Amod, et al. "A Reinforcement Learning-Based Approach for Promoting Mental Health Using Multimodal Emotion Recognition." Journal of Future Artificial Intelligence and Technologies 1.2 (2024): 124-142.

[8]. Rahman, Fathima Abdul, and Guang Lu. "A Contextualized Real-Time Multimodal Emotion Recognition for Conversational Agents using Graph Convolutional Networks in Reinforcement Learning." arXiv preprint arXiv: 2310.18363 (2023).

[9]. Devulapally, Naresh Kumar, et al. "AM^ 2-EmoJE: Adaptive Missing-Modality Emotion Recognition in Conversation via Joint Embedding Learning." arXiv preprint arXiv: 2402.10921 (2024).

[10]. Wang, Zhuozheng, and Yihan Wang. "Emotion recognition based on multimodal physiological electrical signals." Frontiers in Neuroscience 19 (2025): 1512799.

[11]. Zhao, Jiaxing, Xihan Wei, and Liefeng Bo. "R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning." arXiv preprint arXiv: 2503.05379 (2025).