Research on Pilot Behavior Detection Model Based on Multi-Source Heterogeneous Data
Research Article
Open Access
CC BY

Research on Pilot Behavior Detection Model Based on Multi-Source Heterogeneous Data

Ziyi Zu 1* Yu Shen 2
1 School of Aeronautical Engineering, Civil Aviation University of China, Tianjin, China, 300300
2 School of Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai, China, 200240
*Corresponding author: 220140416@cauc.edu.cn
Published on 4 July 2025
Volume Cover
ACE Vol.169
ISSN (Print): 2755-273X
ISSN (Online): 2755-2721
ISBN (Print): 978-1-80590-209-6
ISBN (Online): 978-1-80590-210-2
Download Cover

Abstract

The National Aeronautics and Space Administration proposes that in the process of changing the civil aviation driving mode from Dual-Pilot Operation to Single-Pilot Operation, the problem of pilot workload assessment needs to be solved. To address this problem, the first need is to realize high-precision real-time detection of pilot behavior. In this paper, based on multi-source heterogeneous data, we realized the pixel-level weighted fusion of multiple sets of differentiated weights, proposed an improved pilot behavior detection model on the basis of YOLOv8m-pose, and carried out the model evaluation in the two dimensions of operation speed and detection accuracy, on the mixed behavioral dataset, the proposed model achieves a mAP@0.5:0.95 improvement of 208.7% over original model, with a detection speed of 39 FPS. It enables high-precision, real-time pilot behavior detection, providing partial theoretical support for Single-Pilot Operation workload assessment.

Keywords:

pilot behavior, heterogeneous data, data fusion, behavior detection, YOLO model

View PDF
Zu,Z.;Shen,Y. (2025). Research on Pilot Behavior Detection Model Based on Multi-Source Heterogeneous Data. Applied and Computational Engineering,169,86-100.

References

[1]. Comerford D , Brandt S L , Lachter J B , et al.NASA's Single-Pilot Operations Technical Interchange Meeting: Proceedings and Findings [J]. 2013: 67-97.

[2]. WANGM, XIAO G, WANGGQSingle-Pilot operation mode technology [J. Acta Aeronautica et Astronautica Sinica, 2020, 41(4); 323541 (in Chinese). doi10.7527/S1000-6893.2019.23541.

[3]. Mohanavelu K , Poonguzhali S , Ravi D , et al.Cognitive Workload Analysis of Fighter Aircraft Pilots in Flight Simulator Environment [J].Defence Science Journal, 2020, 70(2): 131-139.DOI: 10.14429/dsj.70.14539.

[4]. Cao, X.; MacNaughton, P.; Cadet, L.R.; Cedeno-Laurent, J.G.; Flanigan, S.; Vallarino, J.; Donnelly-McLay, D.; Christiani, D.C.; Spengler, J.D.; Allen, J.G. Heart Rate Variability and Performance of Commercial Airline Pilots during Flight Simulations.  Int. J. Environ. Res. Public Health  2019,   16, -237.

[5]. Alaimo, A.; Esposito, A.; Orlando, C.; Simoncini, A. Aircraft Pilots Workload Analysis: Heart Rate Variability Objective Measures and NASA-Task Load Index Subjective Evaluation. Aerospace 2020, 7-137.

[6]. Wagner, M.; Sahar, Y.; Elbaum, T.; Botzer, A.; Berliner, E. Grip force as a measure of stress in aviation. Int. J. Aviat. Psychol. 2015, 25, 157–170.

[7]. Li, Y.; Li, K.; Wang, S.; Chen, X.; Wen, D. Pilot Behavior Recognition Based on Multi-Modality Fusion Technology Using Physiological Characteristics. Biosensors 2022, 12, 404.

[8]. WANG Lei, 20U Ying, WANG Shuo. Ouantitative evaluation of pilots'unsate operation behavior driven by OAR data.China Safety Science Journal, 2023.33(5).49-56.

[9]. yu shen, "Airline Pilot Action (APA)", IEEE Dataport, November 14, 2024, doi: 10.21227/kat2-s264.

[10]. Kumar M .Total variation regularization-based pixel level image fusion [C]//Signal Processing Systems.IEEE, 2010.DOI: 10.1109/SIPS.2010.5624819.

[11]. Yan S , Rao Y , Hou W .Detection in Complex Scenes Using Rgb and Depth Multimodal Feature Fusion [C]//ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).0 [2025-04-30].DOI: 10.1109/ICASSP48485.2024.10448205.

[12]. ]Yao J , Zhang Y , Liu F , et al.Object Detection Based On Decision Level Fusion [J]. 2019.DOI: 10.1109/CAC48633.2019.8997222.

[13]. Chen H , Zhou G , Jiang H .Student Behavior Detection in the Classroom Based on Improved YOLOv8 [J].Sensors (14248220), 2023, 23(20).DOI: 10.3390/s23208385.

[14]. M. Li and F. Liu, "A Novel Finetuned YOLOv8 Transfer Learning Model for Smoking Behavior Detection, "  2024 International Conference on Image Processing, Computer Vision and Machine Learning (ICICML), Shenzhen, China, 2024, pp. 1944-1949, doi: 10.1109/ICICML63543.2024.10957847.

[15]. R. Varghese and S. M., "YOLOv8: A Novel Object Detection Algorithm with Enhanced Performance and Robustness, " 2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS), Chennai, India, 2024, pp. 1-6, doi: 10.1109/ADICS58448.2024.10533619.

References

[1]. Comerford D , Brandt S L , Lachter J B , et al.NASA's Single-Pilot Operations Technical Interchange Meeting: Proceedings and Findings [J]. 2013: 67-97.

[2]. WANGM, XIAO G, WANGGQSingle-Pilot operation mode technology [J. Acta Aeronautica et Astronautica Sinica, 2020, 41(4); 323541 (in Chinese). doi10.7527/S1000-6893.2019.23541.

[3]. Mohanavelu K , Poonguzhali S , Ravi D , et al.Cognitive Workload Analysis of Fighter Aircraft Pilots in Flight Simulator Environment [J].Defence Science Journal, 2020, 70(2): 131-139.DOI: 10.14429/dsj.70.14539.

[4]. Cao, X.; MacNaughton, P.; Cadet, L.R.; Cedeno-Laurent, J.G.; Flanigan, S.; Vallarino, J.; Donnelly-McLay, D.; Christiani, D.C.; Spengler, J.D.; Allen, J.G. Heart Rate Variability and Performance of Commercial Airline Pilots during Flight Simulations.  Int. J. Environ. Res. Public Health  2019,   16, -237.

[5]. Alaimo, A.; Esposito, A.; Orlando, C.; Simoncini, A. Aircraft Pilots Workload Analysis: Heart Rate Variability Objective Measures and NASA-Task Load Index Subjective Evaluation. Aerospace 2020, 7-137.

[6]. Wagner, M.; Sahar, Y.; Elbaum, T.; Botzer, A.; Berliner, E. Grip force as a measure of stress in aviation. Int. J. Aviat. Psychol. 2015, 25, 157–170.

[7]. Li, Y.; Li, K.; Wang, S.; Chen, X.; Wen, D. Pilot Behavior Recognition Based on Multi-Modality Fusion Technology Using Physiological Characteristics. Biosensors 2022, 12, 404.

[8]. WANG Lei, 20U Ying, WANG Shuo. Ouantitative evaluation of pilots'unsate operation behavior driven by OAR data.China Safety Science Journal, 2023.33(5).49-56.

[9]. yu shen, "Airline Pilot Action (APA)", IEEE Dataport, November 14, 2024, doi: 10.21227/kat2-s264.

[10]. Kumar M .Total variation regularization-based pixel level image fusion [C]//Signal Processing Systems.IEEE, 2010.DOI: 10.1109/SIPS.2010.5624819.

[11]. Yan S , Rao Y , Hou W .Detection in Complex Scenes Using Rgb and Depth Multimodal Feature Fusion [C]//ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).0 [2025-04-30].DOI: 10.1109/ICASSP48485.2024.10448205.

[12]. ]Yao J , Zhang Y , Liu F , et al.Object Detection Based On Decision Level Fusion [J]. 2019.DOI: 10.1109/CAC48633.2019.8997222.

[13]. Chen H , Zhou G , Jiang H .Student Behavior Detection in the Classroom Based on Improved YOLOv8 [J].Sensors (14248220), 2023, 23(20).DOI: 10.3390/s23208385.

[14]. M. Li and F. Liu, "A Novel Finetuned YOLOv8 Transfer Learning Model for Smoking Behavior Detection, "  2024 International Conference on Image Processing, Computer Vision and Machine Learning (ICICML), Shenzhen, China, 2024, pp. 1944-1949, doi: 10.1109/ICICML63543.2024.10957847.

[15]. R. Varghese and S. M., "YOLOv8: A Novel Object Detection Algorithm with Enhanced Performance and Robustness, " 2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS), Chennai, India, 2024, pp. 1-6, doi: 10.1109/ADICS58448.2024.10533619.

Cite this article

Zu,Z.;Shen,Y. (2025). Research on Pilot Behavior Detection Model Based on Multi-Source Heterogeneous Data. Applied and Computational Engineering,169,86-100.

Data availability

The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.

About volume

Volume title: Proceedings of CONF-MSS 2025 Symposium: Machine Vision System

ISBN: 978-1-80590-209-6(Print) / 978-1-80590-210-2(Online)
Editor: Cheng Wang, Marwan Omar
Conference website: https://www.confmss.org/chicago.html
Conference date: 5 June 2025
Series: Applied and Computational Engineering
Volume number: Vol.169
ISSN: 2755-2721(Print) / 2755-273X(Online)

© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license. Authors who publish this series agree to the following terms:

1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this series.

2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this series.

3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See Open access policy for details).

For authors
For editors
For reviewers

Cookies are used by this site.
Copyright © 2024 EWA Publishing, its licensors, and contributors. All rights reserved, including text and data mining, AI training, and similar technologies.