Network Security Analysis Based on Web Crawler Technology
Research Article
Open Access
CC BY

Network Security Analysis Based on Web Crawler Technology

Jizhou Liu 1*
1 Trinity Collegiate School, South Carolina, Darlington, 5001 Hoffmeyer Road, SC 29532, United States
*Corresponding author: L2025326830AM@outlook.com
Published on 22 October 2025
Journal Cover
ACE Vol.196
ISSN (Print): 2755-273X
ISSN (Online): 2755-2721
ISBN (Print): 978-1-80590-451-9
ISBN (Online): 978-1-80590-452-6
Download Cover

Abstract

With the increasing complexity of network security threats, web crawler technology has become an important tool in the field of network security due to its automated information collection capability. This study synthesizes 10 relevant literatures to explore the application of crawler technology in network security. It first outlines the main types of crawlers and their technical principles, with a focus on analyzing Python-based frameworks. Then, it examines its specific applications in vulnerability detection, such as XSS cross-site scripting vulnerability detection, automated SQL injection detection, and malicious crawler identification. In addition, combined with some researches, it discusses the challenges faced by crawler technology, such as anti-crawler mechanisms, compliance, and privacy protection. Finally, it looks forward to integrating machine learning into crawler strategy optimization and constructing an intelligent security detection framework, aiming to provide references for network security research and practice. It also provides future research directions for researchers in related fields.

Keywords:

Web crawler technology, Network security, Vulnerability detection, Anti-crawler mechanism.

View PDF
Liu,J. (2025). Network Security Analysis Based on Web Crawler Technology. Applied and Computational Engineering,196,31-37.

References

[1]. Hu, H.L. (2025). Application and Prevention Strategies of Python Web Crawler Technology in Information Security Monitoring. Information and Computer (Theoretical Edition), (12), 152-154.

[2]. Liu, F.F. (2025). Research on the Application of Web Crawler Technology in XSS Vulnerability Detection. Automation Information Technology, 30(3), 20-22.

[3]. Chai, A. (2017). Design and implementation of dynamic and efficient web crawler for XSS vulnerability detection. Advances in Engineering, 126, 1169-1176.

[4]. Xiao, Q.P., Zhao, S.Q., Zhai, J.Q. (2017). Design of Automated SQL Injection Detection System Based on Web Crawler. Computer and Network, (23), 70-72.

[5]. Eswaran, S., Rani, V., Daniel, D., Ramakrishnan, J., & Selvakumar, S. (2022). An enhanced network intrusion detection system for malicious crawler detection and security event correlations in ubiquitous banking infrastructure. International Journal of Pervasive Computing and Communications, 18(1), 59-78.

[6]. Desai, K., Agrawal, S., Devulapalli, V., Kathiria, P., & Patel, A. (2017). Web crawler: Review of different types of web crawler, its issues, applications and research opportunities. International Journal of Advanced Research in Computer Science, 8(3), 1199-1202.

[7]. Sheng, J., Xu, C., Zhou, T. (2024). Analysis of Python Web Crawler Technology. China Information Times, (09), 210-212.

[8]. Ikerionwu, C., Nnadi, L. C., Okpala, I., Onuoha, M., Amadi, E. C., & Chukwudebe, G. (2020). A focused web crawler for strengthening cyber security and building a knowledge-based domain. In Proceedings of the International Conference on Emerging Applications and Technologies for Industry 4.0 (EATI 2020), 157-162.

[9]. Cui, H.H. (2023). Research on Python-based Web Crawler Technology. Information Recording Materials, 24(6), 172-174.

[10]. Wan, B., Xu, C., & Koo, J. (2023). Exploring the effectiveness of web crawlers in detecting security vulnerabilities in computer software applications. International Journal of Informatics and Information Systems, 6(2), 56-65.

References

[1]. Hu, H.L. (2025). Application and Prevention Strategies of Python Web Crawler Technology in Information Security Monitoring. Information and Computer (Theoretical Edition), (12), 152-154.

[2]. Liu, F.F. (2025). Research on the Application of Web Crawler Technology in XSS Vulnerability Detection. Automation Information Technology, 30(3), 20-22.

[3]. Chai, A. (2017). Design and implementation of dynamic and efficient web crawler for XSS vulnerability detection. Advances in Engineering, 126, 1169-1176.

[4]. Xiao, Q.P., Zhao, S.Q., Zhai, J.Q. (2017). Design of Automated SQL Injection Detection System Based on Web Crawler. Computer and Network, (23), 70-72.

[5]. Eswaran, S., Rani, V., Daniel, D., Ramakrishnan, J., & Selvakumar, S. (2022). An enhanced network intrusion detection system for malicious crawler detection and security event correlations in ubiquitous banking infrastructure. International Journal of Pervasive Computing and Communications, 18(1), 59-78.

[6]. Desai, K., Agrawal, S., Devulapalli, V., Kathiria, P., & Patel, A. (2017). Web crawler: Review of different types of web crawler, its issues, applications and research opportunities. International Journal of Advanced Research in Computer Science, 8(3), 1199-1202.

[7]. Sheng, J., Xu, C., Zhou, T. (2024). Analysis of Python Web Crawler Technology. China Information Times, (09), 210-212.

[8]. Ikerionwu, C., Nnadi, L. C., Okpala, I., Onuoha, M., Amadi, E. C., & Chukwudebe, G. (2020). A focused web crawler for strengthening cyber security and building a knowledge-based domain. In Proceedings of the International Conference on Emerging Applications and Technologies for Industry 4.0 (EATI 2020), 157-162.

[9]. Cui, H.H. (2023). Research on Python-based Web Crawler Technology. Information Recording Materials, 24(6), 172-174.

[10]. Wan, B., Xu, C., & Koo, J. (2023). Exploring the effectiveness of web crawlers in detecting security vulnerabilities in computer software applications. International Journal of Informatics and Information Systems, 6(2), 56-65.

Cite this article

Liu,J. (2025). Network Security Analysis Based on Web Crawler Technology. Applied and Computational Engineering,196,31-37.

Data availability

The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.

About volume

Volume title: Proceedings of CONF-MLA 2025 Symposium: Intelligent Systems and Automation: AI Models, IoT, and Robotic Algorithms

ISBN: 978-1-80590-451-9(Print) / 978-1-80590-452-6(Online)
Editor: Hisham AbouGrad
Conference website: https://www.confmla.org/london.html
Conference date: 12 November 2025
Series: Applied and Computational Engineering
Volume number: Vol.196
ISSN: 2755-2721(Print) / 2755-273X(Online)

© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license. Authors who publish this series agree to the following terms:

1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this series.

2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this series.

3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See Open access policy for details).

For authors
For editors
For reviewers

Cookies are used by this site.
Copyright © 2024 EWA Publishing, its licensors, and contributors. All rights reserved, including text and data mining, AI training, and similar technologies.