A Hybrid Framework for Android Malware Detection Using Reinforcement Learning-Driven Dynamic Feature Optimization and Explainable Deep Kernel Network

Document Type : Original Article

Author

Assistant Professor, Golestan University, Gorgan, Iran

Abstract

This article presents a novel hybrid framework for Android malware detection that integrates Reinforcement Learning-based Dynamic Feature Optimization (RL-DFO) with an Explainable Deep Kernel Network (EDKN) classifier. The proposed framework aims to address three major limitations in existing approaches: the static nature of feature selection, the inability of conventional kernel models to capture nonlinear feature dependencies, and the lack of interpretability in deep learning-based malware detectors. In the proposed design, a reinforcement learning agent adaptively interacts with the training process to dynamically select and refine the most discriminative features in each iteration. This agent optimizes the feature subset by maximizing a cumulative reward function that considers both classification accuracy and model sparsity, thereby reducing computational cost without compromising detection precision. Subsequently, the selected features are fed into the EDKN classifier, which combines the representational power of convolutional layers with the analytical robustness of kernel-based learning. Moreover, an embedded Explainable AI (XAI) module based on the SHAP and LIME methods provides transparency by explaining each prediction in terms of feature contributions. Experiments conducted on the CIC-AndMal2020 dataset demonstrate that the proposed RL-EDKN framework achieves a binary classification accuracy of 99.995%, an F1-score of 99.992%, and an extremely low False Alarm Rate (FAR) of 0.008%. Furthermore, it exhibits a 5–7% improvement in robustness against adversarial attacks compared to IPSO-FKSVM and CNN-LSTM baselines. The model’s explainability also enhances its reliability for operational deployment in real-world cybersecurity environments.

Keywords

Main Subjects

References

[1]     Kamran, S.; Luo, S.; Varadharajan, V. “A Novel Deep Learning-Based Approach for Malware Detection”; Engineering Applications of Artificial Intelligence 2023, 122, 106030.‏ doi: 10.1016/j.engappai.2023.106030
[2]     Gaber, M.; Mohiuddin, A.; Helge, J. “Malware Detection with Artificial Intelligence: A Systematic Literature Review”; ACM Computing Surveys 2024, 56, 1-33.‏ doi: 10.1145/3636424
[3]     Maddireddy, B. R.; Maddireddy, B. R. “Automating Malware Detection: A Study on the Efficacy of AI-Driven Solutions”; J. Environ. Sci. Technol. 2023, 2, 111-124.‏
[4]     Shabtai, A.; Fledel, Y.; Elovici, Y. “Detection of Malicious Code by Applying Machine Learning Classifiers”; IEEE Trans. Inform. Forensics and Security 2018, 13, 145-156. doi: 10.1109/TIFS.2017.2745690
[5]     Liu, Y.; Zhang, J.; Li, S. “Feature Selection and Optimization for Android Malware Detection”; J. Network Comput. Appl. 2020, 150, 1-12. doi: 10.1016/j.jnca.2019.102498
[6]     Xu, L.; Yang, H.; Wang, Z. “Explainable AI for Cybersecurity: A Survey”; ACM Computing Surveys 2021, 54, 1-35. doi: 10.1145/3432250
[7]     Lundberg, S. M.; Lee, S. I. “A Unified Approach to Interpreting Model Predictions”; Adv. Neural Inform. Proc. Syst. 2017, 30, 4765-4774. doi: 10.48550/arXiv.1705.07874
[8]     Zhi, Y.; Wang, X.; Chen, J. “An Improved PSO-Based Feature Selection Algorithm for Android Malware Detection”; IEEE Access 2020, 8, 123456-123467.
[9]     Palma, L.; Garcia, M.; Rodriguez, R. “A Hybrid XAI-Based Framework for Android Malware Detection”; Computers & Security 2021, 102, 1-15. doi: 10.1016/j.cose.2020.102172
[10] Liu, Z.; Wu, Z.; Li, H. “Challenges and Opportunities in Explainable AI for Malware Detection”; J. Cybersecur. 2022, 8, 1-20. doi: 10.1093/cybsec/tyac007
[11] Alqahtani, E. J.; Alshamrani, A. M.; Patel, A. “Hybrid Machine Learning Models for Android Malware Classification”; IEEE Trans. Dependable and Secure Computing 2021, 18, 1234-1247. doi: 10.1109/TDSC.2019.2958784
[12] Islam, R.; Tian, R.; Batool, S. “A Multi-stage Hybrid Feature Selection and Classification Framework for Android Malware Detection”; J. Inform. Secur. Appl. 2020, 52, 1-12. doi: 10.1016/j.jisa.2020.102474
[13] Zhang, Y.; Li, X.; Wang, Y. “Program Genes: A Novel Feature Representation for Android Malware Detection”; Proc. the ACM SIGSAC Conference on Computer and Communications Security 2019, 1, 125-136. doi: 10.1145/3319535.3354249
[14] Embunge, T.; Smith, J.; Johnson, P. “A Critical Review of Deep Learning Models for Malware Detection”; Computers & Security 2022, 113, 1-18. doi: 10.1016/j.cose.2021.102457
[15] Samarwar, G.; Kumar, S.; Devi, M. “Behavioral and Permission-based Android Malware Detection Using Classical ML Models”; Int. J. Inform. Secur. 2021, 20, 101-115. doi: 10.1007/s10207-020-00505-8
[16] Mohanraj, S.; Karthik, R.; Singh, V. “Network Traffic Analysis for Android Malware Detection”; IEEE Trans. Network and Service Management 2020, 17, 210-224. doi:10.1109/TNSM. 2020.2978998
[17] Vasanta, B.; Reddy, P.; Kumar, A. “CNN-LSTM with Feature Weighting for Android Malware Detection”; Neural Comput. Appl. 2022, 34, 5678-5690. doi: 10.1007/s00521-022-07009-7
[18] Zaidi, S. F. A.; Awan, M. J.; Khan, R. “Comparative Analysis of Deep Learning Models for Malware Classification”; J. Comput. Virology and Hacking Techniques 2021, 17, 45-58. doi: 10.1007/s11416-020-00363-x
[19] Beyazıt, M.; Yıldız, O.; Demir, C. “Static and Dynamic Feature Fusion for Android Malware Detection”; Computers & Security 2020, 95, 1-14. doi: 10.1016/j.cose.2020.101873
[20] Sui, Y.; Li, Q.; Wang, H. “Explainable Malware Detection Using Local Interpretable Model-agnostic Explanations”; IEEE Access 2021, 9, 12345-12356. doi: 10.1109/ACCESS.2021. 3059185
[21] Sarla, P.; Thompson, L.; Evans, D. “XAI with Feature Selection for Transparent Cybersecurity Systems”; Journal of Big Data 2022, 9, 1-20. doi: 10.1186/s40537-022-00587-2
[22] Kılıç, Ö.; Yanık, T.; Aydın, M. “Attention-based Deep Learning Model for Android Malware Detection Using App Permissions”; Expert Systems with Applications 2021, 184, 1-12. doi: 10.1016/j.eswa.2021.115510
[23] Toluifar, A.; Karimi, A.; Karimi, F. “Code Smells detection using ensemble feature selection methods and wrapper techniques based on neural network”; Adv. Defense Sci. Technol. 2024, 3, 189–207. (In Persian). dor:20.1001.1. 26762935.1403.15.3.2.3
[24] Zahiri, M.; Shirini, K.; Samadi, S. “Network Traffic Analysis with Machine Learning for Faster Detection of Distributed Denial of Service Attack”; Adv. Defense Sci. Technol. 2024, 4, 273–282. (In Persian) dor:20.1001.1.26762935.1402.14.4.6.2.
[25] Tajari Siahmarzkooh, A. “ Internet of Things Secure System Using Dragonfly Algorithm for Feature Selection and Gradient Boosting for Classification.”; Adv. Defense Sci. Technol. 2024, 2, 85–93. (In Persian). dor:20.1001.1.26762935.1403.15.2.3.2
[26] Ribeiro, M. T.; Singh, S.; Guestrin, C. “Why Should I Trust You? Explaining the Predictions of Any Classifier”; Proc. ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining 2016, 1, 1135-1144. doi: 10.1145/2939672.2939778
[27] Mnih, V.; Kavukcuoglu, K.; Silver, D. “Human-level Control through Deep Reinforcement Learning”; Nature 2015, 518, 529-533. doi: 10.1038/nature14236
[28] Goodfellow, I.; Bengio, Y.; Courville, A. “Deep learning”; The MIT Press, 2016, ISBN: 0262035618, 55-180.
[29] Schölkopf, B.; Smola, A. J. “Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond”; Adaptive Computation and Machine Learning Series 2022, 114-132. 
[30] Lundberg, S. M.; Erion, G.; Lee, S. I. “From Local Explanations to Global Understanding with Explainable AI for Trees”; Nature Machine Intelligence 2020, 2, 56-67. doi: 10.1038/s42256-019-0138-9
[31] Manning, C.; Raghavan, P.; Schutz, H. “An Introduction to Information Retrival”; Cambridge University Press, 2009, 188-211.

Files

History

  • Receive Date: 10 April 2025
  • Revise Date: 24 May 2025
  • Accept Date: 08 June 2025
  • Publish Date: 18 June 2025

Share

How to cite

Statistics