طبقه‌بندی سیگنال‌های ترکیبی EEG-fNIRS با استفاده از ویژگی‌های عمیق کاهش‌یافته برای کاربردهای BCI BCI

اصغرزاده بناب, اکبر; حاتمیان, امیر; اوریا, مهدی

طبقه‌بندی سیگنال‌های ترکیبی EEG-fNIRS با استفاده از ویژگی‌های عمیق کاهش‌یافته برای کاربردهای BCI BCI

نوع مقاله : مقاله پژوهشی

نویسندگان

¹ استادیار، دانشگاه فرماندهی و ستاد آجا، تهران، ایران

² دانشجوی دکتری ، دانشگاه ارومیه، ارومیه، ایران

چکیده

رابط مغز و کامپیوتر (BCI) مبتنی بر تخیل حرکتی (MI) به‌عنوان یک روش مؤثر برای برقراری ارتباط مستقیم بین مغز و دستگاه‌های الکترونیکی خارجی ارائه شده است. مسئله اصلی در سیستمهای BCI تبدیل سیگنالهای تولید شده در مغز به دستورات قابل‌اعتماد برای کنترل دستگاههای الکترونیکی است. سیگنال الکتروانسفالوگرافی (EEG) پرکاربردترین سیگنال در پژوهشهای مرتبط با BCI است. اخیراً ترکیب با برخی سیگنالهای حیاتی دیگر نظیر طیفسنجی نزدیک مادون‌قرمز (NIRS) برای افزایش کار آیی سیستمهای BCI مورد توجه قرار گرفته است. مهمترین چالش در سیستمهای BCI با چندین سیگنال حیاتی، استخراج ویژگی و ترکیب ویژگیهای سیگنالهای مختلف است. برای این منظور، در این مقاله ابتدا سیگنالهای EEG و اجزای NIRS، شامل HbO و HbR، به باندهای فرکانسی مختلف تجزیه شدند. در ادامه با استفاده از شبکههای عصبی کانولوشنی یکبعدی، ویژگیهای عمیق از هر زیرباند استخراج شده و با هم ادغام میشوند. با توجه به ابعاد بالای بردار ویژگی نهایی، با استفاده از تجزیه‌وتحلیل اجزای اصلی با هسته (KPCA)، ویژگیهای غیرمفید را حذف کرده و ویژگیهای باقیمانده با استفاده از بردار پشتیبان ماشین طبقهبندی میشوند. نتایج نشان میدهند روش پیشنهادی دقت بالایی دارد و روشهای ارائهشده اخیر را بهبود میدهد.

کلیدواژه‌ها

20.1001.1.26762935.1402.14.3.1.5

عنوان مقاله [English]

Classification of Hybrid EEG-fNIRS Signals using Reduced Deep Features for BCI Applications

نویسندگان [English]

Akbar Asgharzadeh-Bonab ¹
Amir Hatamian ²
mehdi URIA ¹

¹ Assistant Professor,,AJA Command and Staff University, Tehran, Iran

² PhD student,Urmia University, Urmia, Iran

چکیده [English]

The Motor Imagery (MI)-based Brain-Computer Interface (BCI) has been proposed as an effective method for direct communication between the brain and external electronic devices. In BCI systems, the main challenge is converting brain signals into reliable commands to control electronic devices. Electroencephalogram (EEG) is the most widely used signal in BCI-related research. Recently, it has been considered with some other biological signals, such as Near-Infrared Spectroscopy (NIRS), to increase the efficiency of BCI systems. The most important challenge in multi-modal BCI systems is combining the extracted features from different signals. For this purpose, in this paper, EEG and NIRS components, including HbO and HbR, were first decomposed into different frequency bands. Next, deep features are extracted from each band using a One-Dimensional (1D) Convolutional Neural Network (CNN). Since the final feature vector has a high dimension, Kernel Principal Component Analysis (KPCA) is employed to remove the irrelevant features, and the remaining ones are classified using the Support Vector Machine (SVM). The results show that the proposed method has high accuracy and improves the recently presented methods.

کلیدواژه‌ها [English]

EEG
Convolutional Neural Network
NIRS
Feature Reduction
BCI

https://creativecommons.org/licenses/by/4.0/

مراجع

[1] Khare, S. K.; Bajaj, V. “A Facile and Flexible Motor Imagery Classification Using Electroencephalogram Signals”; Comput. Meth. Prog. Bio. 2020, 197, 105722. http://doi.org/10.1016/j.cmpb.2020.105722.

[2] Ang, K.K.; Chin, Z.Y.; Zhang, H.; Guan, C. “Filter Bank Common Spatial Pattern (FBCSP) in Brain-Computer Interface”; IEEE Int. Conf. Neu. Net. 2008, 2390-2397. http://doi.org/10.1109/IJCNN.2008.4634130.

[3] Zhang, R.; Li, Y.; Yan, Y.; Zhang, H.; Wu, S.; Yu,T; Gu, Z. “Control of a Wheelchair in an Indoor Environment Based on a Brain–Computer Interface and Automated Navigation”; IEEE Trans. Neu. Syst. Rehabil. Eng. 2015, 24, 128-139.

http://doi.org/10.1109/TNSRE.2015.2439298.

[4] Wang, H.; Dong, X.; Chen, Z.; Shi, B. E. “Hybrid Gaze/EEG Brain Computer Interface for Robot Arm Control on a Pick and Place Task”; Int. Conf. IEEE Eng. Med. Bio. Soc. 2015, 1476-1479. http://doi.org/10.1109/EMBC.2015.7318649.

[5] Paszkiel, S. “Using BCI Technology for Controlling a Mobile Vehicle”; Analysis and Classification of EEG Signals for Brain–Computer Interfaces: Springer 2020, 71-77. http://doi.org/10.1007/978-3-030-13273-6_34.

[6]

LaFleur, K.; Cassady, K.; Doud, A.; Shades, K.; Rogin, E.; He, B. “Quadcopter Control in Three-Dimensional Space Using a Noninvasive Motor Imagery-Based Brain–Computer Interface”; J. Neural Eng., 2013, 046003. http://doi.org/ 10.1088/1741-2560/10/4/046003.

[7] Schirrmeister, R.T.; Springenberg, J.T.; Fiederer, L.D.J., Glasstetter, M.; Eggensperger, K.; Tangermann, M.; Hutter, F.; Burgard, W.; Ball, T. “Deep Learning with Convolutional Neural Networks for EEG Decoding and Visualization”; Hum. Brain Mapp. 2017, 38, 5391-5420. http://doi.org/ 10.48550/arXiv.1703.05051.

[8] Sawangjai, P.; Hompoonsup, S.; Leelaarporn, P.; Kongwudhikunakorn, S.; Wilaiprasitporn, T. “Consumer Grade EEG Measuring Sensors as Research Tools: A Review”; IEEE Sensor. J. 2019, 20, 3996-4024. http://doi.org/10.1109/JSEN.2019.2962874.

[9] Ergün, E.; Aydemir, Ö. “A Hybrid BCI Using Singular Value Decomposition Values of the Fast Walsh Hadamard Transform Coefficients”; IEEE Trans. Cogn. Devel. Syst. 2020. http://doi.org/10.1109/TCDS.2020.3028785.

[10] Ghonchi, H.; Fateh, M.; Abolghasemi, V.; Ferdowsi, S.; Rezvani, M. “Deep Recurrent–Convolutional Neural Network for Classification of Simultaneous EEG–fNIRS Signals”; IET Signal Proc. 2020, 14, 142-153. http://doi.org/10.1049/iet-spr.2019.0297.

[11] Chae, Y.; Jeong, J.; Jo, S. “Toward Brain-Actuated Humanoid Robots: Asynchronous Direct Control Using an EEG-Based BCI”; IEEE Trans. on Robot. 2012, 28, 1131-1144. http://doi.org/10.1109/TRO.2012.2201310.

[12] Li, Y.; Li, X.; Ratcliffe, M.; Liu, L.; Qi, Y.; Liu, Q. “A Real-Time EEG-Based BCI System for Attention Recognition in Ubiquitous Environment”; Int. Workshop Ubiquitous Affective Awareness and Intelligent Interaction 2011, 33-40.

http://doi.org/10.1145/2030092.2030099

[13] Alazrai, A.; Alwanni, H.; Daoud, M. I. “EEG-Based BCI System for Decoding Finger Movements Within the Same Hand”; Neurosci. Lett. 2019, 698, 113-120. http://doi.org/ 10.1016/j.neulet.2018.12.045.

[14] Mondini, V.; Mangia, A. L.; Cappello, A. “EEG-Based BCI System Using Adaptive Features Extraction and Classification Procedures”; Computat. Intell. Neurosci. 2016, 4562601. http://doi.org/10.1155/2016/4562601.

[15] Arvaneh, M.; Guan, C.; Ang, K.; Quek, C. “Optimizing the Channel Selection and Classification Accuracy in EEG-Based BCI”; IEEE Trans. Bio-Med. Eng. 2011, 58, 1865-1873. http://doi.org/10.1109/TBME.2011.2131142.

[16] Gaur, P.; Pachori, R. B.; Wang, H.; Prasad, G. “A Multi-Class EEG-Based BCI Classification Using Multivariate Empirical Mode Decomposition Based Filtering and Riemannian Geometry”; Expert Syst. Appl. 2018, 95, 201-211. http://doi.org/10.1016/j.eswa.2017.11.007.

[17] Paszkiel, S. “Using Neural Networks for Classification of the Changes in the EEG Signal Based on Facial Expressions”; Analysis and Classification of EEG Signals for Brain–Computer Interfaces: Springer 2020, 41-69. http://doi.org/ 10.1016/j.bspc.2016.11.018.

[18] Pan, J.; Li, Y.; Wang, J. “An EEG-Based Brain-Computer Interface for Emotion Recognition”; Int. Conf. Neural Networks 2016, 2063-2067. http://doi.org/10.1109/TAFFC. 2019.2901456.

[19] Tan, C.; Sun, F.; Zhang, W. “Deep Transfer Learning for EEG-Based Brain-Computer Interface”; IEEE Int. Conf. Acoustics, Speech and Signal Processing 2018, 916-920. http://doi.org/10.48550/arXiv.1808.01752.

[20] Fahimi, F.; Zhang, Z.; Goh, W. B.; Lee, T. -S.; Ang, K.; Guan, C. “Inter-Subject Transfer Learning with an End-to-End Deep Convolutional Neural Network for EEG-Based BCI”; J. Neural Eng. 2019, 16, 026007. http://doi.org/ 10.1088/1741-2552/aaf3f6.

[21] Mousavi, H.; Shahrokh-Abadi, M. H.; Tavakkoli, H. “Classification of Brain Signals in BCI System Using Wavelet Transform” J. New Appr. Basic Sci., Tech. Eng. Res. 2020, 2, 6, 20-29. (In Persian)

[22] Borgheai, S. B.; McLinden, J.; Zisk, A. H.; Hosni, S. I.; Deligani, R.J.; Abtahi, M.; Mankodiya, K.; Shahriari, Y. “Enhancing Communication for People in Late-Stage ALS Using an fNIRS-Based BCI System”; IEEE Trans. Neural Syst. Rehabil. Eng. 2020, 28, 1198-1207. http://doi.org/ 10.1109/TNSRE.2020.2980772.

[23] Bauernfeind, G.; Steyrl, D.; Brunner, C.; Müller-Putz, G.R. “Single Trial Classification of fNIRS-Based Brain-Computer Interface Mental Arithmetic Data: a Comparison Between Different Classifiers”; Int. Conf. IEEE Engineering in Medicine and Biology Society 2014, 2004-2007. http://doi.org/10.1109/EMBC.2014.6944008.

[24] Zhang, S.; Zheng, Y.; Wang, D.; Wang, L.; Ma, J.; Zhang, J.; Xu, W.; Li, D.; Zhang, D. “Application of a Common Spatial Pattern-Based Algorithm for an fNIRS-Based Motor Imagery Brain‐Computer Interface”; Neurosci. Lett. 2017, 655, 35-40. http://doi.org/10.1016/j.neulet.2017.06.044

[25] Erdoĝan, S. B.; Özsarfati, E.; Dilek, B.; Kadak, S.L.; Hanoĝlu, K.; Akın, A. “Classification of Motor Imagery and Execution Signals with Population-Level Feature Sets: Implications for Probe Design in fNIRS Based BCI”; J. Neural Eng. 2019, 16, 026029. http://doi.org/10.1088/1741-2552/aafdca

[26] Noori, F. M.; Naseer, N.; Qureshi, N. K.; Nazeer, H.; Khan, R.A. “Optimal Feature Selection from fNIRS Signals Using Genetic Algorithms for BCI”; Neurosci. Lett. 2017, 647, 61-66. http://doi.org/10.1016/j.neulet.2017.03.013.

[27] Trakoolwilaiwan, T.; Behboodi, B.; Lee, J.; Kim, K.; Choi, J.-W. “Convolutional Neural Network for High-Accuracy Functional Near-Infrared Spectroscopy in a Brain–Computer Interface: Three-Class Classification of Rest, Right-, and Left-hand Motor Execution”; Neurophotonics 2017, 5, 011008. http://doi.org/10.1117/1.NPh.5.1.011008.

[28] Chhabra, H.; Shajil, N.; Venkatasubramanian, G.; “Investigation of Deep Convolutional Neural Network for Classification of Motor Imagery fNIRS Signals for BCI Applications”; Biomed. Signal Proc. Cont. 2020, 62, 102133. http://doi.org/10.1016/j.bspc.2020.102133.

[29] Fazli, S.; Mehnert, J.; Steinbrink, J.; Curio, G.; Villringer, A.; Müller, K.R.; Blankertz, B. “Enhanced Performance by a Hybrid NIRS–EEG Brain Computer Interface”; Neuroimage 2012, 59, 519-529. http://doi.org/10.1016/j.neuroimage. 2011.07. 084.

[30] Liu, Y.; Ayaz, H.; Shewokis, P. A. “Multi-Subject Learning for Mental Workload Classification Using Concurrent EEG, fNIRS, and Physiological Measures”; Fronti. Hum. Neurosci. 2017, 11, 389. http://doi.org/10.3389/fnhum. 2017.00389.

[31]

Firooz. S.; Setarehdan, S. K. “IQ Estimation by Means of EEG-fNIRS Recordings During a Logical-Mathematical Intelligence Test”; Computers in biology and medicine, 2019, 110, 218-226. http://doi.org/ 10.1016/j.compbiomed. 2019.05.017.

[32] Kim, H. J.; Wang, I. N.; Kim, Y. T.; Kim, H.; Kim, D. J. “Comparative analysis of NIRS-EEG Motor Imagery Data Using Features from Spatial, Spectral and Temporal Domain”; IEEE Int. Conf. Brain-Computer Interface (BCI) 2020, 1-4. http://doi.org/10.1109/BCI48061.2020.9061636.

[33] Chiarelli, A. M.; Croce, P.; Merla, A.; Zappasodi, F. “Deep Learning for Hybrid EEG-fNIRS Brain–Computer Interface: Application to Motor Imagery Classification”; J. Neural Eng. 2018, 15, 3, 036028. http://doi.org/10.1088/1741-2552/aaaf82.

[34] Rahman, M.; Uddin, M. S.; Ahmad, M. “Modeling and Classification of Voluntary and Imagery Movements for Brain–Computer Interface from fNIR and EEG Signals Through Convolutional Neural Network”; Health Info. Sci. Syst. 2019, 7, 1-22. http://doi.org/10.1007/s13755-019-00815.

[35] Kwak, Y., Song; W.-J.; Kim, S. E. “FGANet: fNIRS-Guided Attention Network for Hybrid EEG-fNIRS Brain-Computer Interfaces,” IEEE Trans. Neu. Sys. Reh. Eng. 2022, 30, 329-33. http://doi.org/10.1109/TNSRE.2022.3149899.

[36] Chen, J.; Wang, D.; Hu, B.; Yi, W.; Xu, M.; Chen, D.; Zhao, Q. “MCFHNet: Multi-Channel Fusion Hybrid Network for Efficient EEG-fNIRS Multi-modal Motor Imagery Decoding” in 44th Annual Intl. Conf. IEEE Eng. Med. & Bio.Soc.(EMBC) 2022, 4821-4825. http://doi:.org/10.1109/ EMBC48229.2022.9871385.

[37] Hosni, S. M. I.; Borgheai, S. B.; McLinden, J.; Zhu, S.; Huang, X.; Ostadabbas, S.; Shahriari, Y. “A Graph-Based Nonlinear Dynamic Characterization of Motor Imagery Toward an Enhanced Hybrid BCI” Neuroinformatics 2022, 20, 1169-1189. http://doi.org/10.1007/s12021-022-09595-2.

[38] Shin, J.; von Lühmann, A.; Blankertz, B.; Kim, D.W.; Jeong, J.; Hwang, H. J; Müller, K.R. “Open Access Dataset for EEG+ NIRS Single-Trial Classification”; IEEE Trans. Neural Syst. Rehabil. Eng. 2016, 25, 1735-1745. http://doi.org/10.1109/TNSRE.2016.2628057.

[39] Khan, M. J.; Hong, K. S. “Hybrid EEG–fNIRS-Based Eight-Command Decoding for BCI: Application to Quadcopter Control”; Fronti. Neurorobotics 2017, 11, 6. http://doi.org/ 10.3389/fnbot.2017.00006.

[40] Yang, J.; Yao, S.; Wang, J. “Deep Fusion Feature Learning Network for MI-EEG Classification”; IEEE Access 2018, 6, 79050-79059. http://doi.org/10.1109/ACCESS.2018. 2877452

[41] Zeng, H.; Yang, C.; Dai, G.; Qin, F.; Zhang, J.; Kong, W.; “Classification of Driver Mental States by Deep Learning Cognitive”; Neurodyn 2018, 12, 597-606. http://doi.org/ 10.1007/ s11571-018-9496-y

[42] Zhang, D.; Yao, L.; Zhang, X.; Wang, S.; Chen, W.; Boots, R; Benatallah, B. “Cascade and Parallel Convolutional Recurrent Neural Networks on EEG-Based Intention Recognition for Brain Computer Interface”; AAAI Conf. Art. Intell. 2018. http://doi.org/10.48550/arXiv.1708.06578

[43] Ho, T. K. K.; Gwak, J.; Park, C. M.; Khare, A.; Song, J. I. “Deep Leaning-Based Approach for Mental Workload Discrimination from Multi-Channel fNIRS”; Recent Trends in Communication, Computing, and Electronics: Springer 2019, 431-440. http://doi.org/10.1007/978-981-13-2685-1-41.