Classification of Hybrid EEG-fNIRS Signals using Reduced Deep Features for BCI Applications

Document Type : Original Article

Authors

1 Assistant Professor,,AJA Command and Staff University, Tehran, Iran

2 PhD student,Urmia University, Urmia, Iran

Abstract

The Motor Imagery (MI)-based Brain-Computer Interface (BCI) has been proposed as an effective method for direct communication between the brain and external electronic devices. In BCI systems, the main challenge is converting brain signals into reliable commands to control electronic devices. Electroencephalogram (EEG) is the most widely used signal in BCI-related research. Recently, it has been considered with some other biological signals, such as Near-Infrared Spectroscopy (NIRS), to increase the efficiency of BCI systems. The most important challenge in multi-modal BCI systems is combining the extracted features from different signals. For this purpose, in this paper, EEG and NIRS components, including HbO and HbR, were first decomposed into different frequency bands. Next, deep features are extracted from each band using a One-Dimensional (1D) Convolutional Neural Network (CNN). Since the final feature vector has a high dimension, Kernel Principal Component Analysis (KPCA) is employed to remove the irrelevant features, and the remaining ones are classified using the Support Vector Machine (SVM). The results show that the proposed method has high accuracy and improves the recently presented methods.

Keywords

Smiley face

https://creativecommons.org/licenses/by/4.0/

References

[1]       Khare, S. K.; Bajaj, V. “A Facile and Flexible Motor Imagery Classification Using Electroencephalogram Signals”; Comput. Meth. Prog. Bio. 2020, 197, 105722. http://doi.org/10.1016/j.cmpb.2020.105722.
[2]       Ang, K.K.; Chin, Z.Y.; Zhang, H.; Guan, C. “Filter Bank Common Spatial Pattern (FBCSP) in Brain-Computer Interface”; IEEE Int. Conf. Neu. Net. 2008, 2390-2397. http://doi.org/10.1109/IJCNN.2008.4634130.
[3]        Zhang, R.; Li, Y.; Yan, Y.; Zhang, H.; Wu, S.; Yu,T; Gu, Z. “Control of a Wheelchair in an Indoor Environment Based on a Brain–Computer Interface and Automated Navigation”; IEEE Trans. Neu. Syst. Rehabil. Eng. 2015, 24, 128-139.
http://doi.org/10.1109/TNSRE.2015.2439298.
[4]       Wang, H.; Dong, X.; Chen, Z.; Shi, B. E. “Hybrid Gaze/EEG Brain Computer Interface for Robot Arm Control on a Pick and Place Task”; Int. Conf. IEEE Eng. Med. Bio. Soc. 2015, 1476-1479. http://doi.org/10.1109/EMBC.2015.7318649.
[5]       Paszkiel, S. “Using BCI Technology for Controlling a Mobile Vehicle”; Analysis and Classification of EEG Signals for Brain–Computer Interfaces: Springer 2020,          71-77. http://doi.org/10.1007/978-3-030-13273-6_34.
[6]       
 
LaFleur, K.; Cassady, K.; Doud, A.; Shades, K.; Rogin, E.; He, B. “Quadcopter Control in Three-Dimensional Space Using a Noninvasive Motor Imagery-Based Brain–Computer Interface”; J. Neural Eng., 2013, 046003. http://doi.org/ 10.1088/1741-2560/10/4/046003.
[7]       Schirrmeister, R.T.; Springenberg, J.T.; Fiederer, L.D.J., Glasstetter, M.; Eggensperger, K.; Tangermann, M.; Hutter, F.; Burgard, W.; Ball, T. “Deep Learning with Convolutional Neural Networks for EEG Decoding and Visualization”; Hum. Brain Mapp. 2017, 38, 5391-5420. http://doi.org/ 10.48550/arXiv.1703.05051.
[8]       Sawangjai, P.; Hompoonsup, S.; Leelaarporn, P.; Kongwudhikunakorn, S.; Wilaiprasitporn, T. “Consumer Grade EEG Measuring Sensors as Research Tools: A Review”; IEEE Sensor. J. 2019, 20, 3996-4024. http://doi.org/10.1109/JSEN.2019.2962874.
[9]       Ergün, E.; Aydemir, Ö. “A Hybrid BCI Using Singular    Value Decomposition Values of the Fast Walsh Hadamard Transform Coefficients”; IEEE Trans. Cogn. Devel. Syst. 2020.  http://doi.org/10.1109/TCDS.2020.3028785.
[10]    Ghonchi, H.; Fateh, M.; Abolghasemi, V.; Ferdowsi, S.; Rezvani, M. “Deep Recurrent–Convolutional Neural Network for Classification of Simultaneous EEG–fNIRS Signals”; IET Signal Proc. 2020, 14, 142-153. http://doi.org/10.1049/iet-spr.2019.0297.
[11]    Chae, Y.; Jeong, J.; Jo, S. “Toward Brain-Actuated Humanoid Robots: Asynchronous Direct Control Using an EEG-Based BCI”; IEEE Trans. on Robot. 2012, 28,       1131-1144. http://doi.org/10.1109/TRO.2012.2201310.
[12]    Li, Y.; Li, X.; Ratcliffe, M.; Liu, L.; Qi, Y.; Liu, Q. “A Real-Time EEG-Based BCI System for Attention Recognition in Ubiquitous Environment”; Int. Workshop Ubiquitous Affective Awareness and Intelligent Interaction 2011, 33-40.
http://doi.org/10.1145/2030092.2030099
[13]    Alazrai, A.; Alwanni, H.; Daoud, M. I. “EEG-Based BCI System for Decoding Finger Movements Within the Same Hand”; Neurosci. Lett. 2019, 698, 113-120. http://doi.org/ 10.1016/j.neulet.2018.12.045.
[14]    Mondini, V.; Mangia, A. L.; Cappello, A. “EEG-Based BCI System Using Adaptive Features Extraction and Classification Procedures”; Computat. Intell. Neurosci. 2016, 4562601. http://doi.org/10.1155/2016/4562601.
[15]    Arvaneh, M.; Guan, C.; Ang, K.; Quek, C. “Optimizing the Channel Selection and Classification Accuracy in EEG-Based BCI”; IEEE Trans. Bio-Med. Eng. 2011, 58,        1865-1873. http://doi.org/10.1109/TBME.2011.2131142.
[16]    Gaur, P.; Pachori, R. B.; Wang, H.; Prasad, G. “A          Multi-Class EEG-Based BCI Classification Using Multivariate Empirical Mode Decomposition Based Filtering and Riemannian Geometry”; Expert Syst. Appl. 2018, 95, 201-211. http://doi.org/10.1016/j.eswa.2017.11.007.
[17]    Paszkiel, S. “Using Neural Networks for Classification of the Changes in the EEG Signal Based on Facial Expressions”; Analysis and Classification of EEG Signals for Brain–Computer Interfaces: Springer 2020, 41-69. http://doi.org/ 10.1016/j.bspc.2016.11.018.
[18]    Pan, J.; Li, Y.; Wang, J. “An EEG-Based Brain-Computer Interface for Emotion Recognition”; Int. Conf. Neural Networks 2016, 2063-2067. http://doi.org/10.1109/TAFFC. 2019.2901456.
[19]    Tan, C.; Sun, F.; Zhang, W. “Deep Transfer Learning for EEG-Based Brain-Computer Interface”; IEEE Int. Conf. Acoustics, Speech and Signal Processing 2018, 916-920. http://doi.org/10.48550/arXiv.1808.01752. 
[20]    Fahimi, F.; Zhang, Z.; Goh, W. B.; Lee, T. -S.; Ang, K.; Guan, C. “Inter-Subject Transfer Learning with an End-to-End Deep Convolutional Neural Network for EEG-Based BCI”; J. Neural Eng. 2019, 16, 026007. http://doi.org/ 10.1088/1741-2552/aaf3f6.
[21]    Mousavi, H.; Shahrokh-Abadi, M. H.; Tavakkoli, H. “Classification of Brain Signals in BCI System Using Wavelet Transform” J. New Appr. Basic Sci., Tech. Eng. Res. 2020, 2, 6, 20-29. (In Persian)
[22]    Borgheai, S. B.; McLinden, J.; Zisk, A. H.; Hosni, S. I.; Deligani, R.J.; Abtahi, M.; Mankodiya, K.; Shahriari, Y. “Enhancing Communication for People in Late-Stage ALS Using an fNIRS-Based BCI System”; IEEE Trans. Neural Syst. Rehabil. Eng. 2020, 28, 1198-1207. http://doi.org/ 10.1109/TNSRE.2020.2980772.
[23]    Bauernfeind, G.; Steyrl, D.; Brunner, C.; Müller-Putz, G.R. “Single Trial Classification of fNIRS-Based Brain-Computer Interface Mental Arithmetic Data: a Comparison Between Different Classifiers”; Int. Conf. IEEE Engineering in Medicine and Biology Society 2014, 2004-2007. http://doi.org/10.1109/EMBC.2014.6944008.
[24]    Zhang, S.; Zheng, Y.; Wang, D.; Wang, L.; Ma, J.; Zhang, J.; Xu, W.; Li, D.; Zhang, D. “Application of a Common Spatial Pattern-Based Algorithm for an fNIRS-Based Motor Imagery Brain‐Computer Interface”; Neurosci. Lett. 2017, 655, 35-40. http://doi.org/10.1016/j.neulet.2017.06.044
[25]    Erdoĝan, S. B.; Özsarfati, E.; Dilek, B.; Kadak, S.L.; Hanoĝlu, K.; Akın, A. “Classification of Motor Imagery and Execution Signals with Population-Level Feature Sets: Implications for Probe Design in fNIRS Based BCI”; J. Neural Eng. 2019, 16, 026029. http://doi.org/10.1088/1741-2552/aafdca
[26]    Noori, F. M.; Naseer, N.; Qureshi, N. K.; Nazeer, H.; Khan, R.A. “Optimal Feature Selection from fNIRS Signals Using Genetic Algorithms for BCI”; Neurosci. Lett. 2017, 647, 61-66.  http://doi.org/10.1016/j.neulet.2017.03.013.
[27]    Trakoolwilaiwan, T.; Behboodi, B.; Lee, J.; Kim, K.; Choi, J.-W. “Convolutional Neural Network for High-Accuracy Functional Near-Infrared Spectroscopy in a Brain–Computer Interface: Three-Class Classification of Rest, Right-, and Left-hand Motor Execution”; Neurophotonics 2017, 5, 011008. http://doi.org/10.1117/1.NPh.5.1.011008.
[28]    Chhabra, H.; Shajil, N.; Venkatasubramanian, G.; “Investigation of Deep Convolutional Neural Network for Classification of Motor Imagery fNIRS Signals for BCI Applications”; Biomed. Signal Proc. Cont. 2020, 62, 102133. http://doi.org/10.1016/j.bspc.2020.102133.
[29]    Fazli, S.; Mehnert, J.; Steinbrink, J.; Curio, G.; Villringer, A.; Müller, K.R.; Blankertz, B. “Enhanced Performance by a Hybrid NIRS–EEG Brain Computer Interface”; Neuroimage 2012, 59, 519-529. http://doi.org/10.1016/j.neuroimage. 2011.07. 084.
[30]    Liu, Y.; Ayaz, H.; Shewokis, P. A. “Multi-Subject Learning for Mental Workload Classification Using Concurrent EEG, fNIRS, and Physiological Measures”; Fronti. Hum. Neurosci. 2017, 11, 389. http://doi.org/10.3389/fnhum. 2017.00389.
[31]    
 
Firooz. S.; Setarehdan, S. K. “IQ Estimation by Means of EEG-fNIRS Recordings During a Logical-Mathematical Intelligence Test”; Computers in biology and medicine, 2019, 110, 218-226. http://doi.org/ 10.1016/j.compbiomed. 2019.05.017.
[32] Kim, H. J.; Wang, I. N.; Kim, Y. T.; Kim, H.; Kim, D. J. “Comparative analysis of NIRS-EEG Motor Imagery Data Using Features from Spatial, Spectral and Temporal Domain”; IEEE Int. Conf. Brain-Computer Interface (BCI) 2020, 1-4. http://doi.org/10.1109/BCI48061.2020.9061636.
[33] Chiarelli, A. M.; Croce, P.; Merla, A.; Zappasodi, F. “Deep Learning for Hybrid EEG-fNIRS Brain–Computer Interface: Application to Motor Imagery Classification”; J. Neural Eng. 2018, 15, 3, 036028. http://doi.org/10.1088/1741-2552/aaaf82.
[34]    Rahman, M.; Uddin, M. S.; Ahmad, M. “Modeling and Classification of Voluntary and Imagery Movements for Brain–Computer Interface from fNIR and EEG Signals Through Convolutional Neural Network”; Health Info. Sci. Syst. 2019, 7, 1-22. http://doi.org/10.1007/s13755-019-00815.
[35]    Kwak, Y., Song; W.-J.; Kim, S. E. “FGANet: fNIRS-Guided Attention Network for Hybrid EEG-fNIRS Brain-Computer Interfaces,” IEEE Trans. Neu. Sys. Reh. Eng. 2022, 30,   329-33. http://doi.org/10.1109/TNSRE.2022.3149899.
[36]    Chen, J.; Wang, D.; Hu, B.; Yi, W.; Xu, M.; Chen, D.; Zhao, Q. “MCFHNet: Multi-Channel Fusion Hybrid Network for Efficient EEG-fNIRS Multi-modal Motor Imagery Decoding” in 44th Annual Intl. Conf. IEEE Eng. Med. & Bio.Soc.(EMBC) 2022, 4821-4825. http://doi:.org/10.1109/ EMBC48229.2022.9871385.
[37]    Hosni, S. M. I.; Borgheai, S. B.; McLinden, J.; Zhu, S.; Huang, X.; Ostadabbas, S.; Shahriari, Y. “A Graph-Based Nonlinear Dynamic Characterization of Motor Imagery Toward an Enhanced Hybrid BCI” Neuroinformatics 2022, 20, 1169-1189. http://doi.org/10.1007/s12021-022-09595-2.
[38]    Shin, J.; von Lühmann, A.; Blankertz, B.; Kim, D.W.; Jeong, J.; Hwang, H. J; Müller, K.R. “Open Access Dataset for EEG+ NIRS Single-Trial Classification”; IEEE Trans. Neural Syst. Rehabil. Eng. 2016, 25, 1735-1745. http://doi.org/10.1109/TNSRE.2016.2628057.
[39] Khan, M. J.; Hong, K. S. “Hybrid EEG–fNIRS-Based Eight-Command Decoding for BCI: Application to Quadcopter Control”; Fronti. Neurorobotics 2017, 11, 6.  http://doi.org/ 10.3389/fnbot.2017.00006.
[40] Yang, J.; Yao, S.; Wang, J. “Deep Fusion Feature Learning Network for MI-EEG Classification”; IEEE Access 2018, 6, 79050-79059. http://doi.org/10.1109/ACCESS.2018. 2877452
[41] Zeng, H.; Yang, C.; Dai, G.; Qin, F.; Zhang, J.; Kong, W.; “Classification of Driver Mental States by Deep Learning Cognitive”; Neurodyn 2018, 12, 597-606. http://doi.org/ 10.1007/ s11571-018-9496-y
[42] Zhang, D.; Yao, L.; Zhang, X.; Wang, S.; Chen, W.; Boots, R; Benatallah, B. “Cascade and Parallel Convolutional Recurrent Neural Networks on EEG-Based Intention Recognition for Brain Computer Interface”; AAAI Conf. Art. Intell. 2018.  http://doi.org/10.48550/arXiv.1708.06578
[43]    Ho, T. K. K.; Gwak, J.; Park, C. M.; Khare, A.; Song, J. I. “Deep Leaning-Based Approach for Mental Workload Discrimination from Multi-Channel fNIRS”; Recent Trends in Communication, Computing, and Electronics: Springer 2019, 431-440. http://doi.org/10.1007/978-981-13-2685-1-41.
Journal of Advanced Defense Science & Technology
Volume 14, Issue 3 - Serial Number 53
November 2023
Pages 141-151

Files

History

  • Receive Date: 26 July 2023
  • Revise Date: 17 October 2023
  • Accept Date: 27 October 2023
  • Publish Date: 04 December 2023

Share

How to cite

Statistics