A Transformer Model with Spatiotemporal Input Embedding for fNIRS data-Driven Neural Decoding

Model structures proposed as spatial embedding of fNIRS data have a problem of embedding channels in a row into one channel without considering the brain area. This paper proposes a neural decoding model, ORC-T, using input embedding that reflects fNIRS data’s spatiotemporal characteristics. ORC-T embeds the fNIRS channels into new channels with a depth-wise convolution layer based on the fNIRS optod layout. The embedded data is transferred to the classifier via a 2D convolution layer and a transformer encoder. An open dataset (mental arithmetic, BNCI Horizon 2020) and a consciousness dataset were trained to validate model performance. The ORC-T achieved a k-fold CV accuracy of 81.78% and LOSO CV accuracy of 90.19% in open data, showing lower or equivalent performance than the existing model, fNIRS-T. In the consciousness state dataset, the ORC-T achieved a k-fold CV accuracy of 97.09% and LOSO CV accuracy of 59.43%, showing the equivalent performance as the fNIRS-T. The performance of ORC-T and fNIRS-T hardly decreased even though the number of channels input to the model was reduced. Considering the possibility that spatial embedding introduced in ORC-T did not effectively reflect the spatial information of the fNIRS channels, a model using a convolution layer with a kernel in the channel array’s horizontal, vertical, and diagonal directions is in the planning process.