In the past, semantic category-level decoding performance has, at least partly, been attributed to a robust response to low-level visual features59.
Alexander Huth, a postdoctoral researcher in neuroscience at UC Berkeley, has won a prestigious early-career award for his work in decoding semantic language in the brain as a recipient of the Burroughs Wellcome Fund's Career Awards at the Scientific Interface.
For this model, decoding across semantic category was significant in all 17 participants (mean = 77.2%, SD = 6.3%, min = 64.4%, max = 87.3%).
The semantic information obtained by decoding the sentence uttered is but one example of such information.
Decoding accuracies across semantic category were significant (permutation test: p < 0.05) in all participants, whereas decoding accuracies within a semantic category were significant in 12 out of 17 participants (see also Supplementary Figure 2 for a confusion matrix).
To demonstrate the mapping between the brain and semantic space learned by the zero-shot decoding algorithm, we have created an interactive visualization (https://aaltoimaginglanguage.github.io/guess/) that shows, for each target object, its coordinates in the semantic space and the corresponding BOLD activation pattern, averaged across the six trials.
Essentially, semantic search is a new way of decoding a given search query.
In order to predict the brain activity to a given object/feature, we use a linear-regression decoding approach which, for each target object or semantic feature, maps its coordinates in a multi-dimensional semantic space to a corresponding BOLD activation pattern33,45.
Third, the development of EEG-based semantic-decoding algorithms is interesting from an applications perspective since the temporal resolution of EEG allows decoding in real-time [37].
