The latter essentially corresponds to the generation of the respective single-modality analysis results.
Modality fusion aims at exploiting the correlations between data coming from different modalities to improve single-modality analysis results [6].
A BN structure is initially defined for performing the fusion of the computed single-modality analysis results.
We present RIPMMARC (Rotation Invariant Patch-based Multi-Modality Analysis aRChitecture), a flexible and widely applicable method for extracting information unique to a given modality from a multi-modal data set.
In all sub-figures, the vertical bars indicate the difference in classification accuracy compared to the best single-modality analysis result for each domain; the latter are given in parentheses.
Specifically, an integrated Bayesian Network is introduced for simultaneously performing information fusion of the individual modality analysis results and exploitation of temporal context, contrary to the usual practice of performing each task separately.
In Figure 5, the results for TW = 1, 2 and 3 are reported in detail, in terms of the difference in classification accuracy compared to the best single-modality analysis result for each domain.
The proposed analysis architecture employs state-of-the-art methods for the analysis of each individual modality (visual, audio, text) separately and proposes a novel fusion technique based on the particular characteristics of news-related content for the combination of the individual modality analysis results.
In particular, it adds variable significance to every single-modality analysis value (i.e. values a j, c j and m j ) by calculating the conditional probabilities P(a j |cl j ), P(c j |cl j ) and P(m j |cl j ) during training, instead of determining a unique impact factor for every modality.
Considering the corresponding SVM results (experiment 2), it is shown in Figure 5 that a significant increase (up to 9.91% in the tennis domain) in the overall classification accuracy can also be obtained for TW = 1 compared to the best single-modality analysis result, when a SVM-based classifier is used instead of the developed BN for all domains.
In the combined-modalities analysis, a common grey matter association of voice, face and name familiarity was identified in right fusiform gyrus; common grey matter associations of voice and face cross-modal recognition were identified in right temporal pole and anterior fusiform gyrus.
