For simplicity and efficiency, we directly catenate the visual feature vector with multimodal feature vector begin{array}rcl@ begin{aligned} f V,M =left[alpha V^{mathrm{T}}, beta M^{mathrm{T}}right] end{aligned} end{array} (5).
where V is the visual feature vector, i.e., activations-based or FC-based feature vector, and M=[m1,⋯,m⋯,m p ]T denotes the multimodal feature vector.
The visual feature vector o v t is formed by 4 of the components of vector α, related to the movements of the mouth ( o v 1 : upper lip vertical movement, o v 2 : lip stretching, o v 3 : mouth opening control, and o v 4 : lip corner movement).
As described in Section 4.2, the visual feature vector is estimated according to (7) in a recursive way.
Here we characterize each patch using three channels of local visual descriptors: vector quantized SIFT [25], color histograms, and Gabor textons.
The well known pose ambiguity arising from the use of linear camera models is solved at the control level by choosing a hybrid visual state vector including both image space (2D) information and 3D object parameters.
Consciousness vectors, however, analogous in function to visual motion vectors in area V5/MT and arising from prefrontal function, have been proposed to underlie this function [56], [57].
The codebook CB = {C 1, C 2, …, C Nc } is a set of representative visual image vectors derived from the Linde-Buzo-Gray training algorithm [23], and each significant vector C i is called a codeword.
The audio-visual observation o a v t is partitioned as o avt ≜ o at T, o vt T T, where o a t and o v t are the audio and visual observation vectors, respectively.
