The visual speech synthesis comprises the tasks of model training, animation control, and co-articulation modelling.
Articulation-disordered speech has a co-articulation effect between phonemes.
In all the cases, the co-articulation parameter was set to t c =3, which proved to be the optimal value in the given range.
It must be noted that the system has a constant time delay to take into account the co-articulation in the audio-visual features.
Due to chunking, there is also a co-articulation effect at the beginning and the end of each analysed phone sample.
This combination of number of states, number of mixtures and co-articulation parameter was among the ones that yielded the best results.
We consider that this is because slow queries usually posses a clearer pronunciation, and less co-articulation, than faster (medium and fast) queries.
In these experiments, the optimal value of the co-articulation parameter was t c = 3, which corresponds to a co-articulation timea of 100 ms. Compared to the previous experiment, where a reduced vocabulary (digits utterances) was employed, the number of states needed in this case increases considerably due to the fact that a larger vocabulary is being represented by the AV-HMM.
For the optimal value of the co-articulation parameter t c =3 set in this article, the system has a time delay of approximately 100 ms. These computational times are suitable for a real-time operation of the system.
The true and estimated visual parameters for the case of full covariance matrices with N = 28 states, M = 5 mixtures and co-articulation parameter t c = 3 are represented in Figure 8, where a good agreement between them can be observed.
To take into account the audio-visual co-articulation, t c preceding and t c subsequent frames are used to form the audio feature vector o at = a t - t c T, …, a t - 1 T, a t T, a t + 1 T, …, a t + t c T T associated with the visual feature vector o v t at the current frame t.
