Robust geometrical-based lip-reading using Hidden Markov models

Lip reading is a process used to recognize speech from the viewed physical movements of the lips. In this paper, we present a new automatic lip-reading system that uses geometrical information extracted from video sequences in the classification of dynamic lip movements and implemented in four variants of Hidden Markov Models. In the recognition of the English digits 0 to 9 as spoken by the subjects available in the CUAVE database, the proposed system is able to produce a word recognition performance of up to 68%, a result better than that obtained using a conventional appearance-based Discrete Cosine Transform technique. The two approaches are also compared when operating under simulated changes in environment conditions that arise from head movements and alterations in image illumination. The performance of the appearance-based approach was adversely affected by such rotational and brightness changes, yet the performance of the geometrical-based method remained consistent, demonstrating its potential to be effective as part of a multimodal speech recognition system for use in noisy environments.