Multimodal user interface for the communication of the disabled

In this paper, a novel system is proposed to provide alternative tools and interfaces to blind and deaf-and-mute people and enable their communication and interaction with the computer. Several modules are developed to transform signals into other perceivable forms so that the transmitted message is conveyed despite one’s disabilities. The proposed application integrates haptics, audio and visual output, computer vision, sign language analysis and synthesis, speech recognition and synthesis to provide an interactive environment where the blind and deaf-and-mute users can collaborate. All the involved technologies are integrated into a treasure hunting game application that is jointly played by the blind and deaf-and-mute user. The integration of the multimodal interfaces into a game application serves both as an entertainment and a pleasant education tool to the users.