HandTalker: A Multimodal Dialog System Using Sign Language and 3-D Virtual Human

In this paper, we describe HandTalker: a system we designed for making friendly communication reality between deaf people and normal hearing society. The system consists of GTS (Gesture/Sign language To Spoken language) part and STG (Spoken language To Gesture/Sign language) part. GTS is based on the technology of sign language recognition, and STG is based on 3D virtual human synthesis. Integration of the sign language recognition and 3D virtual human techniques greatly improves the system performance. The computer interface for deaf people is data-glove, camera and computer display, and the interface for hearing-abled is microphone, keyboard, and display. HandTalker now can support no domain limited and continuously communication between deaf and hearing-abled Chinese people.