Segmentation-robust representations, matching, and modeling for sign language

Distinguishing true signs from transitional, extraneous movements as the signer moves from one sign to the next is a serious hurdle in the design of continuous sign language recognition systems. This problem is further compounded by the ambiguity of segmentation and occlusions. This short paper provides an overview of our experience with representations and matching methods, particularly those that can handle errors in low-level segmentation and uncertainties of sign boundaries in sentences. We have formulated a novel framework that can address both these problems using a nested, level-building based dynamic programming approach that works for matching two instances of signs as well as for matching an instance to an abstracted statistical model in the form of a Hidden Markov Model (HMM). We also present our approach to sign recognition that does not need hand tracking over frames, but rather abstracts and uses the global configuration of low-level features from hands and faces. These global representations are used not only for recognition, but also to extract and to automatically learn models of signs from continuous sentences in a weakly unsupervised manner. Our publications that discuss these issues and solutions in more detail can be found at http://marathon.csee.usf.edu/ASL/.