Recognition confidence scoring and its use in speech understanding systems

In this paper we present an approach to recognition confidence scoring and a set of techniques for integrating confidence scores into the understanding and dialogue components of a speech understanding system. The recognition component uses a multi-tiered approach where confidence scores are computed at the phonetic, word, and utterance levels. The scores are produced by extracting confidence features from the computation of the recognition hypotheses and processing these features using an accept/reject classifier for word and utterance hypotheses. The scores generated by the confidence classifier can then be passed on to the language understanding and dialogue modeling components of the system. In these components the confidence scores can be combined with linguistic scores and pragmatic constraints before the system makes a final decision about the appropriate action to be taken. To evaluate the system, experiments were conducted using the jupiter weather information system. An evaluation of the confidence classifier at the word-level shows that the system detects 66% of the recognizer’s errors with a false detection rate on correctly recognized words of only 5%. An evaluation was also performed at the understanding level using key-value pair concept error rate as the evaluation metric. When confidence scores were integrated into the understanding component of the system, a relative reduction of 35% in concept error rate was achieved.