Method for Selecting Appropriate Sentence from Documents on the WWW for the Open-ended Conversation Dialog System

We propose an open-ended dialog system that generates a proper sentence to a user's utterance using abundant documents on the World Wide Web as sources. Existing knowledge-based dialog systems give meaningful information to a user, but they are unsuitable for open-ended input. The system Eliza can handle open-ended input, but it gives no meaningful information. Our system lies between the above two dialog systems; it converses on various topics and gives meaningful information related to the user's utterances. The system selects an appropriate sentence as a response from documents gathered through the Web, on the basis of surface cohesion and shallow semantic coherence. The surface cohesion follows centering theory and the semantic coherence is calculated on the basis of the conditional distribution and inverse document frequency of content words (nouns, verbs, and adjectives.) We developed a trial system to converse about movies and experimentally found that the proposed method generated 66% appropriate responses.