Referent identification requests in multi-modal dialogs

This paper describes an empirical study on what kinds of information are appropriate for referent identification requests in multi-modal dialogs, and how that information should be communicated in order to achieve the request desired. We conduct experiments in which experts explain the installation of a telephone in four situations: spoken-mode monolog; spoken-mode dialog; multi-modal monolog; and multi-modal dialog. Referent identification requests could be well analyzed from two perspectives: information communicated and the style of goal achievement. We find that there is a close relationship between the information conveyed via different communicative modes, and sketch a model that explains these results. In the model, information cannot be divided into the semantic content conveyed and the communicative modes employed, and is treated as the primitive unit for consideration. Pointing is considered as information in this sense. We also find that in dialogs, especially in spoken-mode dialogs, the speakers realize identification requests as series of fine-grained steps, and try to achieve them step by step.