Collaborative plan-based dialogue system and method

Methods and systems for multimodal collaborative conversational dialogue are disclosed. The multimodal collaborative conversational dialogue system include a multimodal avatar interface and one or more sensors, which obtains one or more multimodal inputs. A multimodal semantic parser generates one or more logical form representations based on the one or more multimodal inputs. A collaborative dialogue manager infers a goal of the user from the one or more logical form representations, and develops a plan including communicative actions and non-communicative actions with regard to the goal. The multimodal avatar interface outputs one or more multimodal collaborative plan-based dialogue system-generated communications with respect to execution of at least one communicative action. The collaborative dialogue manager maintains a collaborative dialogue with the user until obtainment of the goal.