System and method of providing conversational visual prosody for talking heads

A system and method of controlling the movement of a virtual agent while the agent is listening to a human user during a conversation are disclosed. The method includes receiving speech data from the user to yield received speech data and analyzing the received speech data according to a prosodic characterization of the speech data to yield the speech data analysis. The method further includes controlling listening movement of the agent according to the speech data analysis, wherein the listening movement of the agent occurs while the user is speaking the received speech data to the agent such that the agent appears to respond to the specific received speech data.