Multi-modal smartpen computing system

In a pen-based computing system, a smart pen allows user interaction with the pen- based computing system using multiple modalities. Generally, the modalities are categorized as input (or command) modalities and output (or feedback) modalities. Examples of input modalities for the smart pen include writing with the smart pen to provide written input and/or speaking or otherwise providing sound to give audio input to the smart pen. Examples of output modalities for the smart pen include reading visual information displayed by the system, receiving haptic feedback and/or listening to sound played by the system