Hearing with the eyes: modulating lyrics typography for music visualization

In human–computer interaction (HCI), typography was initially used for visual communication, which enhanced visual interest in graphic design. The investigation of how modulating visual elements (e.g., typography) to visualize sound (e.g., voice) has received substantial attention. Musical lyrics typography is a commonly used form of visual communication. However, the mapping of musical features to lyrics typography features is rarely diverse studied. And modulating lyrics typography from musical features by a certain model may not strongly arouse people’s perception of auditory and visual connections. In this paper, we first proposed several models of modulating typography from musical features. Then, we investigated which model can modulate lyrics typography well to visualize music. Finally, the experiment results show that the lyrics typography by the mapping of musical features (loudness, pitch, and duration) and typography feature (character size, baseline shift, and character width) can better arouse people’s perception of auditory and visual connection. And the lyrics typography modulated by a moderate mapping parameter can evoke people’s high visual aesthetic preference. We aspire for our work to offer a novel perspective on music visualization, assisting the hard-of-hearing people to experience musical content.