Songle Widget: A web-based development framework for making animation and physical devices synchronized with music

This paper describes a web-based multimedia development framework, Songle Widget (http://widget.songle.jp), that makes it possible to control computer-graphic animation and physical devices such as lighting devices and robots in synchronization with music publicly available on the web. Development of applications featuring rigid synchronization with music playback on the web was difficult because audio signals of the playback are not accessible on a web browser and manual annotation is time-consuming. To overcome such difficulties, Songle Widget makes it easy to develop web-based applications with rigid music synchronization by leveraging music-understanding technologies based on signal processing. Songle Widget is implemented by using our public web service called Songle (http://songle.jp) that automatically analyzes songs on the web and annotates four important types of musical elements (music structure, hierarchical beat structure, melody line, and chords). More than 1,000,000 songs on music- or video-sharing services have been analyzed by Songle and can readily be used by music-synchronized applications. Since errors are inevitable when elements are annotated automatically, Songle Widget takes advantage of a Songle's crowdsourcing interface that enables users to correct errors. This is effective when applications require error-free annotation. We made Songle Widget open to the public, and its capabilities and usefulness have been demonstrated.