Automated Recognition System of Musical Score (Vision System of the WABOT-2)

The Waseda University project team built up an intelligent robot WABOT-2 which can play an electric piano, using ten fingers and feet, while reading printed music. It can hold a conversation with a man using an artificial voice. This paper reports on its vision system, which can recognize not only a commercially available printed score but also instant lettering score in approximately 15 seconds, using an extraction algorithm based on the music score structure with special hardware and knowledge base music understanding software. Several automated recognition system for printed music have been reported However, none of them are completely adequate for use as music playing robot vision, where real time recognition is required under an insufficient data environment where a video camera is placed in the robot's head and music sheet is often set on the music stand in its curved or deformed style. In order to directly convey the vision system output to the robot hands, contradictions and inharmonic codes in the pattern discrimination results are checked, using knowledge base musical syntax. Moreover, pattern interpolations for drop-out are performed by the relaxation method at the discrimination level. Music information interpolations for missing discrimination are also achieved, based on harmonic codes and music grammar at the following understanding level. The resulting music robot vision performance is sufficient, wherein one sheet of commercially available printed of nursery song music for an electric piano with three parts can be recognized in approximately 15 seconds, with 100% accuracy.