Evaluation of object-based audio—What is the reference?

During the development of audio coding schemes, a number of methods for evaluation of the perceived audio quality have been developed. To enable comparisons across test sites, several methods have been standardized. In standardized methods like ITU-R Recommendations BS.1116 and BS.1534 (MUSHRA), the output of a codec (signal under test) is compared to an open reference. This reference is the unimpaired input of the codec. Assuming that the codec is “transparent,” the signal under test should sound exactly like this reference. For object-based audio, the input of a codec is a combination of raw audio channels and metadata describing position and other properties of the audio objects. It does not make sense to listen directly to the raw data. For listening, it is necessary to calculate the driving signal for each loudspeaker available in the listening room (rendering). Therefore, the comparison of different renderers is difficult: the renderer used to generate the reference signal has an advantage. Using a dedicated loudspeaker as the reference does not solve the problem either: loudspeakers always sound different than virtual sound objects. The presentation will discuss problems and solutions in more detail. Some promising setups based on multi attribute testing are presented.